Patent Application
20240341715
Embodiments generally relate to methods, systems, and computer-readable media for training a feature encoder for encoding sound samples, such as respiratory sounds. Some embodiments further relate to methods, systems, and computer-readable media for training an audio classifier, such as a respiratory sound classifier, using the pre-trained feature encoder. Some embodiments relate to methods, systems, and computer-readable media for classifying a sample of an audio file, such as a respiratory sound, as being a positive example or a negative example of a condition, such as a respiratory condition.
By Feb. 1, 2021, the total number of coronavirus disease 2019 (COVID-19) confirmed cases exceeded 103 million world-wide. Given that the global vaccination effort is still in its early stage, a practical and effective defensive procedure against the highly contagious COVID-19 is large-scale and timely testing, aimed at detecting and isolating the infected individuals as soon as possible.
Any discussion of documents, acts, materials, devices, articles or the like which has been included in the present specification is not to be taken as an admission that any or all of these matters form part of the prior art base or were common general knowledge in the field relevant to the present disclosure as it existed before the priority date of each claim of this application.
Some embodiments relate to a system comprising:
In some embodiments, the computer executable instructions, which when executed by the one or more processors, further cause the system to:
In some embodiments, the pre-trained feature encoder comprises a first pre-trained feature encoder and a second pre-trained feature encoder, and wherein the computer executable instructions, which when executed by the one or more processors, cause the system to provide the candidate sample to the pre-trained feature encoder further cause the system to:
Some embodiments relate to a computing device comprising:
In some embodiments, providing the sample of the audio file to the respiratory sound classification module further causes the system to:
The computing device is a mobile phone device or a smart sensor. The respiratory sound classification module may be a cough classification module.
Some embodiments relate to a method comprising:
In some embodiments, the method further comprises:
The pre-trained feature encoder may comprise a first pre-trained feature encoder and a second pre-trained feature encoder, and providing the candidate sample to the pre-trained feature encoder may comprise:
The first and second pre-trained feature encoders may be initialized using: (i) the same pre-trained weights; or (ii) different pre-trained weights.
In some embodiments, the pre-trained feature encoder comprises a third pre-trained feature encoder and wherein providing the candidate sample to the pre-trained feature encoder comprises:
The fourth and/or fifth sampling matrices may be generated using a pseudo-random number generator. The first and/or second and/or third sampling matrices may be generated using a pseudo-random number generator.
In some embodiments, determining the database comprising the plurality of samples comprises:
The feature encoder may be a Transformer based feature encoder. The feature encoder may be a Convolutional Neural Network (CNN) based feature encoder. The feature encoder may be a Recurrent Neural Network (RNN) based feature encoder.
The machine learning model may generate a similarity measure using one of: (i) a cosine similarity metric; and (ii) a bilinear similarity metric.
In some embodiments, each audio file of the first training set comprises a respiratory sound. In some embodiments, each audio file of the second training set comprises a respiratory sound. The respiratory sound may be a cough. The condition may be a respiratory condition. The respiratory condition may be a classification of a cough. The respiratory condition may be COVID-19.
In some embodiments, the audio files of the first training set do not include labels indicative of whether the sound of the audio file is a positive example of a condition or a negative example of the condition.
In some embodiments, the trained feature encoder and the trained classifier on a computing device for use.
Some embodiments relate to a method comprising:
In some embodiments, providing the sample of the audio file to the cough classification module further comprises:
In some embodiments, the output is provided to a user, for example, via a user interface.
In some embodiments, the respiratory condition or cough classification module is trained according to any one of the described methods.
Some embodiments relate to a system comprising:
Some embodiments relate to a computer-readable storage medium storing instructions that, when executed by a computer, cause the computer to perform any one of the described methods.
Some embodiments relate to a computing device comprising:
In some embodiments, memory of the computing device comprises a trained respiratory condition or cough classification module.
In some embodiments, the respiratory condition or cough classification module has been trained according to any one of the described methods.
In some embodiments, the computing device is a mobile phone device or a smart sensor.
Some embodiments relate to computer-readable media storage medium storing instructions that, when executed by a computer, cause the computer to perform any one of the described methods.
Throughout this specification the word “comprise”, or variations such as “comprises” or “comprising”, will be understood to imply the inclusion of a stated element, integer or step, or group of elements, integers or steps, but not the exclusion of any other element, integer or step, or group of elements, integers or steps.
Various ones of the appended drawings merely illustrate example embodiments of the present disclosure and cannot be considered as limiting its scope.
Embodiments generally relate to methods, systems, and computer-readable media for training a feature encoder for encoding sound samples, such as respiratory sounds. Some embodiments further relate to methods, systems, and computer-readable media for training a sound classifier, such as a respiratory sounds classifier, using the pre-trained feature encoder. Some embodiments relate to methods, systems, and computer-readable media for classifying a sample of an audio file, such as a respiratory sound, as being a positive example or a negative example of a condition, such as a respiratory condition.
Described embodiments may provide for a reliable, readily-accessible, and/or contactless approach for preliminary diagnosis of respiratory conditions, such as COVID-19. This can be of particular benefit in regions where medical supplies/workers and personal protective equipment are limited.
Cough is one of the major symptoms of COVID-19 patients. Compared to PCR (Polymerase Chain Reaction) tests and radiological images, diagnosis using cough sounds can be readily accessed by people using a computing device such as a smartphone. However, cough is also a common symptom of many other medical conditions that are not related to COVID-19. Therefore, automatically classifying respiratory sounds for a specific respiratory condition, such as COVID-19 diagnostic, is a non-trivial and challenging task.
Described embodiments relate to a self-supervised learning enabled framework for respiratory sound, such as cough, classification. The described embodiments comprise a two-phase framework, as illustrated in the schematic of
The second phase involves using the pre-trained feature encoder in conjunction with a classifier to perform respiratory sound (e.g. cough) classification. During this phase, labelled data is used to train the classifier. The weights of the contrastive pre-trained feature encoder transferred from the first phase may be fine-tuned based on the labelled dataset. In some embodiments of the second phase, multiple instances of the pre-trained feature encoder (different ensembles) with varied masking rates are provided.
Evaluations on the described embodiments as discussed in further detail below demonstrate the effectiveness of the described contrastive pre-training, the random masking mechanism, and the ensemble architecture in providing improved respiratory sound classification performance.
This approach differs from that of designing training a classifier in a fully fully-supervised way and addresses some of the limitations of such an approach. A fully-supervised approach limits the applicability, effectiveness and impact of the collected datasets, since the classifier is trained and tested on the same dataset. This means additional datasets can-not be directly used to boost the predictive performance of the classifier; it is limited to the same source dataset. Further, such fully-supervised based classification training methods would tend to need to rely on well-annotated (labelled) respiratory sound data. For example, the annotations or labels may be provided by experts or user response surveys. There are two inherent limitations of these annotation approaches: (i) Annotation Cost: Annotation of a large-scale dataset generally comes at an expensive cost (both financial cost and human power cost). In addition, unlike the data labelling in other tasks such as image classification, the annotation of respiratory sounds tends to require specific knowledge from experts. This further aggravates the difficulty of obtaining accurate annotations. (ii) Privacy Concern: Although directly asking participants to report their health status (e.g., whether the COVID-19 is tested positive or negative) during the respiratory sounds collection avoids annotation cost, the medical information is highly sensitive. Such privacy concerns also limit the distribution and publicity of gathered datasets. For example, some datasets have to be accessed by one-to-one legal agreements and specific licenses.
Described embodiments leverage large scale unlabelled respiratory sounds to improve the performance of respiratory condition classification. For example, the described embodiments can take advantage of unlabelled respiratory sounds to reduce the annotation dependency in the training process of the classifier.
Respiratory sound classification modules trained according to the described embodiments may be deployed on computing devices, such as smart phones or smart sensors to perform respiratory classification locally. In some embodiments, applications deployed on computing devices, such as smart phones or smart sensors, may be configured to capture audio files, or samples, indicative of a respiratory sound of an individual, and to transmit the audio file or sample to a system or server, such as a remote server whereon the respiratory sound classification module is deployed, and to receive a response from the system or server indicative of the output of the respiratory sound classification module, i.e., positive or negative for a particular respiratory condition.
Referring now to
A sample pair 202 of pre-processed audio or sound samples is provided as an input to the pipeline and a similarity measure based on a contrastive loss function 204 is provided as an output. The similarity measure is indicative of the similarity of the samples of the sample pair to one another. In other words, the system 200 is configured to encode audio samples into a latent space using a feature encoder so that the similarity of positive samples if larger than the negative samples in the latent space.
The samples are derived from audio files 206, pre-processed to generate one or more feature matrices from which the audio samples or clips are generated. Prior to selection of the audio files, the audio files may be filtered to remove audio files that are silent and/or too noisy. Pre-processing is performed to read and transform each audio file into a matrix format suitable for providing as an input to a feature encoder, such as feature encoder 210. In some embodiments, Mel Frequency Cepstral Coefficients (MFCC) or log-compressed mel-filterbanks may be used to generate the matrix format of the audio files, For example, the Python Speech Features package8 may be used for computing log-compressed mel-filterbanks. After such preprocessing, each raw audio file is mapped to a feature matrix a∈N×T, where N is a number of frequency bins and T is a total number of time frames in the audio file. Since different audio files in a dataset may have different lengths and/or different T values after pre-processing, a sliding window with window size Tw may be applied to generate multiple samples for each processed audio file.
If sample i: ai∈N×T
N×T
System 200 comprises a masking generator 208 for generating masking matrices with particular masking rates. The generated masking matrix is configured to have a dimension of the sample to which it is to be applied. The masking rate is a percentage of the time steps of the sample to be masked. In some embodiments, the masking matrix is pseudo randomly generated according to the masking rate such that of the time steps of the sample to be masked are pseudo randomly selected. For example, the masking generator 208 may generate a masking matrix with a masking rate of 60% where a first second and third time step (of a five time step sample) are masked and the fourth and fifth time step are not masked. The masking generator 208 may also generate a masking matrix with a masking rate of 60% where a second, fourth and fifth time step (of a five time step sample) are masked and the first and second time step are not masked. Masking elements or time steps of a sample means that the masked one or more elements of the masked sample are removed from an attention calculation of the feature encoder.
In some embodiments, the masking generator 208 is provided to generate a first masking matrix with a first masking rate and a second masking matrix with a second masking rate. The first masking matrix with a first masking rate is applied to a first sample of the sample pair, and the second masking matrix with the second masking rate is applied second sample of the sample pair. In some embodiments, the masking generator 208 used a pseudo random number generator to generate pseudo random masks and/or masking rates; in other words, the masking generated randomly generates the masking matrices with the specific masking rates. The masking rates are adjustable hyper-parameters. The first and second masking rates may be the same or different from one another. Based on the masking matrix and the masking rate, select inputs are masked, and in some embodiments, randomly masked, and removed from an attention calculation of the feature encoder 210, as discussed in more detail below with reference to
The masked samples are provided to a feature encoder 210 to generate first and second numerical representations of the respective first and second samples, 212A and 212B. The goal of the feature encoder 210 is to embed each sample, ai∈N×T
d, where d the dimension of a representation vector. This step may be formulated as:
where f (⋅) represents the feature encoder 210, Wf are trainable weights of the feature encoder 210, and Mi is the masking matrix for sample at. Similarly, hi∈d for sample aj is obtained.
In some embodiments, a Transformer structure is selected as the encoder f (⋅). As shown in
The first and second numerical representations, 212A and 212B are provided to a machine learning program or module 214 to determine a loss function value indicative of the similarity measure between the two samples.
The system 200 is configured to train the machine learning module 214 and the feature encoder 210 by contrastive learning. The machine learning module 214 comprises a projection head (⋅), which is applied to map the numerical representations 212A and 212B of the sample pair to the latent space where the similarity measure is determined.
In some embodiments, a cosine similarity metric is used, and the similarity of a sample pair is given by:
In some embodiments, a bilinear similarity metric is used, and the similarity of a sample pair is given by:
sim(ai, aj)=g(hi)TWsg(hj), where Ws is a bilinear parameter.
The performance of the Cosine similarity metric and the bilinear similarity metric is discussed in more detail below.
The loss function for contrastive learning used in the pre-training phase of system 200 may be a multi-class cross-entropy function working with the similarity metric. During the training in this phase, each training instance involves a pair of samples. Each training batch may comprise a positive pair and a plurality of negative pairs. Two samples derived from the same audio file and/or from the same individual constitute a positive pair, and two samples derived from different audio files and/or different individuals constitute a negative pair. Each training instance may be considered as a unique class, and accordingly, multi-class cross0entropy may be applied. The loss function is calculated over the batch; the batch has size B, with 2B samples per batch. In some embodiments, the loss function may be calculated as:
where τ denotes the temperature parameter for scaling, ai and aj are the positive pair, and ai and ak ((k≠i) are the negative pairs.
Weights of the machine learning program or module 214 and the feature encoder 210 may be adjusted based on the loss function value by way of backward propagation to thereby train the machine learning program or module 214 and the feature encoder 210 to provide an improved performance.
As described below with reference to method 600 of
Referring to
As an input to the pipeline 400, a candidate sample is selected from a training set of processed audio 402 and provided for each training iteration. The samples of the training set are each derived or sampled from an audio file in a manner similar to that of the pre-processing of the audio files described above in relation to
A masking generator 404 generates a first masking matrix having a first masking rate and a second masking matrix having a second masking rate. The first masking matrix with the first masking rate is applied to the candidate sample to generate a first masked candidate sample, and the second masking matrix with the second masking rate is applied to the candidate sample to generate a second masked candidate sample. In this way, two separate branches of input to the feature encoder(s) are generated.
In some embodiments, the first and second masked candidate samples are provided to a first and second instance of the pre-trained feature encoder 210, 406A and 406B, respectively; that is, the feature encoder pre-trained in accordance with the pipeline of
In some embodiments, the first and second masking matrices applied to the candidate sample are generated randomly. Accordingly, the two branches of masked candidate samples provided to the ensemble structure of feature encoders may have different masked time steps. This means that the two instances of the feature encoders 406A and 406B may model their inputs and yield encoded features from a different perspective. Thus, the ensemble structure and the random masking mechanism is a harmonized match and beneficial to each other.
In some embodiments, the ensemble structure may comprise two pre-trained feature encoders 210. In some embodiments, the ensemble structure may comprise three or more feature encoders 210. The two or more feature encoders 210 of the ensemble structure may be multiple instances of the same pre-trained encoder 210; in other words, each feature encoder of the ensemble structure may be initialised using the same pre-trained weights. In some embodiments, the ensemble structure may comprise multiple pre-trained feature encoders 210, wherein at least two of the pre-trained feature encoders were pre-trained independently of one another and are initialized using different pre-trained weights. The first and second instances of the pre-trained feature encoder 210 generate first and second numerical representations, 408A and 408B, of the first and second sampled candidate samples, respectively.
The first and second numerical representations are then provided as inputs to a classifier 410 to determine a predictive score. Unlike a more straightforward architecture, the classifier 410 has 2d input neurons as it takes the concatenated feature as input; in other words, the concatenation of two encoded features from the pre-trained feature encoder 406A and 406B, respectively.
The predictive score may be indicative of the likelihood of the candidate sample being a positive example of the condition. For example, if the probability is greater than a threshold, such as 0.5, the sample is considered to be a positive example, and vice versa.
In some embodiments, the classifier 410 may be is a fully connected layer with d (feature dimension of the encoded feature hi) input neurons and one output node with sigmoid activation function to output the probability and indicate whether the input sound sample For example, the classifier may output the probability that the input respiratory sound sample is COVID-19 positive (probability larger than a threshold, e.g. 0.5) or negative (probability smaller than the threshold).
The determined predictive score and the label associated with the candidate sample may be used to adjust one or more weights of the classifier 410 and/or one or more weights of the instances of the pre-trained feature encoder 406A and 406B to more accurately train the classifier 410 and/or the pre-trained feature encoders 406A and 406B. For example, the label may be used as a measure of how accurate the predictive score for the sample was. In some embodiments, the pre-trained feature encoder 406A and 406B may be updated differently.
By using an ensemble structure comprises at least two pre-trained feature encoders, the robustness and the reliability of the feature encoder 210 and/or classifier 410 may be improved. This may be due to the averaging of the two of more branches associated with the outputs of the pre-trained feature encoders. For example, assume that a sample has a five time-step and the most distinguishable feature is on the second time-step. With only one branch (one feature encoder 210), this feature might be missed due to the random masking. However, with the two-branch design, this feature would have a higher chance of being captured by the feature encoder 210.
At 602, the system 902 determines a first training set comprising a plurality of audio files. In some embodiments, the audio files of the first training set may not include labels indicative of whether the sound of the audio file is a positive example of a condition or a negative example of a particular condition. In embodiments where at least some of the audio files of the first training set include labels indicative of whether the sound of the audio file is a positive or negative example of a particular condition, those label are not used or required for the purposes of performing the method 600; in other words, the method involves a contrastive approach to pre-training a feature encoder 916 for audio or sound samples, such as respiratory sounds.
In some embodiments, each audio file of the first training set comprises a respiratory sound, such as a cough. In some embodiments, the condition is a respiratory condition, or a classification of a cough, such as COVID-19.
At 604, the system 902 determines a database 904 comprising a plurality of samples for each of the audio files of the first training set. Each sample may be associated with an identifier. The identifier may be indicative of the audio file from which the sample was derived and/or may be indicative of a person associated with the sound of the audio file, for example, the person the sound, such as a respiratory sound, was made by.
In some embodiments, determining the database 904 comprising the plurality of samples comprises transforming each audio file into a feature matrix. In some embodiments, pre-processing techniques such as Mel Frequency Cepstral Coefficient (MFCC) or log-compressed mel-filterbanks are used to transform audio signals into 2D matrices.
For example, each feature matrix may comprise a first dimension corresponding to a number of frequency bins, and a second dimension corresponding to a number of time frames in the audio file. The system may, for each feature matrix, generate at least one sample. Each sample may comprise the first dimension corresponding to a number of frequency bins, and a third dimension corresponding to a predefined number of time frames. The third dimension may be a subset of the second dimension. Each of the plurality of samples may comprises at least a portion of a sound of a respective audio file.
At 606, the system 902 may determine a batch of sample pairs including a positive sample pair having a common identifier, and a plurality of negative sample pairs having different identifiers. For example, the batch may comprise one positive sample pair comprising a first sample and second sample. The first sample and the second sample may be associated with a common identifier. The batch may comprise a plurality of negative sample pairs, and each negative sample pair may comprise the first sample and a respective third sample, wherein the identifier associated with each respective third sample is different to the identifier associated with the first sample. Selection and processing of a first batch constitutes a first training iteration or training count.
At 608, the system 902 may apply a first masking matrix with a first masking rate to the first sample to mask one or more select elements of the first sample in accordance with the first masking rate. The system 902 may also apply a second masking matrix having a second masking rate to the second sample to mask one or more select elements of the second sample in accordance with the second masking rate. In some embodiments, the first masking rate is the same as the second masking rate. In some embodiments, the first masking rate is different from the second masking rate.
The first and/or second and/or third sampling matrices may be generated by a masking generator 921. The making generator may comprise a pseudo-random number generator. For example, a Python random function may be used. The masking generator 921 may be configured to generate an initialized matrix corresponding to the dimension of a sample to which the masking matrix is to be applied. For example, if the time step of the sample is L, the size of the matrix is L×L. All elements of the initialised matrix comprises the same value, for example, either a ‘0’ or a ‘1’. The masking generator 921 generates a pseudo random number and determine the masking rate based on the pseudo random number. The masking generator 921 selects one or more elements of the matrix based on the masking rate and masks the selected element(s). For example, the selected elements are updated from a “0” to a “1” or vice versa. Thus, the generated masked matrix may be a two dimensional matrix filled with elements that are either “0” or “1”; “0” may be interpreted as meaning an attention operation in the feature encoder 910 will not be performed, and “1” may be interpreted as meaning an attention operation in the feature encoder 910 will be performed.
At 610, the system 902 provides the masked first sample to a feature encoder 916 to generate a first numerical representation and provides the masked second sample to the feature encoder to generate a second numerical representation.
In some embodiments, the feature encoder 916 is a Transformer based feature encoder. It has been found that Transformer based feature encoder shows promising performance in modelling time-series sequence data due to the self-attention mechanism of Transformer. Since the preprocessed audio data can be seen as time-series data as well, the Transformer may be selected as the feature encoder 910. Table 1 below provides a comparison of the Transformer encoder and other encoders. As discussed below, the Transformer based encoder demonstrated superior performance compared to the others under the same setting (e.g., whether using pre-training or fine-tuning).
In some embodiments, the feature encoder 916 may be a Convolutional Neural Network (CNN) based classifier, such as ResNet and VGG. In some embodiments, the classifier 920 may be a Recurrent Neural Network (RNN) based classifier, or similar, such as Long Short Term Memory (LSTM) networks or Gated Recurrence Units (GRUs). The performance of these two types of feature encoders (VGGish (CNN based feature encoder) and GRU) is presented in Table 1, discussed in detail below.
At 612, the system 902 provides the first and second numerical representations to a machine learning module 918 to generate a first similarity measure indicative of the similarity between the first and second samples. For example, the machine learning module 918 may generate a similarity measure using one of: (i) a cosine similarity metric; and (ii) a bilinear similarity metric.
At 614, the system 902 determines a plurality of second similarity measures. Each second similarity measure may be indicative of the similarity between the first and third samples of each of the negative sample pairs. For example, for each negative sample pair, the system 902 may apply 616 a third masking matrix having a third masking rate to the third sample to mask one or more select elements of the third sample in accordance with the third masking rate, provide 618 the masked third sample to the feature encoder to generate a third numerical representation of the masked third sample, and provide 620 the first numerical representation and the third numerical representation to the machine learning module to generate a second similarity measure between the first and third samples.
At 624, the system 902 determines a loss function value using a loss function based on the first similarity measure, and the plurality of second similarity measures.
At 626, the system 902 adjusts one or more weights of the feature encoder 916 based on the loss function value. The system 902 may also adjust one or more weights of the machine learning module 918 based on the loss function value.
At 628, responsive to the iteration count being less than a threshold value, the system 902 increments the iteration count, and again performs steps 604 to 624. If the iteration count is not less than the threshold value, the process moves to 630.
At 630, responsive to the iteration count reaching or being equal to the threshold value, the system 902 determines the feature encoder as a pre-trained feature encoder.
At 702, the system 902 determines a second training set of a plurality of audio files. For example, the system 902 may retrieve the second training set from database 904. Each audio file of the second training set comprises a label indicative of whether the sound of the audio file is a positive example of a condition or a negative example of the condition. For example, the sound may be a respiratory sound, such as a cough, and the condition may be a respiratory condition, such as COVID-19.
At 704, the system 902 determines a batch of samples from the second training set. Each sample may be generated from a respective audio file and may be associated with the label of the audio file from which it was generated. For example, the batch of samples may be generated or sampled from the audio files in a similar manner to the pre-processing of the audio files to generate the samples discussed above in relation to method 600 of
The system 902 performs 706 to 716 for each of the plurality of samples of the second batch.
At 706, the system 902 selects a candidate sample from the second batch.
The system 902 provides the candidate sample to the pre-trained feature encoder 916 to generate a candidate numerical representation of the candidate sample. In some embodiments, at 708, the system 902 applies a fourth masking matrix having a fourth masking rate to the candidate sample, and at 710, applies a fifth masking matrix having a fifth masking rate to the candidate sample. The fourth and fifth sample rates may be the same or different sampling rates. In some embodiments, at 712, the system 902 provides the fourth masked sample to a first instance of a pre-trained encoder 916 to generate a fourth numerical representation and provides the fifth masked sample to a second instance of the pre-trained encoder 916 to generate a fifth numerical representation.
The system 902 provides the candidate numerical representation(s) or the candidate numerical representation (fourth and fifth masked candidate samples) to a classifier 920 to determine a predictive score. The predictive score may be indicative of the likelihood of the candidate sample being a positive example of the condition.
At 716, the system 902 adjust one or more weights of the classifier 920 based on the label associated with the sample and the predictive score. The system 902 may also adjust one or more weights of the pre-trained feature encode 916 to fine-tune the feature encoder 916.
Once all of the samples of the batch have been applied to the classifier according to the method of
At 802, the system 902 or computing device 906 determines an audio file comprising a respiratory sound. For example, the audio file may be captured using the computing device 906, such as a microphone (not shown) and audio processing components (not shown) of the computing device 906. Where the system 902 determines the audio file, the audio file may be received by the system from a computing device having the audio file stored thereon, and/or which may have been used to capture the respiratory sound(s) of the audio file. The audio files may be preprocessed in a manner similar to that described above in connection with method 600 of
At 804, the system 902 or computing device 906 provides a sample of the audio file to a trained cough classification module 928. The cough classification module 928 has been trained using a contrastive pre-trained feature encoder 916. The cough classification module 928 comprises a trained feature encoder 916 configured to transform the input sample into a numerical representation of the sample. The numerical representation of the sample is then provided to a cough classifier 920 of the cough classification module 928 to classify the respiratory sound of the sample.
In some embodiments, a masking matrix having an associated masking rate is applied to the sample of the audio file to provide a masked sample, and it is the masked sample that is provided as an input to the trained cough classification module 928.
At 806, the system 902 or computing device 906 determines an output from the cough classifier 920 and cough classification module 928. The output is indicative of whether or not the respiratory sound is a positive or negative example of a respiratory condition.
In some embodiments, the computing device 906 may be provided to a user of the computing device 906 via the user interface 930, for example, as a display or an audio output.
In some embodiments, the output may be transmitted, by the system 902 or computing device 96, to a user device, such as a computing device 906, associated with the individual identified as being associated with the audio sample, or, for example, a clinician. For example, the output may be emailed or provided as an SMS to the computing device 906.
Referring now to
The system 902 may comprise one or more servers configured to perform functionality and/or provide services to user devices, such as the one or more computing devices 906. The system 902 comprises one or more processors 910 and memory 912 storing instructions (e.g. program code) which when executed by the processor(s) 910 causes the system 902 to function according to the described methods. The processor(s) 910 may comprise one or more microprocessors, central processing units (CPUs), graphical/graphics processing units (GPUs), application specific instruction set processors (ASIPs), application specific integrated circuits (ASICs) or other processors capable of reading and executing instruction code.
Memory 912 may comprise one or more volatile or non-volatile memory types. For example, memory 912 may comprise one or more of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. Memory 912 is configured to store program code accessible by the processor(s) 910. The program code comprises executable program code modules. In other words, memory 912 is configured to store executable code modules configured to be executable by the processor(s) 910. The executable code modules, when executed by the processor(s) 910 cause the system 902 to perform certain functionality, as described in more detail below.
Memory 912 may comprise a pre-processing module 914 configured to pre-process and prepare audio files and/or samples for processing by the feature encoder 916, as discussed above with reference to method 600 of
The system 902 further comprises a network interface 922 to facilitate communications with components of the architecture 900 across the communications network 908, such as the one or more computing devices 906, database 904 and/or other systems or servers (not shown). The network interface 922 may comprise a combination of network interface hardware and network interface software suitable for establishing, maintaining and facilitating communication over a relevant communication channel.
The database 904 may form part of, or be local to, the system 902, or may be remote from and accessible to the system 902. The database 904 may be configured to store a plurality of audio files associated with a respective plurality of individuals. Each audio file may be associated with one or more identifiers. In some embodiments, the one or more audio files may be associated with a unique identifier configured to identify one audio file from another. In some embodiments, one or more audio files may be associated with an identifier indicative of the individual associated with the sound of the audio file, i.e., the individual that made the sound, such as a respiratory sound of the individual. In some embodiments, at least some of the audio files are associated with a label indicative of whether the sound of the audio file is a positive or a negative example of a respiratory condition, such as COVID-19 (labeled audio files). The database 904 may be configured to store user details or encrypted credentials, such as username(s), password(s), biometric data and/or other user data associated with the audio files. The system 902, and in some embodiments, the computing device(s) 906, may be configured to retrieve audio files from the database 904 for processing.
The computing device(s) 906 may be a mobile device, a tablet, a laptop computer, a smart sensor or any other suitable device. The computing device(s) 906 may comprise one or more processors 924 and memory 926 storing instructions (e.g. program code), which when executed by the processor(s) 924, causes the computing device(s) 906 to perform certain functionality, including, for example, the method of
The computer device(s) 906 may comprise a user interface 930 to provide outputs to users, such as individuals, patients, and/or clinicians. For example, the user interface 930 may comprise one or more displays, touchscreens, light indicators (LEDs), sound generators and/or haptic generators which may be configured to provide feedback (e.g. visual, auditory or haptic feedback) to a user.
The computing device(s) 906 may comprise a network interface or communications module 932 to facilitate communication with the components of the communications network 110, such as the system 902 and/or database 904 and/or other computing devices.
For the evaluation of the described embodiments, two public COVID-19 respiratory datasets were used; the Coswara dataset and the COVID-19 Sounds dataset.
The Coswara dataset (N Sharma, P Krishnan, R Kumar, S Ramoji, SR Chetupalli, R Nirmala, P Kumar Ghosh, and S Ganapathy. 2020. Coswara-A database of breathing, cough, and voice sounds for COVID-19 diagnosis. In Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, Vol. 2020. International Speech Communication Association, 4811-4815) is part of Project Coswara5 which aims to build a diagnostic tool for COVID-19 based on respiratory, cough, and speech sounds. Upon until Dec. 21, 2020, there are 1,486 crowdsourced samples (collected from 1,123 males and 323 females) available at Coswara data repository6. The majority of the participants are from India (1,329 participants) and the rest participants are from other countries of five continents: Asia, Australia, Europe, North America, and South America. Four types of sounds (breathing, coughing, counting, and sustained phonation of vowel sounds) are gathered from each participant.
Similar to Coswara dataset, COVID-19 Sounds is another crowdsourcing based respiratory sound dataset. Audios are collected world-widely with a web-based app, an Android app, and an Apple app. The same curated dataset that is introduced and used in Brown et al (Chloe Brown, Jagmohan Chauhan, Andreas Grammenos, Jing Han, Apinan Hasthanasombat, Dimitris Spathis, Tong Xia, Pietro Cicuta, and Cecilia Mascolo. 2020. Exploring Automatic Diagnosis of COVID-19 from Crowdsourced Respiratory Sound Data. In KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, Aug. 23-27, 2020, Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash (Eds.). ACM, 3474-3484. https://dl.acm.org/doi/10.1145/3394486.3412865) was used. After filtering out silent and noisy samples, in this released version of dataset7, there are 141 COVID-19 positive audio recordings collected from 62 participants and 298 COVID-19 negative audio recordings from 220 participants. Both coughs and breaths appear in these recordings. Positive samples are from participants who claimed that they had tested positive for COVID-19.
Because the Coswara dataset has more participants and contains more audio samples than COVID-19 Sounds dataset, the Coswara dataset was adopted as the pre-training dataset for phase one. Note that for this pre-training dataset, the annotated labels (indicating whether the user is COVID-19 positive or negative) are not used. Furthermore, respiratory sounds including breathing sounds and cough sounds were specifically selected for pre-processing and sampling, whereas audios of sustained phonation of vowel sounds and counting sounds were ignored in the pre-training phase. Consequently, COVID-19 Sounds is used as the dataset in the second (downstream) phase. To be more specific, in the downstream phase, the whole COVID-19 Sounds dataset is randomly divided into the training set (70%), validation set (10%), and testing set (20%). For each raw audio sample for the second phase, the same pre-processing procedure is applied as well, as described above in relation to method 600 of
In the pre-processing, the shape of a processed clip ai is R64×96 as the number of mel-spaced frequency bins N is set to 64 and the sliding window size Tw is 96, which corresponds to 960 ms. The feature dimension d is set to 64. In the contrastive pre-training phase, the batch size B is selected as a relatively large number (1024). The contrastive learning may benefit from larger batch sizes (within GPU capacity) because a larger batch allows the machine learning model or module to compare the positive pair against more negative pairs. In the downstream network, the dropout is also applied to avoid over-fitting in the end-to-end fine-tuning process. The validation set in the downstream dataset is used for tuning hyperparameter d (the feature dimension of the feature encoder) and the dropout rate. The batch size is 128 for the downstream phase. All experiments (both the contrastive pre-training and the downstream phases) are trained with Adam optimiser (Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.).), a 0.001 initial learning rate with ReduceLROnPlateau9 decay setting, and executed on a desktop with an NVIDIA GeForce RTX-2080 Ti GPU with PyTorch.
To evaluate the performance of different methods, several standard classification evaluation metrics including the Receiver Operating Characteristic-Area Under Curve (ROC-AUC), Precision, and Recall were selected. The experiments demonstrate the average performance as well as the standard deviation of five runnings of each method or configuration, and the average F1 score which is calculated based on the average Precision and average Recall.
To investigate how the dimension of encoded feature and dropout rate influence the classification performance, all the combination of the following hyperparameter values (within reasonable ranges) were evaluated on the validation set: (1) Feature Dimension: [64, 128, 256]; (2) Dropout Rate: [0.0, 0.2, 0.5] (resulting in 9 combinations in total); each combination was run five times.
To evaluate the performance of contrastive pre-training and the Transformer feature encoder, the Transformer feature encoder, Transformer-CP (the suffix -CP meaning the feature encoder is contrastive pre-training enabled), is compared with several techniques with multiple configurations. Other techniques being compared include VG-Gish/GRU/Transformer (without contrastive pre-training) and GRU-CP. Recurrent Neural Networks (RNNs) are designed for handling sequence data and so, the GRU (Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. arXiv preprint arXiv:1412.3555 (2014).) is also included in the comparison. VGGish (Shawn Hershey, Sourish Chaudhuri, Daniel P W Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. 2017. CNN architectures for large-scale audio classification. In 2017 ieee international conference on acoustics, speech and signal processing (icassp). IEEE, 131-135.) is a popular convolutional neural network for audio classification. A pre-trained version10 that is pre-trained on a large scale general audio dataset Audioset (Jort F Gemmeke, Daniel PWEllis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. 2017. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 776-780.) is also used in the community.
The configuration of with or without the pre-training is summarised in the third column of Table 1, below. In addition, the second column indicates the pre-training setting. A ✓ means that the proposed self-supervised contrastive pre-training is applied. For example, both the second and third columns are ✓ for our Transformer-CP.
The experimental results of the above techniques are reported in Table 1. To be more specific, for techniques using pre-trained weights (either contrastive pre-training or conventional pre-training for VGGish), the fine-tuning option was also explored. In the fourth column of Table 1, an “x” represents that the pre-trained weights Wf are frozen and not be updated in the downstream phase, whereas a “✓” indicates that the Wf is allowed to be updated.
According to the results of Table 1, the Transformer-CP of the described embodiments, with fine-tuning, achieves the best performance (shown in bold) against all the other techniques. There are several additional findings that can be noticed from the table. First, without pre-training, VGGish has the worst performance (the first row) compared to GRU and Transformer. Using pre-trained VGGish weights (without fine-tuning) provides almost 6% accuracy gain, which indicates that the pre-trained VGGish representation is well-trained and powerful. Of all configurations that use frozen pre-trained representations (the second, sixth, and eighth rows), although VGGish (the second row) is the top performer, the performance of our Transformer-CP (the eighth row) is very close to VGGish. It may therefore be understood that the contrastive pre-trained (on a smaller scale and unlabelled dataset) self-supervised feature representation of the described embodiments is competitive with a well-trained fully-supervised VGGish representation (pre-trained on a much larger scale and well-annotated Audioset). Second, the fine-tuning in the downstream task is important for all pre-trained models, which is as expected. For both the conventional pre-trained VGGish and contrastive pre-trained GRU/Transformer, the fine-tuning could improve the accuracy by around 3%. Third, when comparing GRU vs. Transformer and GRU-CP vs. Transformer-CP, the Transformer-based techniques outperform GRU-based techniques consistently. Overall, the results show that the framework with contrastive pre-training of the described embodiments achieves a superior performance of cough classification.
In Table 2 below, two similarity metrics in contrastive learning were compared. For a fair comparison, two different feature encoder structures, GRU-CP and Transformer-CP, were also explored. As shown in the table, using bilinear similarity achieves consistent better performance with both structures on all evaluation metrics, which demonstrates that the bilinear similarity is more suitable for a cough classification task.
The random masking mechanism and different masking rates in the contrastive pre-training phase of the described embodiments was also tested. The experiment guideline for these tests involved pre-training several Transformer-CPs with multiple masking rates (0% to 100%) and then fine-tuning the pre-trained models in the downstream phase. The cough classification performance of these classification models are listed in Table 3. Please note that in the downstream phase, we the ensemble architecture was not applied, and so there is no random masking in the downstream phase for results reported in the table.
As a baseline for comparison, the performance of Transformer (without any pre-training) in Table 3 is also included. In general, all pre-trained models yield better results than the baseline Transformer and 50% masking outperforms other masking rates. When the masking rate is increasing from 0% (no masking at all) to 50%, a performance gain is observed from the table. However, when the masking rate is too large (e.g., 75% and 100%), the performance decreases. This is not surprising. For example, in the extreme 100% masking case, all the inputs would be masked, which would mean that there is no attention between any time steps. As a result, the 100% masking has the worst performance among different masking rate settings.
Different ensembles performance was also explored. Table 4 below summarises three ensemble methods. The first two are the ensemble of the Transformer feature encoder and other feature encoder structures (VGGish and GRU). No pre-trained weights were applied to these two ensembles. The third ensemble combines GRU-CP and Transformer-CP with contrastive pre-trained weights. By jointly comparing results given in Table 1 and Table 4, it can be seen that the ensemble version demonstrates a better ability than a single feature encoder based method.
Networks where the random masking is incorporated with the ensemble architecture were also investigated (as shown in
Table 6 lists the inference time (for one input instance) of each model or configuration. Since the fine-tuning does not affect the inference time, the fine-tuning configuration is removed for comparison in the table. Generally, for three different base feature encoder structures, the inference time of Transformer is on par with GRU, whereas VGGish leads Transformer/GRU by a small margin (around 0.002 milliseconds only). Although Transformer includes attention computation, it processes each time step in the input sequence in parallel, whereas GRU has to process each time step re-currently. This might explain the similar computation cost between Transformer and GRU. From the table, it appears that using contrastive pre-trained weights does not introduce a longer time for inference. This is as expected as the major difference between Transformer and Transformer-CP (or GRU vs. GRU-CP) is whether loading the pre-trained weights. This weights initialisation process appears to have almost has no influence on the inference speed.
An interesting and surprising finding is about the inference time of using different downstream random masking rates (the last five rows of Table 6). In theory, a larger masking rate would be expected to run faster as more time steps are masked and not used in attention calculation. According to the table, however, 75% rate has the largest inference time and 0% and 100% are all smaller than the rest masking rates. This can be explained by the implementation of the masking generator. In the implementation, the default masking matrix is an all-ones matrix or an all-zeros matrix (only used for masking rate 100%), where 0 means being masked and vice versa. For a given masking rate, 1 will be updated to 0 in the matrix through a “for” loop. This loop operation takes longer if more elements need to be updated (e.g., 75% rate), which causes the larger inference time for the 75% setting. Overall, even the largest time cost in the table is only 32.36×10−6 seconds (around 0.03 milliseconds). Such a low time cost is unlikely to be a bottleneck or limit the application of the proposed framework.
From another point of view, without the contrastive pre-training methodology of the described embodiments, multiple models need to be trained if multiple datasets are available. As a result, training time will likely be done per model without domain transfer, which is a potential bottleneck for large-scale deployments. However, the described embodiments addresses this training bottleneck through the contrastive pre-training phase.
It will be appreciated by persons skilled in the art that numerous variations and/or modifications may be made to the above-described embodiments, without departing from the broad general scope of the present disclosure. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2021902532 | Aug 2021 | AU | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/AU2022/050892 | 8/15/2022 | WO |
