Patent Application
20210315517
This invention relates to use of biomarkers for detection of inflammation, and more particularly to use of vocal biomarkers in detection of particular types of respiratory inflammation, for example, as might be found in COVID-19.
Inflammation is a major health issue in civilian and military populations in response to injury, infectious disease, health disorders, and environmental exposure. The inflammation can often occur anywhere in the body, including the respiratory, sinus, musculoskeletal, gastrointestinal, and central/peripheral nervous systems. While inflammation has beneficial protective and recuperative functions, too little inflammation can compromise the immune system and lead to tissue damage with injury, and excessive or chronic inflammation may lead to or be associated with long-term physiological (e.g., muscle) and neurological disease (e.g., autism) and/or other problems in psychological health (e.g., depression).
Although inflammatory responses can occur separately in these bodily systems, more often responses are coupled and modulate each other, sometimes resulting in severe concomitant physiological, neurological, and neurodevelopmental disease. For example, gut inflammation is known to modulate the central nervous system in a way that can initiate or exacerbate Parkinson's disease, autism, or depression. In detecting inflammation or its consequences, manifestations of the condition may not be symptomatic; for example, early stages of inflammation in the gastrointestinal or central nervous system does not always manifest in pain (i.e., due to reduced concentration or absence of pain receptors) and thus may be difficult to detect. Another current, highly relevant example is inflammatory response as a result of respiratory infection associated with COVID-19 which initially can often be asymptomatic in its early stages. Early warning is needed in the form of nonintrusive objective biomarkers that are sensitive (and specific) to subtle variations in neurophysiology due to inflammation in upper and lower respiratory tracts, while also able to track progression of a condition when symptoms are present, and subsequently during treatment and recovery.
In one general aspect, a multi-modal approach is used to detect and track inflammation, and identify the location of inflammation, which is revealed in changes in tissue and physiological properties that underlie sub-systems, including respiratory, sinus, laryngeal, articulatory, facial and gastrointestinal components. The approach is also applicable to inflammation of the lower diaphragm, which also underlies vocal subsystems, and may be responsible in part for breathing dysfunction and coughing. In at least some examples, the biomarkers used reflect changes in function and coordination of coupled modalities, measured through, for example, acoustic, optical, ultrasound, and infrared measures of vocal subsystems and underlying physiology. The application can provide early warning and persistent assessment of a variety of inflammatory conditions and their pathological phenotypes.
In some examples, the biomarkers involve physiological and neurocomputational models of the inflammation process and the coupling that occurs across modalities. Given the nonintrusive nature of the sensing, the approach provides a key capability for scalable, longitudinal studies that seek to capture human behavior dynamics in naturalistic environments. As an illustration of the approach, preliminary results distinguishing pre-COVID-19 and post-COVID-19 (pre-symptomatic) conditions, utilize vocal features exploiting change in respiratory function and its relationship with laryngeal and articulatory motion due to post-COVID-19 immune response and associated inflammation. These features could be potentially measured non-intrusively through mobile devices such as smartphones and tablets. Use cases include early warning and monitoring of COVID-19.
In another aspect, in general, an approach to characterizing a potential inflammatory condition in a subject involves acquiring a plurality of concurrent time signals, the time signals characterizing a plurality of physiological or neurological subsystems of the subject. Biomarkers are generated from the acquired time signals (e.g., by a computational procedure from numerically represented time signals). At least some of the biomarkers characterize a time correlation between signals characterizing multiple different subsystems. The biomarkers are then processed (e.g., again with a computational procedure) to yield a characterization of the potential inflammatory condition from the generated biomarkers.
Aspects can include one or more of the following features.
At least some of the time signals representing an acoustic signal produced via the subject's vocal tract.
The characterization of the potential inflammatory condition is provided to a clinician for providing medical services to the subject.
The subsystems comprise two or more subsystems from a group consisting of a respiratory, a sinus, a laryngeal, an articulatory, a facial and a gastrointestinal subsystem.
Acquiring the time signals comprises acquiring signals of two of more types of signals from a group consisting of acoustic, optical, ultrasound, infrared, accelerometer, and impedance or conductance signals.
An advantage of at least some examples of the approach is that acquiring the biomarkers is not intrusive and the biomarkers are simple to measure, exploiting the coupled nature of the relevant neurophysiological systems, while being specific to the underlying inflammation source. One such system comprises the many sub-systems responsible for vocal production, including the laryngeal and articulatory mechanisms during speaking and/or breathing, and is part of a much larger more complex coordinated physiological and neurological network. For example, the coupling of vocal and facial measures with other neurophysiological measures (e.g., respiration, sinus, gut) provides a sensitive indicator of inflammation, both for early warning and long-term, persistent monitoring of inflammation, of great importance to both military and civilian populations.
Yet other advantages of one or more of the described approaches are as follows:
The analysis of sub-components across multiple modalities provides a means to phenotype sources of inflammation that will help identify possible sub-groups of disease: respiratory, congestive, neurological, physiological, etc. As of now, there are no known methods to objectively identify and monitor such sub-groups.
The biomarkers are sensitive in detecting inflammation during the asymptomatic duration of a condition, an important recent example being respiratory change due to COVID-19.
The biomarkers can potentially discriminate between comorbidities (e.g., neuromotor retardation from depression vs infection/inflammation).
Specificity of identification of (and/or ranking of) physiological systems affected (respiratory, sinus, gastrointestinal, nervous, musculoskeletal, etc.) that may differentiate between aches/pains associated with the cold/flu vs neurological disorders, as well as physiological differentiation.
Quantification of reflex arc response to assess functional impact on behavioral abilities/tasks. Quantification of efficacy of therapeutic interventions such as pharmaceuticals (e.g., anti-inflammatories), physical therapy, cognitive/behavioral therapy, etc.
In the case of COVID-19 the approaches may address public health response gaps including one or more of the following:
1. Pre-symptomatic & asymptomatic infection screening of at-risk health care workers to prevent COVID-19 transmission
2. Distinguishing more common flu or flu-like symptoms from COVID-19
3. Prioritization of diagnostic test kits and medical services to people at high risk of becoming critically ill during self-isolation
4. Assurance that infection has run its course prior to leaving quarantine and returning to health care work
5. Rapid testing for the masses
Other public health impacts may include one or more of the following:
1. Pre-symptomatic feature tracking: monitor presumptive host's physiological response to the disease to cue self-isolation
2. High specificity of vocal subsystem-based biomarkers will allow important phenotyping across conditions
3. Presumptive host symptom tracking during self-isolation: triage decision support for diagnostic testing and close-contact medical attention
4. Post-quarantine recovery assessment: aid decision support for 2nd COVID diagnostic test and release from quarantine
5. Voice and face measurements on mobile technology is ubiquitous making the approach easily accessible
Another advantage is that the markers that are present for pre-symptomatic subjects (i.e., they have a positive COVID diagnostic test but appear to be asymptomatic), and the disclosed monitoring capability allows for also tracking progression with symptoms present and through recovery when symptom subside but the virus is still present (but subsiding). The approach is also amenable to tracking severity of COVID-19 through vaccine trial studies.
In some examples, a subject follows a prescribed data collection protocol, which may be designed to elicit different types of activity thereby sampling the systems more predictably.
Following such a protocol may provide more consistency between subjects, thereby improving the modeling of the data.
Other features and advantages of the invention are apparent from the following description, and from the claims.
An approach to detection, assessing the severity of and/or determining characteristics of an inflammatory process in a subject involves acquiring time series measurements that represent a number of different physiological/neurological subsystems of the subject. Time correlation characteristics of the measured time series are then used as objective biomarkers that are input to a classification or scoring component, which may be based on machine learning principles. Examples of correlation characteristics include eigenspectra of correlation or covariance matrices formed with time delays of time series from different groups of subsystems.
In some embodiments, the objective biomarkers are within and across vocal and speech sub-systems using measurement of other related modalities to detect and monitor the progression of inflammation under the hypothesis that the inflammatory process measurably affects the functional response of the related nervous and musculoskeletal systems and sensorimotor reflex arc dynamics. These objective, noninvasive biomarkers will reflect response dynamics as well as coordination (correlation) of the temporal dynamics of vocal/speech sub-components within a modality and also of vocal/speech sub-components across modalities.
Measurement modalities may include not only acoustic (speech) and optical (face, motion) approaches but also ultrasonic imaging (which could quantify changes in tissue density and thickness at different depths, potentially due to inflammation) and thermal (facial and body) through infrared imaging, electrophysiology through heart rate, skin conductance, EEG measurements, and vision status through eye tracking and pupillometry measurements, and correlations across these modalities and their sub-components. Use of an acoustic/speech modality is not essential, and only a combination of other modalities can be used. Even if an acoustic/speech modality is not used for a time signal, the other modalities may be acquired during the subject speaking (or breathing/coughing in particular ways), for example, speaking passages from a predetermined protocol, but it should also be recognized that such speaking is not required.
Referring to
Continuing to refer to
The analysis system 110 in
Physical indicators of inflammation can present themselves in many ways in speech production and related mechanisms. As an example, nasal and lung congestion occurs as a fluid buildup and swelling not only in the nasal and/or lung cavities, but also the oral tract and vocal fold tissues within the larynx. Though obvious in late stages, there is difficulty in early identification of congestive and other ailments, especially during phases while the subject is asymptomatic but still infectious. This is also important in monitoring progression during and after treatment (i.e., prior to release from quarantine). Nasal and lung congestion is one of many examples where inflammation can affect multiple sub-components of a peripheral motor or a physiological system (in this case, the nasal, oral, respiratory, and laryngeal sub-components).
Referring to
Changes of features both within and across these sub-components can also be obtained. With the underlying inflammation condition, fluid buildup and swelling can also occur in the face, as well as other body parts, effecting fine and gross motor movement and physiological response such as heart rate and skin conductance.
In an exemplary embodiment, the system makes use of a number of low-level features that are measured based on acoustic, or other, signals acquired from the subject. These features are selected based on the physiologically motivated speech production model in
Low-level univariate features characterize basic properties of the three vocal subsystem components. The speech envelope is used as a proxy for respiratory intensity and is estimated using an iterative time-domain signal envelope estimation algorithm, providing a smooth contour derived from amplitude peaks. At the laryngeal level, an estimate of the fundamental frequency (pitch) is determined using an autocorrelation approach and cepstral peak prominence (CPP) is computed, representing stability of vocal fold vibration. CPP is based on the ratio of the pitch-related cepstral peak relative to aspiration noise level. CPP may be computed as the difference in dB between the magnitude of the highest peak and the noise floor in the power cepstrum (the result of taking the inverse Fourier transform of the logarithm of the estimated spectrum of a signal). As a measure of vocal fold stability, CPP has the potential to reflect change in the coupling of subglottal respiratory and laryngeal subsystems. Finally, as a proxy for articulation, a primary feature set comprises the vocal tract resonances (referred to as formant frequencies) estimated by a Kalman filter technique, smoothly tracking the first three formant frequencies while also smoothly coasting through nonspeech regions. Implementations may make use of the open source software package KARMA to compute the first three formant frequencies (F1, F2, F3). These low-level features are computed only during speaking using a speech activity detector. Other features that may be used include the harmonics-to-noise (HNR) ratio, which is the ratio, in decibels (dB), of the power of the harmonic (periodic) signal from vocal-fold vibration and the power of the speech noise signal at the vocal folds created by turbulence as air rushes past the vocal folds from the lungs. This measure is thought to reflect breathiness in a voice. Another measure is termed “creak,” corresponding to what is often referred to as creaky voice, which reflects large irregularity in pitch periods (often with low average pitch) and high peakiness of airflow pulses that excite the vocal tract. The value here is given as a creak probability. Another feature that may be used is the Glottal Open Quotient (GOQ), which is the ratio of the time duration over which the folds are open relative to the full glottal cycle. A larger open quotient often results in more turbulent airflow at the folds. Implementations of various of the feature extraction procedures may use open source software packages, for example, Praat and VoiceBox.
A number of high-level features are derived from low-level features, and are designed to capture coordination of the temporal dynamics of speech production subsystems at different time scales. The high-level features include computation of multivariate auto- and cross-correlations (or covariances) within and across the underlying speech subsystems. We refer to these features as a “correlation structure.” Specifically, time-delay embedding is used to expand the dimensionality of the feature time series, resulting in a correlation matrix with embedded auto- and cross-correlation patterns that represents coupling strengths across feature channels at multiple time delays. The eigenspectrum of the correlation matrix quantifies and summarizes the frequency properties of the set of feature trajectories.
Higher complexity of coordination across multiple channels is reflected in a more uniform distribution of eigenvalues, and more independent ‘modes’ of the underlying system components, while lower complexity is reflected in a larger proportion of the overall signal variability being concentrated in a small number of eigenvalues. In the latter case, the eigenspectra concentration typically manifests with high-rank eigenvalues being lower in amplitude and thus reflecting more dependent or ‘coupled’ system components.
For each speech segment, correlation matrices are calculated from various combinations of feature trajectories from the different speech subsystems. Each matrix contains the correlation coefficients between the time-series at specific time delays to create the embedding space. Each matrix is computed at a specific delay scale (e.g., 10, 30, 70, or 150 ms) with 15 time-delays used per scale. The delay scales allow for characterization of coupling of signals at different time scales. In a preferred embodiment, eigenspectra are computed with a 10 ms delay scale and thus a relatively fine time resolution. Each matrix comparing n signals has a dimension of (n*15×n*15). For all correlations, an automatic masking technique was used to include only voiced speech segments.
In assessing motor coordination, the eigenvalues of the correlation matrix across individual segments that made up the session are averaged to obtain the mean eigenspectra for the entire data collection for a subject (e.g., the extent of an interview with the subject). Under various conditions, different degrees of dynamical complexity both within (e.g., respiration intensity, formant frequencies) and across features (e.g., respiration vs fundamental frequency), are generally found in these features dependent on the condition, as reflected in eigenvalue distributions.
Using the present multi-modal approach, an objective is to detect and monitor COVID-19 with vocal features to characterize coughing and breathing (with or without speech), location of inflammation (e.g., lung versus sinus for specificity), and changes in motor function of speech subsystems due to inflammation. Breathing is characterized not only by the nature and intensity of respiration but also by words per breath group/intensity while speaking (vs simple breathing), respiration in speaking hypothesized to provide a more sensitive measure of breathing abnormalities. Inflammation location may be determined through anti-resonances in vocal tract due to upper (nasal) vs lower (multi-point lung) regions using our multi-modal approach. The effect of inflammation on the complexity of motor coordination of different speech subsystems will be determined.
In one approach to data acquisition, the subject is instructed to produce various sounds, and during the production of those sounds, the various sensor signals (e.g., the acoustic signal acquired by a microphone, possibly augmented by other sensor signals) are acquired. Various instructions may be provided. As one example of a data collection protocol, the subject receives the following instructions.
These instructions are only an example, and other data acquisition protocols may be used. For example, speech produced during a single passage or recording, or during a conversation may be used.
An example of results in the case of COVID-19 are shown in
Although the group-level eigenspectra-based effect size trends indicate a reduced complexity in coordination, clearly a larger cohort is warranted as well as addressing a number of confounders, including subject- and recording-dependences, in any validation procedure. For example, across all variables, inter-subject analysis shows for one subject a distinctly different trend of larger high-ranking eigenvalues that indicates a more complex but more erratic (or variable) coordination. Regarding signal quality, due to the nature of the online video sources, there is a variety of inter-and intra-subject recording variability, the most perceptually notable effect being reverberation, possibly modifying the true effect sizes, over- or under-estimating their importance. An example given in the Supplementary Material, isolating two of the subjects with more consistent, least reverberant environments, enhances the combined effect sizes relative to the N=5 case.
Novel markers derived from such multi-modal sub-systems (acoustics being one example) will provide persistent assessment of a variety of inflammatory diseases but perhaps more importantly the multi-modal nature of high-temporally and spectrally resolved sub-component measurements will provide a means to discriminate location of the inflammation and phenotype the disease. Refinement of these markers will involve physiological and neural computational modeling of the various sub-system components and their modification due to the inflammatory process and the coupling that occurs across sub-systems. Features derived from these models as well as their associated behavioral measures will provide a means to further phenotype to identify which bodily system and which system sub-components are inflamed (e.g., in speech production: nasal, oral, respiratory, or laryngeal sub-components or in facial expression, eye, nose, month, cheeks, jaw), as well as the degree and location of inflammation within a sub-component (e.g., inflammation of vocal cords, sub-glottal tracheal or super-glottal pharyngeal region, or the location of blockage within the nasal, lung, and tracheal passage). The approach will ultimately provide a key capability for longitudinal studies that seek to capture human behavior dynamics in naturalistic environments during onset and progression.
Implementations of the approaches described above may make use of hardware, software, or a combination of hardware and software. Hardware may include data or signal processing circuitry, for example, in the form of Application Specific Integrated Circuits (ASICs) or Field Programmable Gate Arrays (FPGAs). Software may include instructions stored on non-transitory machine-readable media that are executed by a processor. In some examples, the processor is a general purpose computer processor, for example, embedded in a personal computation device, such as a smartphone or mobile computer/tablet. In some examples, the signals are acquired at one device (e.g., a smartphone), while the processing of the signals is performed on a computation system remote from the signal-acquiring device. In some implementations, the processing of the signals is performed in real time, while in other implementations, the signals are recorded, and processed at a later time. In some implementations, rather than providing results to a clinician, other automated and/or machine implemented methods can be used, for example, to triage subjects, route them in a health care facility, or route a communication session (e.g., telephone call) to an appropriate destination.
A number of embodiments of the invention have been described. Nevertheless, it is to be understood that the foregoing description is intended to illustrate and not to limit the scope of the invention, which is defined by the scope of the following claims. Accordingly, other embodiments are also within the scope of the following claims. For example, various modifications may be made without departing from the scope of the invention. Additionally, some of the steps described above may be order independent, and thus can be performed in an order different from that described.
This application claims the benefit of U.S. Provisional Application No. 63/007,836, filed on Apr. 9, 2020, which is incorporated herein by reference.
This invention was made with Government support under Contract No. FA8702-15-D-0001 awarded by the U.S. Air Force. The Government has certain rights in the invention.
| Number | Date | Country | |
|---|---|---|---|
| 63007836 | Apr 2020 | US |
