Patent Application
20220361759
The described embodiments relate generally to the determination of human heart rate and human breathing rate using smartphone accelerometers, and to related structures and methods.
There are many cellular telephone-based or smartphone-based heart rate monitors. All of these devices have shortcomings and problems. For example, many require the use of an additional sensor or wearable device. Some require the user to wear a finger monitor, a wrist strap, a watch, or to press on a sensor. The sensor might be a sensor that is connected to the smartphone via wires, or might be a wireless sensor. Some require the user to press a finger, or multiple fingers, onto the touchscreen of the smartphone. Some even attempt to detect physiological functions by processing images of the subject taken by the camera of the smartphone. In contrast to these systems, a easy-to-use, reliable, inexpensive and accurate smartphone-based heart rate and breathing rate monitoring system is desired that involves no sensors other than the accelerometers of a smartphone.
U.S. Ser. No. 10/285,650B2 (Heart monitoring device and method) describes a set of processing steps to filter the accelerometer signal of a hypothetical wearable device and to extract the estimated heart rate via ballistocardiography (i.e., the mechanical component of the pulsating blood in the body). Time-domain analysis of the signal (e.g., peak detection) is used, and the device does not extract breathing information. Breathing rate information is considered an artifact to be removed.
U.S. Pat. No. 10,674,424B2 (Method and apparatus for measuring physiological parameters) describes a set of distributed sensors comprising accelerometers and gyroscopes configured to infer heart rate, breathing rate and heart rate variability from the user's wrist and head. A computer-based method dynamically weights the contribution of each signal depending on user characteristics and use case. Weighting of monitored signals are not weighted over individual output values, and the method does not use intrinsic spectral properties of the signal.
U.S. Pat. No. 10,617,308B2 (Device and method for estimating the heart rate during motion) describes a portable device and method to estimate the heart rate of a user in motion. This is done by combining a standard ECG sensor (prone to motion artifacts) with a motion sensor. When the quality of the ECG reading drops below a certain threshold, a heart rate estimate is inferred from user motion (e.g., heart rate proportional to amount of motion). The method does not establish a threshold of accuracy below which the importance of a given sample is proportionally reduced. Rather, the method switches to use a different estimation method.
EP3318179A2 describes a method for measuring respiration rate and heart rate using dual camera of smartphone.
Biofeedback devices generally involve methods and interfaces to provide explicit feedback to the user about her physiological state by mapping physiological parameters to changes in the interface. For example, US20140316191A1 (Biofeedback Virtual Reality Sleep Assistant) describes a virtual reality system comprising physiological sensors and a head mounted display. The system maps the bio signal for sleep tracking (e.g., heart rate, breathing rate) to images on the screen with the goal of dynamically favoring sleep. The system does not use audio signals to guide the user during a breathing exercise for relaxation (with the goal of reducing stress), nor does it use biofeedback to display the performance of the user on the screen (e.g., how well the user followed the breathing rhythm).
WO2006113900A2 (Methods and devices for relieving stress) describes a portable device equipped with a PPG sensor able to measure respiration patterns of the user. The breathing wave pattern is displayed to the user on a screen in real time to help follow a breathing pattern with the goal of reducing stress. The method displays waveforms and not determined rates, and does so in real time and not at the end of an exercise, nor are both breathing and heart information displayed.
The conference paper entitled “Beat-to-Beat Heart Rate Detection By Smartphone's Accelerometers: Validation With ECG,” by Federica Landreani et al., Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society (2016), considers the possibility and the feasibility of measuring beat-to-beat heart cycle during using peak detection time domain signal processing of the x, y and z orthogonal axis seismocardiographic (SCG) accelerometric signals from an Apple iPhone6 smartphone in order to determine heart rate. Fiducial point detection is used as described in the paper entitled “Heart Rate Variability Analysis Using A Seismocardiogram Signal,” by J. Ramos-Castro et al., Conference Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 5642-5645 (2012). Offline processing on a remote computer employs noise-sensitive peak detection for the detection of heart rate. The method does not involve accuracy estimation for adjustment of detected heart rate values, nor does it involve the determination of breathing rate. The offline peak-detection method is so sensitive to noise that the method can only work well in the absence of noise and cannot be effectively employed in real time entirely on a typical smartphone.
A smartphone exercise app and an associated heart rate (HR) and breathing rate (BR) plugin provide biofeedback to a smartphone user in the context of a smartphone-based relaxation exercise to reduce mental stress. The system employs a processing pipeline that combines inference biosignals (for example, heart rate and breathing rate) from the 3-axis accelerometer of the smartphone with an accuracy estimation method that enhances robustness of the estimate. More specifically, predetermined spectral entropy to weight mapping information is stored on the smartphone. This information is stored on the smartphone before the user uses the smartphone exercise app to determine heart rate and breathing rate. There is one such mapping for each of a plurality of use cases. For example, there is a first mapping for a use case when a heart rate (as opposed to breathing rate) is to be detected, when a user holding the smartphone is in a sitting position (as opposed to the smartphone resting on the user's chest when the user is in the supine position), and when the smartphone is detected to a first type of smartphone (as opposed to a second type of smartphone). There is a second mapping for a use case when breathing rate is to be detected, when the user is holding the smartphone in the sitting position, and when the smartphone is detected to be of the first type.
The exercise app starts and uses the 3-axis accelerometer of the smartphone to generate a plurality of accelerometer output data sets. Each accelerometer output data set contains a sequence of accelerometer output samples that are captured by the accelerometer over a period of time (for example, over a 20.48 second period of time). Each accelerometer output sample includes an x-axis accelerometer output value, a y-axis accelerometer output value, a z-axis accelerometer output value, and a timestamp. In one example, accelerometer output samples are received from the accelerometer at a rate of one hundred output samples per second. In one example, one accelerometer output data set of 2048 output samples is generated each two seconds, with the output samples in the data set representing data captured over a 20.48 second period of time.
From each set of x-axis, y-axis and z-axis accelerometer output data sets, a single ballistocardiogram (BCG) data set is generated. Frequency domain processing is performed on the single BCG data set thereby generating an estimated heart rate (EHR) value. Accordingly, for each successive accelerometer output data set a corresponding estimated heart rate (EHR) value is generated.
Next, from each successive accelerometer output data set, a corresponding estimated accuracy (EA) value is determined, thereby determining a sequence of EA values. An EA value is determined using: 1) a measure of the power spectral entropy of the output samples in the accelerometer output data set, as well as 2) the appropriate spectral entropy to weight mapping previously stored on the smartphone. From the EA value, an EHR weight value is determined. Accordingly, there is one such EHR weight value determined for each EHR value.
The EHR weight values are then used to adjust the sequence of EHR values, thereby generating a sequence of adjusted heart rate (AHR) values. This use of EHR weights allows EHR values that are estimated to be less accurate to have less effect on the AHR values than HER values that are estimated to be more accurate. Accuracy of the AHR values as compared to actual heart rate is improved by use of the spectral entropy to weight mapping information and the determined estimated accuracy values of the EHR values.
In addition to determining heart rate, an estimated breathing rate (EBR) value is determined for each accelerometer output data set. Frequency domain signal processing is performed on each of the x-axis accelerometer output data set, the y-axis accelerometer output data set, and the z-axis accelerometer output data set. In one example, the EBR value is determined to be the frequency of the Fast Fourier Transform (FFT) component having the maximum amplitude of the most periodic of the x-axis, y-axis and z-axis data sets.
Next, for each successive accelerometer output data set, a corresponding estimated accuracy (EA) value is determined, thereby determining a sequence of EA values. An EA value is determined using: 1) a measure of the power spectral entropy of the data in the accelerometer output data set, as well as 2) the spectral entropy to weight mapping information previously stored on the smartphone. From the EA value, a weight value is determined. Accordingly, there is one such weight value determined for each EBR value. The weight values are then used to adjust the sequence of EBR values, thereby generating a sequence of adjusted heart rate (ABR) values. Accuracy of the ABR values as compared to actual breathing rate is improved by use of the spectral entropy to weight mapping information and the determined estimated accuracy values of the EBR values.
The determination of the EHR and EBR values occurs entirely on the smartphone, without any of the user's potentially sensitive biosignal information being transmitted or communicated from the smartphone to another device. The only sensor involved is the smartphone's 3-axis accelerometer, and the user does not have to wear or interact with any external additional sensor or measurement device. The user needs merely to hold the smartphone in the user's hand, or to place the smartphone on the user's chest when lying in a face-upward supine position.
In one embodiment, the AHR and ABR values determined by the plugin is used by the exercise app to provide customized feedback to the user o how the user is doing in the overall relaxation exercise. For example, the exercise app causes breathing rhythm prompts to be communicated to the user based on the user's real-time detected heart rate and/or breathing rate. After a period of time of the user's using the exercise app, final results of the relaxation exercise are displayed to the user on the smartphone touchscreen and the final results are logged on the smartphone for future reference.
Further details and embodiments and methods and techniques are described in the detailed description below. This summary does not purport to define the invention. The invention is defined by the claims.
The accompanying drawings, where like numerals indicate like components, illustrate embodiments of the invention.
Reference will now be made in detail to some embodiments of the invention, examples of which are illustrated in the accompanying drawings.
In response to the user 22 pressing an icon on the touchscreen of the smartphone, the exercise app 5 starts (step 102) which in turns starts the HR and BR plugin 15. The plugin 15 via the iOS bridge 16 makes a request of the operating system 2 (via an iOS function call “startaccelerometerupdatestoqueue”) to start a subscription to send accelerometer data back to the plugin. The plugin 15 receives the accelerometer data through the iOS bridge 16 at a rate of one hundred accelerometer samples per second. Each accelerometer sample includes an accelerometer x-axis data value, an accelerometer y-axis data value, an accelerometer z-axis data value, and a time stamp. The accelerometer data values are double precision floating point numbers, and as collected data range from a low value of 0.0 g to a high value of about 7.7 g, and have a resolution of approximately 0.018 g. The heart rate and breathing rate plugin main program 18 receives the incoming stream of accelerometer output samples via the iOS bridge 16 and pushes them sequentially into a circular buffer data structure 21. The circular buffer 21 has 2048 buffer storage locations, each of which can store one accelerometer output sample, where an accelerometer output sample includes an x-axis accelerometer value, a y-axis accelerometer value, a z-axis accelerometer value, and a timestamp. As accelerometer output samples are stored into the circular buffer 21, a head pointer PTR is incremented so that the head pointer PTR always points to the last filled buffer storage location. A tail pointer need not be maintained because the number of buffer slots in the circular buffer (2048 buffer storage locations) is known.
Next, the exercise app main program 12 consults the program structure to determine which of the slides will be shown next and proceeds by making an internal function call to present the correct slide 13 (step 103) to the user 22 on the smartphone touchscreen 9 along with a selectable set of options. One of these user selectable options is a relaxation exercise. In the present example, the user 22 presses on the touchscreen selection for the relaxation exercise. The exercise app 5 gets a callback from the operating system 2 indicating to the exercise app 5 that the user has pressed the touchscreen to make this selection. This causes the slide implementation handler 13 to send a slide selected event to the exercise app main program 12 (step 104) thereby alerting the exercise main program 12 of the user's selection of the relaxation exercise. In response to the user's selection of the relaxation exercise, the exercise app main program 12 makes an internal function call to cause another of the slides 13 to be presented (step 105) to the user 2 on the smartphone touchscreen 9 instructing the user to assume a proper sitting position for the relaxation exercise to be performed.
Next, in response to the instruction, the user 22 assumes the proper sitting position.
Next, the HR and BR plugin 15 selects one of the eight stored spectral entropy to weight mappings 20 (step 106) to be the spectral entropy to weight mapping for heart rate (SPMHR) to be used in the selected relaxation exercise. Which spectral entropy to weight mapping is selected is based on detected use parameters. In the present example, a first use parameter is whether heart rate is to be determined or whether breathing rate is to be determined. A second use parameter is whether the user will be performing the exercise in the sitting position with the user sitting on a chair (with the smartphone in the user's left hand), or in the supine position (user lying on the user's back, face upward, with the smartphone lying on the user's chest). A third use parameter is whether the smartphone is of a first type (for example, an iPhone type) or a second type (for example, an android type). For this parameter, all smartphones are classified as belonging to one of these two general groups. There are eight possible combinations of these three use binary parameters, so eight spectral entropy to weight mappings are stored. One of these mappings is selected to be the spectral entropy to weight mappings for heart rate (SPMHR). In addition, another of the stored spectral entropy to weight mappings is selected (step 106) to be the spectral entropy to weight mapping for breathing rate (SPMBR). Which spectral entropy to weight mapping is selected to be the SPMBR is based on the detected use parameters.
Accelerometer output samples are being stored into the circular buffer 21 at a continuous rate of one hundred accelerometer output samples per second, and the circular buffer 21 at any given time stores the last 2048 accelerometer output samples. The heart rate and breathing rate plugin main program 18, however, reads out the present circular buffer 21 contents (2048 output samples) each two minutes. One such set of output sample values is referred to as an “accelerometer output data set”. Because the circular buffer stores 2048 accelerometer output samples which represents accelerometer output samples collected over one 20.48 second time period, only 200 of the accelerometer output samples in each successive accelerometer output data set are newly stored accelerometer output samples. The time periods corresponding to successive accelerometer output data sets overlap one another by 1848 samples. The accelerometer output data sets of the stream of accelerometer output data sets that are output in this way are downsampled by two (every other accelerometer output sample is discarded) so that each accelerometer output data set consists of 1024 accelerometer output samples. In this way, the following are generated (step 107): 1) a stream of x-axis accelerometer output data sets where each x-axis accelerometer output data set is 1024 x-axis accelerometer output data values, 2) a stream of y-axis accelerometer output data sets where each y-axis accelerometer output data set is 1024 y-axis accelerometer output data values, 3) a set of z-axis accelerometer output data sets is generated where each z-axis accelerometer output data set is 1024 z-axis accelerometer output data values, and 4) a stream of timestamps.
For the time period of each “accelerometer output data set”, the heart rate and breathing rate plugin main program 18 determines (step 108) an estimated heart rate (EHR) value, thereby determining a sequence of EHR values. Also, for each time period of each “accelerometer output data set”, the heart rate and breathing rate plugin main program 18 determines (step 108) an estimated breathing rate (EBR) value, thereby determining a sequence of EBR values.
Determination of an EHR value: The heart rate and breathing rate plugin main program 18 determines one EHR value from each corresponding “accelerometer output data set”. From each of the x-axis accelerometer output data values of the accelerometer output data set, the y-axis accelerometer output data values of the accelerometer output data set, and the z-axis accelerometer output data values of the accelerometer output data set, a time-domain moving average is determined (e.g., with a window size corresponding to a duration of about 0.1 seconds). As the values of this time domain moving average are determined, each successive time domain average is subtracted from the corresponding accelerometer output data value. This reduces the non-stationarity of the accelerometer output data values.
A Butterworth low-pass filter (e.g., with cut-off frequency of 7.0 Hz, and order 1) is applied to each of the three resulting streams of values (three “signals”). The Butterworth low-pass filter is applied again to the reversed version of each resulting signal.
For each of the resulting three signals, the mean determined. From each signal value the mean is subtracted, and each resulting value is divided by its standard deviation. This results in a sequence of modified x-axis values, a sequence of modified y-axis values, and a sequence of modified z-axis values.
The L2 norm of each associated set of three data values (an x-axis value of the modified x-axis values, a y-axis value of the modified y-axis values, and a z-axis value of the modified z-axis values) is determined. There are 1024 such sets. The L2 norm is determined by taking the square root of the sum of the square of the three associated values. This results in a single sequence of 1024 values known as a BCG (or, ballistocardiogram).
A Butterworth band-pass filter (e.g., with low and high cut-off frequencies of 0.6 Hz and 2.7 Hz respectively, and an order of one) is then applied to this BCG, and is applied again to the reversed resulting signal. A Fast Fourier Transform (FFT) is applied to the resulting signal. A moving average is applied to the resulting transform outputs (i.e., a moving average (with a window of five) of the magnitudes of the frequency components in the frequency domain). The frequency component of the FFT output that has the maximum amplitude is determined. The frequency of this component is determined to be the estimated heart rate (EHR) value.
Determination of an EBR value: The heart rate and breathing rate plugin main program 18 determines one EBR value from each corresponding 1024-value “accelerometer output data set”. From each of the x-axis accelerometer output data values of the accelerometer output data set, the y-axis accelerometer output data values of the accelerometer output data set, and the z-axis accelerometer output data values of the accelerometer output data set, the following is done. The mean is subtracted from each output data value, and the result is divided by its standard deviation. A Butterworth band-pass filter (e.g., with low and high cut-off frequencies of 0.07 Hz and 1.6 Hz respectively, and order one) is applied, and is applied again to the reversed resulting signal. A Fast Fourier Transform (FFT) is performed on the resulting signal, thereby generating a set of frequency components each having an amplitude. This process is performed for each of the x-axis accelerometer output data values, the y-axis accelerometer output data values, and the z-axis output data values, thereby generating three resulting FFT output signals. The most periodic one of these three signals is defined to be the signal (either the x-axis derived signal, the y-axis derived signal, or the z-axis derived signal) that has the frequency component of the greatest value. The frequency corresponding to this frequency component of the greatest value is determined to be the estimated breathing rate (EBR) value.
In the manner described above, both an EHR value and an EBR value are determined for the time period of each “accelerometer output data set”, thereby generating sequence of EHR values and a sequence of EBR values (step 107).
For each EHR value, the heart rate and breathing rate plugin main program 18 determines one EHR weight value (step 108) from the corresponding “accelerometer output data set”. The power spectral entropy (PSE) is then computed, and the PSE is computed differently for HR and BR purposes. In the case of HR detection, the power spectral entropy of the BSG signal is determined, whereas in the case of BR the PSE of the most periodic signal amount the three axes is determined. In the present example where heart rate is being determined, the power spectrum X(wi) of the signal is determined first, using a Fast Fourier Transform (FFT) for discrete frequencies wi between zero and half of the sampling frequency. The normalized Power Spectral Density P(wi) is given by equation (1) below:
where N is the number of frequency bins. P(ωi) is then mapped to a probability density function p according to equation (2) below:
The PSE value is given by equation (3) below:
A normalized PSE value is determined by dividing the PSE value by ln(N). Once the normalized PSE value is determined, an estimated accuracy (EA) value is determined according to equation (4) below:
where the input variable x is the normalized PSE value. The parameters L, k and x0 are constants that define a relationship between an input value of x (the determined normalized PSE value) and the output estimated accuracy EA value being determined. The set of the three parameters L, k and x0 is determined from lookup table 26 from the input detected use parameters mentioned in connection with step 106. Note the illustration of the lookup table 26 in
where t is an accuracy threshold parameter, and m is a below-threshold decay parameter. There is one such EHR weight value w(EA) determined (step 109) for each separate EHR value. Importantly, the relation of equation (5) allows more importance to be given to estimated EHR values that have higher measured confidence. The same process is repeated using each EBR value so that one EBR weight value w(EA) is determined (step 109) for each EBR value.
Next, the EHR weights are used in a moving window to adjust each consecutive EHR value to obtain an adjusted heart rate (AHR) value (step 110). The five EHR weights within a moving window of five weights are L−1 normalized to determine five normalized weights. Each normalized weight within the window is multiplied by its corresponding EHR value, and the five resulting products are summed in order to determine one adjusted EHR value for the window. The window is then shifted five weight values in time, and this is how the downsampling with a stride of ten seconds is accomplished. The five EHR weights within the new window are L−1 normalized to determine another five normalized weights, and those five normalized weights are used to determine another adjust EHR value. This process is repeated as the window is shifted through all the 1024 EHR values so that an adjusted AHR value is determined for each of the original EHR values. In the same way that the EHR weights are used to adjust the corresponding 1024 EHR values to obtain 1024 AHR values, the same process is repeated using EBR weights to adjust the 1024 EBR values in order to obtain 1024 ABR values (step 110).
The heart rate and breathing rate plugin main program 18 then downsamples (with a stride of ten seconds) the sequence of 1024 AHR values so there is one AHR value generated each ten seconds. Likewise, the sequence of 1024 BR values is downsampled (with a stride of ten seconds) so there is one ABR value generated each ten seconds.
The heart rate and breathing rate plugin main program 18 then assigns a Boolean accuracy value to each downsampled AHR value. In the present example, the Boolean accuracy value will be positive (True) if the percentage of confidence values, within the relative moving window, is greater than a certain accuracy threshold percentage (for example, 33%). The Boolean accuracy value will be negative (False) otherwise.
The CBT exercise app main program 12 sends a request to the plugin main program 18 via the iOS bridge 16 instructing the plugin main program 18 to return the AHR and ABR results. The CBT exercise app main program 12 receives the AHR and ABR results and causes the AHR and ABR results to be presented (step 111) to the user 22 via the touchscreen of the smartphone.
In the event that there are more than a predetermined number (for example two) of consecutive negative downsampled AHR values with a negative Boolean accuracy (Boolean accuracy value is False), then the AHR values are not displayed, but rather these negative accuracy measures are replaced with values determined by linear interpolation between the nearest positive Boolean accuracy AHR values (Boolean accuracy value is True) to the immediate left and immediate right of the consecutive negative values. These AHR dots are, however, still displayed in orange. For example, in
The CBT exercise app main program 12 calls a function in the music player plugin 29 to play one of the breathing rhythm audio prompt snippets of the audio snippets 14. The audio snippet is played with a controlled rhythm so as to lead the user 22 to breath in a more relaxed fashion.
If it is decided that the relaxation exercise is not yet complete (step 112), then processing proceeds back to step 107 to process another 20.48 seconds worth of accelerometer output samples. If it is decided that the relaxation exercise is complete (step 112), then the CBT exercise app main program 12 causes results of the relaxation exercise to the displayed (step 113) on the smartphone display.
Determination of the Spectral Entropy to Weight Mappings: The spectral entropy to weight mappings referred to in step 101 of the flowchart of
Accuracy Of The AHR and ABR Determinations: A cardiograph is a device that generates electrocardiograms. An electrocardiogram (ECG) is a graph of voltage versus time of the electrical activity of the heart using electrodes placed on the skin of a subject around the heart. These electrodes detect the small electrical changes that are a consequence of cardiac muscle depolarization followed by repolarization during each heartbeat. There are three main components to an ECG: the P wave, which represents the depolarization of the atria; the QRS complex, which represents the rapid depolarization of the right and left ventricles; and the T wave, which represents the repolarization of the ventricles. The inter-beat interval (RR) is the time elapsed between the beginnings of two successive R-waves. This RR time interval is converted into a heart beats per second (HR) value. For purposes of determining the accuracy of the AHR values, the heart rate value determined from the ECG device is considered the gold standard.
Accuracy of the AHR as determined by the method of the flowchart of
A respiration rate monitor is a device used to measure the breathing rate of a subject. One type of suitable respiration rate monitor, which allows monitoring the respiration rate, is a piezoelectric transducer (PZT) respiration sensor, that measures displacement variations induced by inhaling or exhaling. A PZT respiration sensor model includes the Biosignalsplux model PZT01102020, 1 Hertz bandwidth, 106 dB common mode rejection ratio (CMRR), 16-bit channel, sensor available from Plux-Wireless Biosignals, S.A., of Lisbon, Portugal. The data generated is measured as “displacement value in percentage of full scale”. For purposes of determining the accuracy of the ABR values, the breathing rate value determined from the respiration rate monitor is considered the gold standard. Accuracy of the ABR as determined by the method of the flowchart of
Although certain specific embodiments are described above for instructional purposes, the teachings of this patent document have general applicability and are not limited to the specific embodiments described above. Accordingly, various modifications, adaptations, and combinations of various features of the described embodiments can be practiced without departing from the scope of the invention as set forth in the claims.
