Patent Application
20160007935
The present invention relates generally to dynamic weighting of data from a set of sensors, such that weighting changes over time and depends on at least (i) type of sensor and (ii) sensor position relative to a user's body.
In illustrative implementations, a sensor system includes one or more accelerometers, gyroscopes and optical sensors. For example, in some cases, the optical sensors comprise photoplethysmographic (PPG) sensors. The gyroscopes and accelerometers measure subtle body movements caused by heartbeats and breathing. The PPG sensors measure blood volume pulse and other subtle body changes caused by heartbeats and breathing.
The sensor system (a) takes sensor measurements of a human user; (b) based on the sensor measurements, calculates a cardiac waveform and a respiratory waveform; (c) based on the waveforms, calculates heart rate, breathing rate and other physiological parameters, such as heart rate variability; and (d) based on the waveforms, makes a biometric identification of the user. In some implementations, the sensor system also performs feature analysis to extract other information from the waveforms, such as posture, gender, weight and age of the user.
In illustrative implementations, the sensor system is dynamically adaptable. Among other things, the sensor system: (a) dynamically adjusts the weight given to data measured by different types of sensors (accelerometer, gyroscope, optical sensor) or (b) dynamically adjusts the weight given to data gathered at different body locations (e.g., wrist-worn, head-mounted, or handheld).
In illustrative implementations, this dynamic adjustment of weighting depends on one or more of the following trigger factors: quality of data, posture of user (e.g., standing, sitting, or lying down), identity of user, gender of user, weight of user, age of user, activity of user (e.g., reading, listening to music, talking on a phone, or browsing the Internet), availability of data, and purpose for which data is being used (e.g., whether the data is being used to calculate posture, biometric identification, heart rate, or respiratory rate).
This dynamic adjustment is advantageous because which type of sensor (e.g., accelerometer, gyroscope, or PPG sensor) or combination of types of sensors yields the most accurate estimate of a physiological parameter (or yields the most accurate biometric identification) varies under different conditions. These different conditions include: (a) the position of the sensor relative to the user's body (e.g., head-mounted, wrist worn, or carried in a pocket of the user); and (b) the trigger factors mentioned in the preceding paragraph. By dynamically adjusting the weight of data from different sensors, the sensor system is able to achieve more accurate estimates.
For example, in some cases, data from a wrist-mounted gyroscope yields a more accurate estimate of breathing rate than either (i) data from a wrist-mounted accelerometer, or (ii) data from the two sensors combined. Also, for example, in some cases, the accuracy of an estimate of heart rate or breathing rate based on data from a given sensor type (e.g., accelerometer, gyroscope, or PPG sensor) varies dramatically depending on the user's posture (e.g., whether the user is standing, sitting or supine). Similarly, in some cases, the accuracy of a biometric identification based on sensor data from an accelerometer, gyroscope or PPG sensor varies sharply, depending on the user's posture.
In some cases, the sensors include a video camera that measures motion of the user relative to a scene. In some cases, the camera is located in a fixed position in the user's surrounding and captures images of the user, from which images motion of the user is measured. In some other cases, the camera is mounted on the user and captures images of the user's surrounding, from which motion of the user is measured (by assuming that the scene captured in the image is static). Data from the video camera is also dynamically weighted, depending on factor such as (in the case of a camera worn or mounted on a user), where the camera is mounted or worn on the user.
The description of the present invention in the Summary and Abstract sections hereof is just a summary. It is intended only to give a general introduction to some illustrative implementations of this invention. It does not describe all of the details and variations of this invention. Likewise, the descriptions of this invention in the Field of Technology section and Field Of Endeavor section are not limiting; instead they each identify, in a general, non-exclusive manner, a technology to which exemplary implementations of this invention generally relate. Likewise, the Title of this document does not limit the invention in any way; instead the Title is merely a general, non-exclusive way of referring to this invention. This invention may be implemented in many other ways.
In
In
In
In
In
In
In
The above Figures show some illustrative implementations of this invention, or provide information that relates to those implementations. However, this invention may be implemented in many other ways.
In
In the example shown in
In the example shown in
Each wireless transceiver unit (e.g., 205, 215, 225, 235) includes (a) one or more antennas, (b) one or more wireless transceivers, transmitters or receivers, and (c) signal processing circuitry. Each wireless transceiver unit receives and transmits data in accordance with one or more wireless standards.
In the examples shown in
A wide variety of cameras may be used. For example, in some cases, the camera 310 is a video camera. In some cases, the camera is a depth-sensing camera, including a depth-sensing video camera. In other cases, the camera is an infra-red camera, including an infra-red video camera.
A wide variety of support structures may be used.
In the example shown in
More generally, the support structure comprises any headwear, including: (a) any hat, cap, helmet, eyeglasses frame, sunglasses frame, visor, headband, crown, diadem, or head-mounted display, or (b) any structure (including any strap, band, frame, ring, post, scarf, or other item of apparel) that is worn at least partially on or supported at least partially by the skin, hair, nose or ears of a human head or that at least partially surrounds or indirectly rests upon a human neurocranium. However, the term “headwear” does not include any part of a human being.
In the example shown in
In the example shown in
In some cases, the wrist-mounted sensor module is housed in a smartwatch, which includes a watch band worn around a user's wrist. Alternatively, the wrist-mounted sensor module is housed or attached to an elastic wristband or to a wristband that has an adjustable inner diameter. For example, in some cases, the inner diameter of the wristband may be adjusted by inserting a metal prong attached to one end of the wristband into one of a series of holes at the other end of the wrist band.
In some cases, the wrist-mounted sensor module includes a PPG sensor. The PPG sensor includes an LED (light emitting diode) that illuminates a user's skin, and a photodiode that measures the amount of light that is transmitted through or reflected from the user's skin.
A computer (e.g., computer 236) uses sensor data, or data derived therefrom, to calculate HR, HRV and BR. Alternatively, a computer (e.g., 206, 216, 226) onboard a sensor module calculates HR, HRV and BR.
In
In illustrative implementations, a multi-sensor wireless communications network collects data from sensors worn or carried at different locations relative to the user's body to determine the best estimate of the heart-rate or respiration. In addition to, or instead of, one or more of the locations discussed above (head, wrist, pocket), the sensor modules may be worn on the lower calf, worn above the ankle, clipped onto a belt, or worn or carried anywhere else on the body. Each of the sensor modules includes one or more sensors (such as a 3-axis digital accelerometer, a 3-axis digital gyroscope and a PPG sensor) for detecting blood volume pulse or subtle bodily motion caused by heartbeat or respiration.
Calculating Pulse Waveform from Motion Sensor Readings
In some cases, measurements taken by a 3-axis motion sensor (e.g., a 3-axis accelerometer or 3-axis gyroscope) are used to compute a cardiac pulse waveform as follows: First, represent each sample taken by the 3-axis motion sensor as a 3D vector, where the first, second and third elements of the vector are x-axis, y-axis and z-axis measurements, respectively, for that sample. A time sequence of samples by the 3-axis motion sensor is represented by a time series of 3D vectors or, equivalently, by a matrix in which the first, second and third rows represent x-axis, y-axis and z-axis measurements, respectively, and each column represents a sample at a particular time. In this matrix, each row represents a time sequence of readings taken by a particular axis (x, y or z) of the motion sensor. Second, for each row, subtract a moving average window from each entry in the row to detrend readings and to filter artifacts caused by sudden, large movements. The window size depends on the sampling rate of the motion sensor. For example, in some cases: (a) the sampling rate of the motion sensor is 256 Hz; (b) the window size is 15 samples; (c) for each given entry in a row, the values of the given entry and the surrounding 14 entries (7 before and 7 after) in the row are averaged, then the average is subtracted from the given entry. Near the beginning and end of a row, if the number of subsequent entries in the row is less than the window size divided by two, then the window size is reduced in one of the sides. Third, z-score each row of the matrix to have zero mean and unit variance, so that the axes have the same relevance and the analysis is more robust to changing orientations of the motion sensor. Fourth, apply a band-pass Butterworth filter (e.g. with cut-off frequencies of 7 Hz and 13 Hz) to each axis (row of the matrix). Fifth, in order to aggregate the different components (i.e., axes of the motion sensor), compute the square root of the summation of the squared components (i.e., L2 norm) at each sample. The output of this aggregation is a 1D vector, where each entry in the 1D vector represents an aggregated value for all three axes for a given sample. Sixth, apply another Butterworth bandpass filter (e.g., with cut-off frequencies of 0.66 Hz and 2.50 Hz), in order to compute the pulse waveform.
In illustrative implementations, the Butterworth filters (e.g., used in computing pulse waveforms or respiratory waveforms from accelerometer, gyroscope or PPG sensor readings) are applied computationally. The order of the Butterworth filters may vary, for example from first order to fourth order. In many cases, a higher order (e.g., 4) Butterworth is desirable for better filtering. However, in some cases, a lower order Butterworth filter (e.g., first or second order) is desirable because of a low signal amplitude or in order to simplify computation.
Calculating Respiratory Waveform from Motion Sensor Readings
In some cases, measurements taken by a 3-axis motion sensor (e.g., a 3-axis accelerometer or 3-axis gyroscope) are used to compute a respiratory waveform as follows: First, represent each sample taken by the 3-axis motion sensor as a 3D vector, where the first, second and third elements of the vector are x-axis, y-axis and z-axis measurements, respectively, for that sample. A time sequence of samples by the 3-axis motion sensor is represented by a time series of 3D vectors or, equivalently, by a matrix in which the first, second and third rows represent x-axis, y-axis and z-axis measurements, respectively, and each column represents a sample at a particular time. In this matrix, each row represents a time sequence of readings taken by a particular axis (x, y or z) of the motion sensor. Second, for each row, subtract a moving average window (e.g., with a window size of 8.5 seconds) from each entry in the row. Near the beginning and end of a row, if the number of subsequent entries in the row is less than the window size divided by two, then the window size is reduced. Third, z-score each axis (row of the matrix) to have zero mean and unit variance. Fourth, apply Independent Components Analysis (ICA) to each axis (row of the matrix) to determine independent components of the axis and remove sources of artifact. Fifth, to isolate movements caused by respiration, apply a band-pass Butterworth filter (e.g. with cut-off frequencies of 0.13 Hz and 0.66 Hz) to each independent component. Sixth, calculate Fast Fourier Transforms (FFTs) for each independent component of each axis. Seventh, determine which FFT (out of the FFTs of independent components of all of the axes) is the most periodic—i.e., which FFT has the maximum magnitude. Seventh, select the independent component that has the most periodic FFT within a specific frequency range (e.g. 0.13 Hz and 0.66 Hz), and output it as the respiratory waveform (or output its FFT as the FFT of the respiratory waveform). Alternatively: (a) in the fourth step, use Principal Components Analysis (PCA) instead of ICA; and (b) substitute principal components for independent components in the remainder of the above steps.
Calculating Pulse and Respiratory Waveforms from Other Sensor Readings
In some cases: (a) a sensor module includes a PPG sensor; and (b) readings from the PPG sensor are processed and filtered in order to extract pulse and respiratory waveforms. Likewise, in some cases: (b) a sensor module includes a video camera or other sensor; and (b) readings from the camera or other sensor are processed and filtered in order to extract pulse and respiratory waveforms.
Extracting Physiological Parameters from Pulse and Respiratory Waveforms
In illustrative implementations, a computer extracts heart rate from the pulse waveform. To do so, a computer computes FFTs of the pulse waveform and analyzes the frequency response in a range from 0.66 Hz-2.5 Hz (corresponding to 45 to 150 beats per minute). The computer identifies the frequency component in that range that has the highest amplitude, and then multiplies that frequency (which is expressed in Hertz) by 60 seconds/minute, in order to output a heart rate expressed as beats per minute.
In illustrative implementations, a computer extracts breathing rate form the respiratory waveform. To do so, a computer computes a FFT of the respiratory waveform (or, if the FFT of the respiratory waveform was computed in an earlier step, uses that FFT). The computer analyzes the frequency response in a range from 0.13 Hz to 0.66 Hz (corresponding to 8 to 45 breaths per minute). The computer identifies the frequency component in that range that has the highest amplitude, and then multiplies that frequency (which is expressed in Hertz) by 60 seconds/minute, in order to output a breathing rate expressed as breaths per minute.
The sampling rate of the motion sensors (and any other sensors used to calculate a physiological parameter) may vary, depending on the particular implementation or use scenario. In some cases, sensor data is stored for long periods onboard a sensor module before being transmitted for further analysis, and thus a low sampling rate (e.g., between 20 Hz and 50 Hz) is desirable. In some other cases, a higher sampling rate (e.g., between 100 Hz and 256 Hz, or higher than 256 Hz) is desirable, such as for calculating heart rate variability, for which a high temporal resolution is advantageous.
In illustrative implementations, a sensor system includes one or more accelerometers, gyroscopes and optical sensors. The sensor system (a) takes sensor measurements of a human user; (b) based on the sensor measurements, calculates a cardiac waveform and a respiratory waveform; (c) based on the waveforms, calculates heart rate, breathing rate and other physiological parameters, such as heart rate variability; and (d) based on the waveforms, makes biometric identifications of humans. In some implementations, the sensor system also performs feature analysis to extract other information from the waveforms, such as posture, gender, weight and age of the user.
In illustrative implementations, the sensor system is dynamically adaptable. Among other things, the sensor system: (a) dynamically adjusts the weight given to data measured by different sensors (accelerometer, gyroscope, optical sensor) or (b) dynamically adjusts the weight given to data gathered at different body locations (e.g., wristworn, headmounted, or handheld).
In illustrative implementations, this dynamic adjustment of weighting depends on one or more of the following trigger factors: sensor position (e.g., head, wrist, pocket), quality of data, posture of user, identity of user, gender of user, weight of user, age of user, activity of user, availability of data, and purpose for which data is being used (e.g., whether the data is being used to calculate posture, biometric identification, heart rate, or respiratory rate).
In illustrative implementations, the weights given to data from each of the sensors is adjusted based on one or more trigger factors. Here are some non-limiting examples:
When both optical (e.g., PPG) sensors and motion-based sensors (e.g., accelerometer, gyroscope) are positioned on the wrist, the sensor system selects a weighted set of one or more sensors that works best for each posture. For instance, in some cases, PPG sensors outperform motion-based sensors when estimating heart rate during sitting down and lying down, and a combination of both types of sensors outperforms each of them alone when standing.
When considering sensors from different body locations, the weights are automatically selected to choose the most advantageous combination. For instance, in some cases, among several motion-based sensors on the head, a gyroscope receives a higher weight than an accelerometer. However, in some cases, when gathering the same sensor information from a smartphone carried in a pocket, the accelerometer receives a higher weight than the gyroscope.
Different sensors receive different weights based on the quality and availability of the data. For instance, if the person carries devices on the head and the wrist, the algorithm will dynamically select the one that contains less noise and best signal to noise ratio. For instance, while typing at the computer, head sensors are expected to have less noise than wrist sensors.
In some implementations, posture is the only trigger factor that controls the dynamic adjustment of weighting. In some other implementations, identity or activity of a user is the only trigger factor that controls weighting. In other more complicated versions, there are multiple trigger factors that interact in more complex and perhaps iterative ways (see list of trigger factors above).
The weighting of sensors (by different sensor type or by different sensor position relative to the body) may be done in different ways, in illustrative implementations of this invention. In some cases, the signal stream for each sensor, respectively, is assigned a weight, so that data from different sensors may have different weights. In some other cases, an intermediate output of processing (e.g., pulse and respiratory waves, heart and breathing rates) performed by each of the sensors are weighted and combined. Both approaches may also be combined. The weighting may be done with different operators such as a weighted average (e.g., continuous weights) or filtering approach (e.g., binary weights).
In some cases, weights are determined in advance, in either a person-independent or a person-dependent way, using machine learning to infer an optimal method of combining the observations from the sensors being worn. For example, a Bayesian fusion method or a decision-tree may be trained on prior data and the weights applied to current data streams. Many types of machine learning may be used; this invention is not limited to a particular type. In many cases, the weighting is dependent not only on the type of sensors and their placement and on other fixed properties (e.g., gender or posture), but also on dynamic qualities of the current data and noise stream (e.g. data indicating that a car is accelerating).
In some cases, physiological parameters (e.g., HR or BR) are estimated separately by each sensor and then a weighted average of the estimates is taken, where the weight assigned to each estimate is equal to the magnitude of the highest magnitude frequency component of the waveform used to calculate the estimate For instance, in some cases, in order to aggregate readings by motion-based and optical sensors on the wrist, a weighted combination of the HR estimates obtained by the two types of sensors is calculated, as follows: (a) the heart rate (HR) measured by the motion-based sensor is estimated as the frequency of the highest magnitude component of the FFT of the pulse waveform measured by the motion-based sensor; (b) the HR measured by the optical sensor is estimated as the frequency of the highest magnitude component of the FFT of the pulse waveform measured by the optical sensor; (c) the motion-based HR estimate is weighted by a first weight equal to the magnitude of the highest magnitude component of the FFT of the cardiac waveform measured by the motion-based sensor; (d) the optical HR estimate is weighted by a second weight equal to the magnitude of the highest magnitude component of the FFT of the cardiac waveform measured by the optical sensor; and (e) the weighted average estimate of heart rate is equal to the sum of the weighted HR estimates divided by the sum of the weights. For instance, if the highest magnitude component of the FFT of a waveform measured by a motion-based sensor occurs at 1 Hz (60 beats per minute) with a magnitude of 0.5 dB and if the highest magnitude component of the FFT of a waveform measured by an optical sensor occurs at 1.5 Hz (90 beats per minute) with a magnitude of 1 dB, then a weighted average estimate of heart rate, weighted according to these magnitudes, is 1.33 Hz, corresponding to 80 beats per minute—that is ((1 Hz×w1)+(1.5 Hz×w2))/(w1+w2)=1.33 Hz, where w1=0.5 and w2=1.0.
As used herein: (a) a “head heart rate estimation” means an estimation of heart rate from measurements taken only by one or more sensors in a head-mounted sensor module 201; (b) a “wrist heart rate estimation” means an estimation of heart rate from measurements taken only by one or more sensors in a wrist-worn sensor module 211; (c) a “pocket heart rate estimation” means an estimation of heart rate from measurements taken only by sensors in a sensor module 221 worn in a pocket; (d) “head accelerometer” means a 3-axis accelerometer (e.g., 202) in a head-mounted sensor module 201; (e) “head gyroscope” means a 3-axis gyroscope (e.g., 203) in a head-mounted sensor module 201; (f) “wrist accelerometer” means a 3-axis accelerometer (e.g., 212) in a wrist-worn sensor module 211; (g) “wrist gyroscope” means a 3-axis gyroscope (e.g., 213) in a wrist-worn sensor module 211; (g) “wrist PPG” means a PPG sensor (e.g., 219) in a wrist-worn sensor module 211; (h) “pocket accelerometer” means a 3-axis accelerometer (e.g., 222) in a sensor module 221 worn in a pocket; and (h) “pocket gyroscope” means a 3-axis gyroscope (e.g., 223) in a sensor module 221 worn in a pocket.
In
Specifically, in the example shown in
In
In some cases, in order to aggregate components in
In
As used herein, a “heart rate estimation” means an estimate of heart rate, and is not merely (a) a pulse waveform, or (b) components used to compute a pulse waveform.
In
Specifically, in the example shown in
In
In
Specifically, in the example shown in
In
In
In illustrative implementations, the sensor module includes sensors that measure different things (i.e., a gyroscope measures rotation, an accelerometer measures linear acceleration, and the camera captures visual images). Having a sensor module with different sensors that measure different things is advantageous, because in some use scenarios, large artifacts reduce the accuracy of one or two of the sensors, but not the remaining sensor(s). For example, in some cases, a computer disregards, for purposes of calculating respiration rate, cardiac pulse rate or heart rate variability, data gathered by the accelerometer during periods in which the magnitude of linear acceleration measured by the accelerometer exceeds a specified threshold. Consider the following example (the “car example”): in a rapidly accelerating car, the car's linear acceleration, in the time domain, produces a large artifact for the accelerometer, but does not affect the gyroscope. In that case, it may be desirable to disregard the accelerometer data gathered during the rapid acceleration of the car.
In the examples shown in
In
In
In
As used herein, the terms “ballistocardiographic” and “BCG” are adjectives that mean: of or relating to estimating pulse rate or breath rate from measurements of body movements caused by cardiac pulse or by respiration. This definition is different than the common meaning of those terms.
In
In
In
In
In
In
HR=(w1HRBCG×w2HRPPG)/(w1+w2), where w1 and w2 are weights. A computer determines breathing rate (BR) as a weighted average of the BCG breathing rate (BRBCG) and PPG breathing rate (BRPPG). For example, in some cases, a computer calculates breathing rate as follows: BR=(w3BRBCG×w4BRPPG)/)/(w3+w4), where w3 and w4 are weights. (Step 1119).
In some cases, a PPG sensor measures blood volume pulse and subtle changes caused by respiration. For example, in some cases, these subtle changes caused by respiration that are measured by the PPG sensor include: (a) changes in pulse wave amplitude due to the fact that blood vessels are more flexible during expiration than inspiration; (b) changes in intrathoric pressure, which in turn cause changes in pulse envelope and cardiac output, which in turn cause changes in pulse wave amplitude; and (c) decreased venous return during inspiration and increased venous return during expiration.
In illustrative implementations of this invention, trigger factors are determined in a number of different ways. The trigger factors are used to determine what weights to apply in a weighted average of estimates of a physiological parameter, such as heart rate or breathing rate.
Quality of Data: In some cases, quality of data is a trigger factor (or the sole trigger factor) that affects weighting.
If the physiological parameter being measured relates to periodic physical phenomena (e.g., respiration and cardiac pulse), then an important aspect of quality of data is periodicity of signal. Pulse waveforms and respiratory waveforms have periodically recurring peaks. The more periodic the waveform, the higher the magnitude of the highest magnitude peak in the frequency domain. Put differently, the more periodic the waveform, the higher the magnitude of the frequency component that has the highest magnitude. In some cases, if a first and second estimate of a physiological parameter are derived from a first and second waveform, respectively, then the two estimates may be aggregated by taking a weighted average of the two estimates, where (i) the weight applied to the first estimate equals the magnitude of the highest magnitude component of the FFT of the first waveform, and (ii) the weight applied to the second estimate equals the magnitude of the highest magnitude component of the FFT of the second waveform.
In some cases, large motion artifacts (e.g., caused by person typing, by a person walking or running, or by rapid acceleration of a car that a user is driving) are an important aspect of quality of data. For example, if rapid linear acceleration in the time domain is detected, then during the acceleration, data from accelerometers (which are affected by the acceleration) may be disregarded or weighted by a lower weight and data from a gyroscope (which is not affected by the acceleration) may be given a higher weight.
In some embodiments of this invention, the amount of motion measured by a specific stream of sensor data is determined. For each specific stream of sensor data, this determination involves computing the first derivative of the sensor data, aggregating the different components (L2 norm), and, computing the standard deviation. Too much or too little standard deviation tends to negatively impact the quality of the readings.
User-Inputted Data: In some cases, certain trigger factors are inputted by a user. For example, in some cases, one or more I/O devices (e.g., 241-246) accept input from a user. The input specifies the identity, gender, weight or age of the user. Alternatively, a computer (e.g., 236) analyzes sensor readings to biometrically identify a user, and then accesses (or downloads from the Internet) data that is stored in electronic memory and that specifies the gender, weight or age of the user. The data stored in memory may have been inputted earlier by the user, or may have been determined by a computer by feature analysis.
Activity: In some cases, the type of activity in which a user is engaged is a trigger factor that affects weighting. For example, in some cases: (a) the sensor system includes multiple sensor modules; (b) one of the sensor modules is housed in a mobile computing device (MCD), such as a smartphone; (c) the MCD detects a type of activity that the user is engaged in, such as listening to music on the MCD, browsing the Internet via the MCD, or reading an e-book on an MCD screen. In some cases, a certain type of activity (such as typing, walking, or running) produce large motion artifacts that are determined by sensors (e.g., accelerometers or gyroscopes) in sensor modules of the sensor system.
Feature Analysis: In some cases, a computer analyses features in pulse or respiratory waveforms in order to determine trigger factors.
In some implementations, one or more trigger factors are determined by feature analysis of pulse or respiratory waveforms.
The following five paragraphs describe a prototype of this invention, in which feature analysis from pulse waveforms is performed. This prototype is a non-limiting example of how feature analysis may be performed.
In this prototype, a head-mounted sensor module and wrist-worn sensor module each include a 3-axis accelerometer and a 3-axis gyroscope. While the accelerometer captures linear accelerations (meters/second), the gyroscope captures the rate of rotation (radians/second) of the device. The average sampling rates are 50 Hz and 100 Hz for the head-mounted and wrist-worn modules, respectively. However, the streams of data are interpolated to a constant sampling rate of 256 Hz.
In this prototype, for purposes of training the sensor system, six separate one-minute recordings are taken per individual user, for twelve individual users. The six recordings per user are: each user is recorded in three postures (standing, sitting and lying down) and for each posture, before and after exercising. In order to create several sample readings for each of the conditions, the data is split into non-overlapping segments of 10 seconds each, yielding 432 segments equally balanced in terms of person and body posture. For each of these segments, several processing steps are employed to amplify BCG motions and to extract representative features. Each sensor modality (accelerometer and gyroscope) is processed separately.
In this prototype, a pulse waveform is recovered from subtle body movements as follows: First, normalize each of the 10-second sensor components (e.g., each axis of the 3 axis of the accelerometer) to have zero mean and unit variance. Next, subtract an averaging filter (window of 35 samples) to detrend the data and to remove relatively slow motions such as respiration and stabilizing body motions. Next, use a Butterworth band-pass filter (cut-off frequencies of 4-11 Hz) to recover the BCG waveform from each component. As not all the components carry relevant cardiac information, select the most periodic component of each sensor modality (accelerometer and gyroscope) by choosing the signal with highest amplitude response in the frequency domain.
In this prototype, features are extracted that may be used to characterize each of the measurements. The feature extraction includes segmenting the parts of the signal associated with different heartbeats, computing the average beat response, and extracting representative features from it. More specifically, the feature extraction includes the following steps: First, locate potential heart beat responses with the findpeaks MATLAB® function (with MIN_PEAK_DISTANCE equal to the length of a heartbeat when the heart rate is 150). (Reason: each heartbeat is characterized by a larger motion peak surrounded by smaller ones.) Then segment the signals by taking 300 milliseconds before and 500 milliseconds after each of the previous peaks. Next, average the different segments resulting in a specific BCG beat response. Next, extract the following features: 1) raw amplitude values, 2) histogram capturing the distribution of values (200 bins), and 3) shape features. For the shape features, extract the angles and distances between five descriptive points that are known to vary due to different trigger factors. Detect the descriptive points by splitting the signal into halves and using the findpeaks function to obtain the maximum and minimum values of each subsegment.
In this prototype, features are classified by a computer executing a linear Support Vector Machine algorithm with probability estimates, which allow for multiple class labels. Specifically, the algorithm uses the libSVM library which offers an efficient MATLAB® implementation. The misclassification cost is optimized with a 10 fold-cross validation approach on the training set. In other words, the training data is divided into 10 groups. Then, train on nine and test on the tenth and repeat the process for each of the groups to find which value yields the highest average classification accuracy. The considered values for misclassification cost are: log2 C, for C={−10, −9, −8, . . . 10}. In order to give the same relevance to each feature type, all the features are standardized to have zero mean and unit variance before training. Moreover, the dimensionality of the feature vector is reduced with Principal Component Analysis (preserving 95% of the energy), resulting in fewer than 100 components per condition.
This invention is not limited to the prototype described above. Feature analysis may be implemented in many other ways, in illustrative implementations of this invention.
A field of endeavor of this invention is dynamic weighting of data from a set of sensors, such that weighting changes over time and depends on at least (i) type of sensor and (ii) sensor position relative to a user's body.
The inventors were faced by a problem: The problem is how to dynamically weight data from a set of sensors, such that weighting changes over time and depends on at least (i) type of sensor and (ii) sensor position relative to a user's body.
In exemplary implementations of this invention, one or more electronic computers (e.g. 206, 216, 226, 236) are programmed and specially adapted: (1) to control the operation of, or interface with, hardware components of a sensor system, including any sensor module, accelerometer, gyroscope, PPG sensor, camera, EDA sensor, thermometer, or wireless transceiver unit; (2) to process sensor measurements to compute a cardiac pulse waveform or a respiratory waveform; (3) to extract physiological parameters from the pulse and respiratory waveforms, including heart rate, breathing rate, and heart rate variability; (4) to dynamically adjust the weight given to sensor data gathered by different sensors or the weight given to information that is calculated from such sensor data, such as pulse waveforms, respiratory waveforms, or estimated HR, HRV and BR; (5) to calculate the weights used in the dynamical adjustment, based on one or more trigger factors including position of sensor, posture of user, identity of user, gender of user, weight of user, age of user, activity of user (e.g., watching a movie, listening to music, or browsing), availability of data, quality of data, and what the weighted average is computing; (6) to determine one or more trigger factors based at least in part on (i) user input, (ii) activities of user detected by a smartphone or other mobile computing device, or (iii) feature analysis of a cardiac or respiratory waveform; (7) to perform any other calculation, computation, program, algorithm, computer function or computer task described or implied above; (8) to receive signals indicative of human input; (9) to output signals for controlling transducers for outputting information in human perceivable format; and (10) to process data, to perform computations, to execute any algorithm or software, and to control the read or write of data to and from memory devices. The one or more computers may be in any position or positions within or outside of the sensor system. For example, in some cases (a) at least one computer is housed in or together with other components of the sensor system, such as a sensor module, and (b) at least one computer is remote from other components of the sensor system. The one or more computers are connected to each other or to other components in the sensor system either: (a) wirelessly, (b) by wired connection, (c) by fiber-optic link, or (d) by a combination of wired, wireless or fiber optic links.
In exemplary implementations, one or more computers are programmed to perform any and all calculations, computations, programs, algorithms, computer functions and computer tasks described or implied above. For example, in some cases: (a) a machine-accessible medium has instructions encoded thereon that specify steps in a software program; and (b) the computer accesses the instructions encoded on the machine-accessible medium, in order to determine steps to execute in the program. In exemplary implementations, the machine-accessible medium comprises a tangible non-transitory medium. In some cases, the machine-accessible medium comprises (a) a memory unit or (b) an auxiliary memory storage device. For example, in some cases, a control unit in a computer fetches the instructions from memory.
In illustrative implementations, one or more computers execute programs according to instructions encoded in one or more tangible, non-transitory, computer-readable media. For example, in some cases, these instructions comprise instructions for a computer to perform any calculation, computation, program, algorithm, computer function or computer task described or implied above. For example, in some cases, instructions encoded in a tangible, non-transitory, computer-accessible medium comprise instructions for a computer to: 1) to control the operation of, or interface with, hardware components of a sensor system, including any sensor module, accelerometer, gyroscope, PPG sensor, camera, EDA sensor, thermometer, or wireless transceiver unit; (2) to process sensor measurements to compute a cardiac pulse waveform or a respiratory waveform; (3) to extract physiological parameters from the pulse and respiratory waveforms, including heart rate, breathing rate, and heart rate variability; (4) to dynamically adjust the weight given to sensor data gathered by different sensors or the weight given to information that is calculated from such sensor data, such as pulse waveforms, respiratory waveforms, or estimated HR, HRV and BR; (5) to calculate the weights used in the dynamical adjustment, based on one or more trigger factors including posture of user, identity of user, gender of user, weight of user, age of user, activity of user (e.g., watching a movie, listening to music, or browsing), availability of data, quality of data, and what the weighted average is computing; (6) to determine one or more trigger factors based at least in part on (i) user input, (ii) activities of user detected by a smartphone or other mobile computing device, or (iii) feature analysis of a cardiac or respiratory waveform computed from sensor readings; (7) to perform any other calculation, computation, program, algorithm, computer function or computer task described or implied above; (8) to receive signals indicative of human input; (9) to output signals for controlling transducers for outputting information in human perceivable format; and (10) to process data, to perform computations, to execute any algorithm or software, and to control the read or write of data to and from memory devices.
In illustrative implementations of this invention, an electronic device (e.g., any of devices 201-204, 206-208, 211-214, 216-219, 221-224, 226-228, 236) is configured for wireless or wired communication with other electronic devices in a network.
For example, in some cases, a computer 236 and sensor modules 201, 211, 221 each include a wireless transceiver unit for wireless communication with other electronic devices in a network. Each wireless transceiver unit (e.g., 205, 215, 225, 235) includes (a) one or more antennas, (b) one or more wireless transceivers, transmitters or receivers, and (c) signal processing circuitry. The wireless transceiver unit receives and transmits data in accordance with one or more wireless standards.
In some cases, one or more of the following hardware components are used for network communication: a computer bus, a computer port, network connection, network interface device, host adapter, wireless transceiver unit, wireless card, signal processor, modem, router, computer port, cables or wiring.
In some cases, one or more computers (e.g., 206, 216, 226, 236) are programmed for communication over a network. For example, in some cases, one or more computers are programmed for network communication: (a) in accordance with the Internet Protocol Suite, or (b) in accordance with any other industry standard for communication, including any USB standard, ethernet standard (e.g., IEEE 802.3), token ring standard (e.g., IEEE 802.5), wireless standard (including IEEE 802.11 (wi-fi), IEEE 802.15 (bluetooth/zigbee), IEEE 802.16, IEEE 802.20 and including any mobile phone standard, including GSM (global system for mobile communications), UMTS (universal mobile telecommunication system), CDMA (code division multiple access, including IS-95, IS-2000, and WCDMA), or LTS (long term evolution)), or other IEEE communication standard.
The terms “a” and “an”, when modifying a noun, do not imply that only one of the noun exists.
To say that X is “adjacent” to Y means that X physically touches Y.
The terms “ballistocardiographic” and “BCG” are defined above.
To compute “based on” specified data means to perform a computation that takes the specified data as an input.
Here are some non-limiting examples of a “camera”: (a) a digital camera; (b) a video camera; (c) a light sensor, (d) a set or array of light sensors; (e) an imaging system; (f) a light field camera or plenoptic camera; (g) a time-of-flight camera; or (h) an optical instrument that records images. A camera includes any computers or circuits that process data captured by the camera.
The term “comprise” (and grammatical variations thereof) shall be construed as if followed by “without limitation”. If A comprises B, then A includes B and may include other things.
The term “computer” includes any computational device that performs logical and arithmetic operations. For example, in some cases, a “computer” comprises an electronic computational device, such as an integrated circuit, a microprocessor, a mobile computing device, a laptop computer, a tablet computer, a personal computer, or a mainframe computer. In some cases, a “computer” comprises: (a) a central processing unit, (b) an ALU (arithmetic logic unit), (c) a memory unit, and (d) a control unit that controls actions of other components of the computer so that encoded steps of a program are executed in a sequence. In some cases, a “computer” also includes peripheral units including an auxiliary memory storage device (e.g., a disk drive or flash memory), or includes signal processing circuitry. However, a human is not a “computer”, as that term is used herein.
“Defined Term” means a term or phrase that is set forth in quotation marks in this Definitions section.
To say that Y “depends at least in part on” X means that Y depends at least in part on (i) X, (ii) an estimate of X, (iii) a probability regarding X, or (iv) a degree of membership in a fuzzy set X. For example, to say that Y “depends at least in part on” a person's weight means that Y depends at least in part on (i) the person's actual weight or weight range, (ii) an estimate of the person's weight or weight range, (iii) a probability regarding the person's weight or weight range, or (iv) a degree of membership in a fuzzy set regarding weight. To say that Y “depends at least in part on” a person's age means that Y depends at least in part on (i) the person's actual age or age range, (ii) an estimate of the person's age or age range, (iii) a probability regarding the person's age or age range, or (iv) a degree of membership in a fuzzy set regarding age.
For an event to occur “during” a time period, it is not necessary that the event occur throughout the entire time period. For example, an event that occurs during only a portion of a given time period occurs “during” the given time period.
The term “e.g.” means for example.
The fact that an “example” or multiple examples of something are given does not imply that they are the only instances of that thing. An example (or a group of examples) is merely a non-exhaustive and non-limiting illustration.
Unless the context clearly indicates otherwise: (1) a phrase that includes “a first” thing and “a second” thing does not imply an order of the two things (or that there are only two of the things); and (2) such a phrase is simply a way of identifying the two things, respectively, so that they each may be referred to later with specificity (e.g., by referring to “the first” thing and “the second” thing later). For example, unless the context clearly indicates otherwise, if an equation has a first term and a second term, then the equation may (or may not) have more than two terms, and the first term may occur before or after the second term in the equation. A phrase that includes a “third” thing, a “fourth” thing and so on shall be construed in like manner.
“For instance” means for example.
To say that data is “from” a sensor means that the data represents measurements taken by the sensor or represents information calculated from the measurements.
“Herein” means in this document, including text, specification, claims, abstract, and drawings.
As used herein: (1) “implementation” means an implementation of this invention; (2) “embodiment” means an embodiment of this invention; (3) “case” means an implementation of this invention; and (4) “use scenario” means a use scenario of this invention.
The term “include” (and grammatical variations thereof) shall be construed as if followed by “without limitation”.
“I/O device” means an input/output device. Non-limiting examples of an I/O device include any device for (a) receiving input from a human user, (b) providing output to a human user, or (c) both. Non-limiting examples of an I/O device also include a touch screen, other electronic display screen, keyboard, mouse, microphone, handheld electronic game controller, digital stylus, display screen, speaker, or projector for projecting a visual display.
The term “mobile computing device” or “MCD” means a device that includes a computer, a camera, a display screen and a wireless transceiver. Non-limiting examples of an MCD include a smartphone, cell phone, mobile phone, tablet computer, laptop computer and notebook computer.
The term “or” is inclusive, not exclusive. For example A or B is true if A is true, or B is true, or both A or B are true. Also, for example, a calculation of A or B means a calculation of A, or a calculation of B, or a calculation of A and B.
As used herein, “parameter” means a variable. For example: (a) if y=f(x), then both x and y are parameters; and (b) if z=f(x(t), y(t)), then t, x, y and z are parameters. For example, in some cases a parameter is a physiological variable, such as heart rate, heart rate variability or respiration rate.
“Physiological gender” of a person means the sex (male or female) of person indicated by the reproductive organs of the person.
As used herein, the term “set” does not include a group with no elements. Mentioning a first set and a second set does not, in and of itself, create any implication regarding whether or not the first and second sets overlap (that is, intersect).
“Some” means one or more.
As used herein, a “subset” of a set consists of less than all of the elements of the set.
“Substantially” means at least ten percent. For example: (a) 112 is substantially larger than 100; and (b) 108 is not substantially larger than 100.
The term “such as” means for example.
As used herein, “trigger factor” means a factor that affects a weight that is given to data.
To say that a machine-readable medium is “transitory” means that the medium is a transitory signal, such as an electromagnetic wave.
To say that two sensors are “a single type of motion sensor” means that the two sensors are both accelerometers or are both gyroscopes.
Except to the extent that the context clearly requires otherwise, if steps in a method are described herein, then the method includes variations in which: (1) steps in the method occur in any order or sequence, including any order or sequence different than that described; (2) any step or steps in the method occurs more than once; (3) different steps, out of the steps in the method, occur a different number of times during the method, (4) any combination of steps in the method is done in parallel or serially; (5) any step or steps in the method is performed iteratively; (6) a given step in the method is applied to the same thing each time that the given step occurs or is applied to different things each time that the given step occurs; or (7) the method includes other steps, in addition to the steps described.
This Definitions section shall, in all cases, control over and override any other definition of the Defined Terms. For example, the definitions of Defined Terms set forth in this Definitions section override common usage or any external dictionary. If a given term is explicitly or implicitly defined in this document, then that definition shall be controlling, and shall override any definition of the given term arising from any source (e.g., a dictionary or common usage) that is external to this document. If this document provides clarification regarding the meaning of a particular term, then that clarification shall, to the extent applicable, override any definition of the given term arising from any source (e.g., a dictionary or common usage) that is external to this document. To the extent that any term or phrase is defined or clarified herein, such definition or clarification applies to any grammatical variation of such term or phrase, taking into account the difference in grammatical form. For example, the grammatical variations include noun, verb, participle, adjective, and possessive forms, and different declensions, and different tenses. In each case described in this paragraph, the Applicant or Applicants are acting as his, her, its or their own lexicographer.
This invention may be implemented in many different ways. Here are some non-limiting examples:
In one aspect, this invention is a sensor system that comprises a set of sensors for measuring motion of a user's body, which set of sensors includes one or more gyroscopes and one or more accelerometers, wherein the sensor system is configured: (a) to make estimations of one or more physiological parameters of a user, based on data from the set of sensors, (b) to assign different weights to data from different sensors when making the estimations, such that (i) for at least one estimation, a weight assigned to data from at least one gyroscope is different than a weight assigned to data from at least one accelerometer; (ii) for at least one estimation, a weight assigned to data from a first sensor located in a first region relative to the user's body is different than a weight assigned to data from a second sensor located in a second region relative to the user's body, which first and second regions do not intersect, the first and second sensors being a single type of motion sensor; and (iii) a weight assigned to data from at least one sensor changes from at least one estimation to another estimation. In some cases, a weight assigned to a sensor depends at least in part on whether a user is standing, sitting or lying down. In some cases, a weight assigned to data from a specific sensor depends at least in part on periodicity of a signal measured by the specific sensor. In some cases, a weight assigned to data from a given sensor depends at least in part on magnitude of the highest magnitude frequency component of the data from the given sensor. In some cases, a weight assigned to data from a sensor depends at least in part on identity of the user. In some cases, a weight assigned to data from a sensor depends at least in part on physiological gender of the user. In some cases, a weight assigned to data from a sensor depends at least in part on age of the user. In some cases, a specific weight assigned to data from a sensor depends at least in part on what is being calculated, in a calculation that involves a multiplication of a term by the given weight. In some cases, a weight assigned to data from a particular sensor depends at least in part on magnitude of linear acceleration measured by the particular sensor. In some cases, the first and second regions are selected from a set of regions that includes (a) a region adjacent to the user's head, and (b) a region that is adjacent to the user's wrist and that does not intersect the region adjacent to the user's head. In some cases, the one or more physiological parameters include cardiac pulse rate. In some cases, the one or more physiological parameters include respiratory rate. In some cases, the one or more physiological parameters include heart rate variability. In some cases, the sensor system is configured: (a) to make a biometric identification of the identity of the user, based at least in part on measurements taken by the one or more accelerometers and one or more gyroscopes; and (b) to assign different weights to data from different sensors, when making the biometric identification. In some cases, the sensor system is configured to assign different weights to data from different sensors, such that, when making the biometric identification, a weight assigned to data from at least one gyroscope is different than a weight assigned to data from at least one accelerometer. In some cases, the sensor system is configured to assign different weights to data from different sensors, such that, when making the biometric identification, a weight assigned to data from a sensor (Sensor A) located in a first region relative to the user's body is different than a weight assigned to data from a sensor (Sensor B) located in a second region relative to the user's body, which first and second regions do not intersect, Sensor A and Sensor B being a single type of motion sensor. In some cases: (a) the sensor system includes one or more optical sensors for measuring light that reflects from or is transmitted through skin; and (b) the sensor system is configured to assign different weights to data from different sensors when making the estimations, such that for at least one estimation, a weight assigned to data from at least one optical sensor is different than a weight assigned to data from at least one accelerometer or from at least one gyroscope. In some cases, at least one optical sensor is a photoplethysmographic sensor. In some cases, at least one optical sensor is a camera that measures motion of a scene relative to the user. Each of the cases described above in this paragraph is an example of the sensor system described in the first sentence of this paragraph, and is also an example of an embodiment of this invention that may be combined with other embodiments of this invention.
In another aspect, this invention is a method comprising, in combination: (a) a set of sensors measuring motion of a user's body, which set of sensors includes one or more gyroscopes and one or more accelerometers; and (b) one or more computers making estimations of one or more physiological parameters of a user, based on data from the set of sensors, such that (i) for at least one estimation, a weight assigned to data from at least one gyroscope is different than a weight assigned to data from at least one accelerometer; and (ii) for at least one estimation, a weight assigned to data from a first sensor located in a first region relative to the user's body is different than a weight assigned to data from a second sensor located in a second region relative to the user's body, which first and second regions do not intersect, the first and second sensors being a single type of motion sensor; and (iii) a weight assigned to data from at least one sensor changes over time. The method described in the first sentence of this paragraph is an example of an embodiment of this invention that may be combined with other embodiments of this invention.
The above description (including without limitation any attached drawings and figures) describes illustrative implementations of the invention. However, the invention may be implemented in other ways. The methods and apparatus which are described above are merely illustrative applications of the principles of the invention. Other arrangements, methods, modifications, and substitutions by one of ordinary skill in the art are therefore also within the scope of the present invention. Numerous modifications may be made by those skilled in the art without departing from the scope of the invention. Also, this invention includes without limitation each combination and permutation of one or more of the abovementioned implementations, embodiments and features.
This application claims the benefit of U.S. Provisional Application No. 62/053,802, filed Sep. 23, 2014 (the “802 Application”), U.S. Provisional Application No. 62/053,805, filed Sep. 23, 2014 (the “805 Application”), and U.S. Provisional Application No. 62/103,782, filed Jan. 15, 2015 (the “782 Application”). This application is a continuation-in-part of U.S. application Ser. No. 14/661,747, filed Mar. 18, 2015 (the “747 Application”), which claims the benefit of U.S. Provisional Application No. 61/955,772, filed Mar. 19, 2014 (the “772 Application”). The entire disclosures of the 802 Application, 805 Application, 782 Application, 747 Application and 772 Application are incorporated herein by reference.
This invention was made with government support under Grant No. IIS-1029585 awarded by the National Science Foundation. The government has certain rights in the invention.
| Number | Date | Country | |
|---|---|---|---|
| 62053802 | Sep 2014 | US | |
| 62053805 | Sep 2014 | US | |
| 62103782 | Jan 2015 | US | |
| 61955772 | Mar 2014 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14661747 | Mar 2015 | US |
| Child | 14861388 | US |
