The present disclosure relates generally to estimating pulse transmit times (PTT) of an individual and, more particularly, to a method and apparatus for non-contact estimation of a PTT.
Developing different ways of monitoring an individual's health is becoming an important issue for individuals and companies. Hardware and applications are continuously being developed to monitor various aspects of an individual's health. Blood pressure is one parameter that can provide a high level indication of whether an individual is healthy or not.
However, blood pressure may be difficult to directly measure unless using a sphygmomanometer. Pulse transmit times can be correlated to the blood pressure of an individual. Pulse transmit times are typically measured with contact sensors that are placed on an individual's finger tips. Requiring contact with a sensor can sometimes be cumbersome or dangerous for some patients (e.g., pre-mature babies in neonatal units). In addition, requiring contact may limit the ways that the pulse transmit time can be measured.
According to aspects illustrated herein, there are provided a method, non-transitory computer readable medium and apparatus for estimating a pulse transmit time (PTT) to calculate a blood pressure. One disclosed feature of the embodiments is a method that receives a series of video images over a period of time that includes a first location of an individual and a second location of the individual, calculates a first set of luminance values of the first location and a second set of luminance values of the second location, estimates the PTT based on an average time difference from consecutive peaks of the first set of luminance values and the second set of luminance values and transmits the PTT to a blood pressure calculation device to calculate the blood pressure based on the PTT that is estimated.
Another disclosed feature of the embodiments is a non-transitory computer-readable medium having stored thereon a plurality of instructions, the plurality of instructions including instructions which, when executed by a processor, cause the processor to perform operations that receive a series of video images over a period of time that includes a first location of an individual and a second location of the individual, calculate a first set of luminance values of the first location and a second set of luminance values of the second location, estimate the PTT based on an average time difference from consecutive peaks of the first set of luminance values and the second set of luminance values and transmit the PTT to a blood pressure calculation device to calculate the blood pressure based on the PTT that is estimated.
Another disclosed feature of the embodiments is an apparatus comprising a processor and a computer-readable medium storing a plurality of instructions which, when executed by the processor, cause the processor to perform operations that receive a series of video images over a period of time that includes a first location of an individual and a second location of the individual, calculate a first set of luminance values of the first location and a second set of luminance values of the second location, estimate the PTT based on an average time difference from consecutive peaks of the first set of luminance values and the second set of luminance values and transmit the PTT to a blood pressure calculation device to calculate the blood pressure based on the PTT that is estimated.
The teaching of the present disclosure can be readily understood by considering the following detailed description in conjunction with the accompanying drawings, in which:
To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures.
The present disclosure broadly discloses a method and apparatus for estimating a pulse transmit time (PTT) to calculate a blood pressure. As discussed above, developing different ways of monitoring an individual's health is becoming an important issue for individuals and companies. Typically, blood pressure is difficult to measure directly without a sphygmomanometer. Some methods use a PTT that is obtained via contact sensors placed on the individual. However, requiring contact with a sensor can sometimes be cumbersome or dangerous for some patients (e.g., pre-mature babies in neonatal units). In addition, requiring contact may limit the ways that the pulse transmit time can be measured.
Embodiments of the present disclosure provide a non-contact method for estimating an individual's PTT. The PTT can be used to calculate a systolic blood pressure of the individual. In one embodiment, a video image of a user is captured and various regions of interest are analyzed to estimate the individual's PTT. As a result, the methods use video of the individual that is non-invasive and does not require any equipment to be placed on the individual.
The AS 104 may be a dedicated computer or machine for performing the functions described herein. The AS 104 may include a processor and non-transitory computer readable memory that stores instructions that are executed by the processor. An example of the dedicated computer or machine is illustrated in
The AS 104 may be in communication with the camera 110.
Alternatively, the video images may be captured and stored on a storage medium (e.g., a memory card, an external hard drive, and the like) and accessed by the AS 104 “offline.” The AS 104 may analyze the video images using the methods described below to estimate or calculate a PTT of the individual 150.
For example, a video image 140 may be captured of the individual 150. In one embodiment, the camera 110 may be positioned in front of the individual 150 such that the video image 140 captures at least two different locations on the individual 150. For example, the locations may include a forehead 112 of the individual 150 and a palm 114 of the individual 150.
The AS 104 may apply existing facial recognition technology to identify the forehead 112 and the palm 114 in the video image 140. In one embodiment, the AS 104 may analyze a region of interest (ROI) 116 within the area of the forehead 112 and a ROI 118 within the area of the palm 114.
As will be discussed in further detail below, the ROI 116 may be further divided into a plurality of macro pixels 1201 to 120m (herein referred to individually as a macro pixel 120 or collectively as macro pixels 120). The ROI 118 may be further divided into a plurality of macro pixels 1221 to 122n (herein referred to individually as a macro pixel 122 or collectively as macro pixels 122). It should be noted that a number of macro pixels 120 can be different, or the same, as the number of macro pixels 122.
A macro pixel 120 or 122 may be defined to be a larger grouping of pixels. For example, each macro pixel 120 or 122 may include approximately 100 pixels on which the analysis is conducted. In one embodiment, the use of the macro pixels 120 and 122 allow for a more accurate estimation of the PTT by accounting for slight variations in the skin color, movement, and the like, of the individual 150.
Luminance values may be obtained for each macro pixel 1201 to 120m and each macro pixel 1221 to 122n over a period of time. A difference in time between peaks of the luminance values in the ROI 116 of the forehead 112 of the individual 150 and peaks of the luminance values in the ROI 118 of the palm 114 of the individual 150 may be used to estimate the PTT. As a result, a series of video images 140 captured by the camera 110 may be analyze and used to estimate the PTT for the individual 150, as described in further detail below.
In one embodiment, the system 100 may be used to continuously estimate the PTT that is used to calculate the blood pressure of the individual 150. For example, while the individual 150 is watching television, the camera 110 may continuously capture the video images 140 of the individual 150. The PTT may be estimated based upon pre-determined time periods (e.g., every 10 minutes, every hour, and the like) based on the video images 140.
In one embodiment, the AS 104 may transmit the PTT to a blood pressure (BP) calculation device 108, over either a wired or wireless communication, via an Internet Protocol (IP) network 102 or the Internet. In one embodiment, the BP calculation device 108 may be in communication with a database (DB) 106 to store the PTT values received from the AS 104, as well as other information (e.g., user profiles, conversion algorithms, and the like).
It should be noted that the IP network 102 has been simplified for ease of explanation. For example, the IP network 102 may also include other network elements or access networks not shown (e.g., gateways, routers, switches, border elements, and the like).
In one embodiment, the BP calculation device 108 and the DB 106 may be located remotely from the AS 104. Alternatively, the BP calculation device 108 and the DB 106 may comprise separate functional modules within a common hardware device as the AS 104.
At block 302, the method 300 begins. At block 304, the method 300 receives a series of video images over a period of time that includes a first location of an individual and a second location of the individual. For example, referring to
In one embodiment, the series of video images may be an RGB color image captured by the camera 110. In one embodiment, the series of video images may be consecutive video images of a single video. In another embodiment, the series of video images may be non-consecutive video images from a plurality of videos. In other words, video images from a plurality of different videos may be stitched together for analysis.
At block 306, the method 300 calculates a first set of luminance values of the first location and a second set of luminance values of the second location. For example, for each location an ROI may be identified. The ROIs may be further divided into a plurality of macro pixels, as described in
In one embodiment, a luminance value may be calculated for each video image that is captured in the video. For example, if a video is captured by the camera 110 that includes a series of 120 video images per second (e.g., the camera 110 may capture video at 120 frames per second (fps)), then a luminance value for each macro pixel within a respective ROI of a respective location may be calculated for each one of the 120 video images.
In one embodiment, the luminance value may be the maximum luminance value of a pixel within the macro pixel. As described above, each macro pixel may comprise a plurality of pixels (e.g., approximately 100 pixels). Thus, the luminance value may be represented by the pixel with the highest luminance value. In another embodiment, the luminance value for each macro pixel within each video image may be an average of the luminance values of the pixels within the macro pixel.
At block 308, the method 300 may estimate the PTT based on an average time difference from consecutive peaks (or peak pairs) of the first set of luminance values and the second set of luminance values.
However, the AS 104 may process the luminance values 202 and 208 to identify a plurality of peaks 2041 to 204n (herein also referred to individually as a peak 204 or collectively as peaks 204) associated with luminance values of the forehead 112 and a plurality of peaks 2061 to 206n (herein also referred to individually as a peak 206 or collectively as peaks 206) associated with luminance values of the palm 114. In one embodiment, luminance values 202 and 208 obtained from the video images may require further processing to identify the peaks 204 and 206 due to noise in the video images.
In one embodiment, a peak 204 or 206 may be identified based on a peak prominence threshold value and a minimum peak separation threshold value (herein also referred to generically as a first threshold and a second threshold, respectively). In one embodiment, the peak prominence threshold value may determine whether a luminance value is large enough to be considered a peak. For example, the peak prominence threshold value may be 0, 0.02, and the like; however, any value may be used. Thus, a luminance value may be considered to be a peak if the luminance value is greater than the peak prominence threshold value.
In one embodiment, the minimum peak separation threshold value may determine if there was enough time between video images to be considered to be a separate peak. In other words, the luminance value may be considered to be a peak if the peak is separated from an adjacent peak (e.g., an adjacent peak within the same set of luminance values) by an amount of time greater than the minimum peak separation threshold value (e.g., 0.5, or any other value). For example, a peak may be followed immediately by another peak that is noise (e.g., the amount of time between the two peaks is below the minimum peak separation threshold value). However, if the additional peak is considered, the peaks 204 of the forehead 112 may not align properly with the peaks 206 of the palm 114 for proper estimation of the PTT as discussed in further detail below.
After the peaks 204 and 206 are identified, the peaks 204 and 206 may be aligned. For example, the blood pumped by the heart should reach the forehead before the palm as the forehead is closer to the heart than the palm. As a result, the peaks 204 associated with luminance values of the forehead should lead (e.g., be earlier in time) the peaks 206 associated with luminance values of the palm. Thus, the identified peaks 204 and 206 may be aligned to ensure that the peaks 204 lead the peaks 206 to ensure an accurate time differential is calculated.
After the peaks 204 and 206 are identified and aligned, a time difference 209 between consecutive peaks may be calculated. In one embodiment, the term “consecutive peaks” may refer to a peak 204 from the first set of luminance values and an adjacent peak 206 from the second set of luminance values. For example, in
In one embodiment, the time difference 209 between consecutive peaks may be an estimated PTT from one cycle of the heartbeat. The estimated PTT may be calculated based on an average time difference of all the time differences 209 between consecutive peaks of the peaks 204 and 206.
As will be discussed in further detail below with reference to
At block 310, the method 300 transmits the PTT to a blood pressure calculation device to calculate the blood pressure based on the PTT that is estimated. For example, the PTT may be used to calculate a systolic blood pressure value of the individual 150 based on the height, weight, gender, and the like of the individual 150. Said another way, the AS 104 may generate and output the estimated PTT that can be used by the blood pressure calculation device to calculate the blood pressure of the individual 150. At block 312, the method 300 ends.
At block 402, the method 400 begins. At block 404, the method 400 receives a video. For example, the video may be an RGB video that includes a series of images recorded over a period of time of an individual. The video may include at least two different locations on the individual (e.g., a forehead and a palm of the individual).
At block 406, the method 400 identifies an ROI of the forehead and an ROI of the palm. The ROIs may be further divided into a plurality of macro pixels.
At block 408, the method may determine luminance values in the ROIs. For example, the luminance values may be determined for each macro pixel in the ROI of the forehead and each macro pixel in the ROI of the palm. In one embodiment, the luminance value may be a maximum luminance value associated with a pixel within a respective macro pixel. In another embodiment, the luminance value may be an average of the luminance value of each pixel within the respective macro pixel.
In one embodiment, each video image of the series of video images of the video that is captured may have a plurality of luminance values associated with it. For example, a luminance value may be determined for each macro pixel of the ROI of the forehead and a luminance value may be determined for each macro pixel of the ROI of the palm.
At block 408, the method 400 may obtain a cyclical component of luminance values. For example, the luminance values (yt) may be comprised of a trend component (τt), a cyclical component (CO and an error component (εt) according to the relationship shown in Equation 1 below:
y
t=τt+Ct+εt Equation 1:
Equation 1 may be rearranged to solve for Ct+εt using an optimization function. One example of an optimization function is the Hodrick and Prescott Decomposition described below in Equation 2.
where t is time from 1 to T and λ is a scalar value that is suitably chosen. For example, a high value of λ makes the luminance values a straight line. In one embodiment, λ may be chosen to be very large (e.g., λ=106).
At block 412, the method 400 may apply a median filter. The median filter may be applied to smooth out the variations and outliers caused by noise or movement of the individual in the luminance values that are determined from each macro block.
In one embodiment, the median filter may apply a sliding window using a pre-defined number of data points to calculate the median of the data points within the sliding window. In one embodiment, the pre-defined number may be an odd number (e.g., 51). To illustrate, the first 6 data points may be 3, 4, 7, 5, 9 and 6. Using a sliding window that has a pre-defined size of 5 data points, the sliding window may first calculate the median of the data points 3, 4, 7, 5 and 9 to be 5. Then, the sliding window may calculate the median of the next 5 data points 4, 7, 5, 9 and 6 as 6, and so forth.
At block 414, the method 400 may determine if more macro pixels need to be processed. If all of the macro pixels in each ROI identified in block 406 have not been processed, then the method 400 may return to block 408 and the blocks 408-414 may be repeated for each additional macro pixel. However, if all of the macro pixels have been processed, the method 400 may proceed to block 416.
At block 416, the method 400 generates random pairs of macro pixels. For example, referring back to
At block 418, the method 400 may align the luminance values for the paired macro pixels. In one embodiment, the luminance values at block 418 may refer to the cyclical portions of the luminance values that were calculated in block 410. As discussed above in
In addition, the peaks of the luminance values of a macro pixel of the forehead are aligned with the peaks of the luminance values of a macro pixel of the palm that are randomly paired. As discussed above, the peaks of the macro pixel of the forehead should lead the peaks of the macro pixel of the palm. In addition, each peak of the macro pixel of the forehead should be paired with a peak of the macro pixel of the palm (e.g., peak pairs). This is because each heartbeat should have a luminance signal detected in the forehead and a corresponding luminance signal detected in the palm.
Error correction may be applied to the luminance values to remove any peaks that do not meet the peak prominence threshold value and minimum peak separation threshold and to remove any outlier peaks that are not paired with a corresponding peak. For example, a peak of the macro pixel of the forehead should be paired with only a single corresponding, adjacent or consecutive peak from the macro pixel of the palm. In addition, the error correction may remove any peaks from a macro pixel of the palm that lead a corresponding peak from a macro pixel of the forehead.
At block 420, the method 400 calculates an average time difference between consecutive peaks. Referring to
At block 422, the method 400 may determine if more paired macro pixels need to be processed. If additional paired macro pixels need to be processed, the method 400 may return to block 418 and repeat blocks 418 to 422. However, if all of the paired macro pixels have been processed, the method 400 may proceed to block 424.
At block 424, the method 400 estimates the PTT based on an overall average of the average time difference of the paired macro pixels. For example, each pair of macro pixels may have an average time difference. The average time difference for each paired macro pixel may then be averaged again to calculate the overall average time difference. The overall average time difference may represent the estimated PTT for the individual.
At block 426, the method 400 may transmit the PTT to a blood pressure calculation device. As noted above, the PTT may be used to calculate a systolic blood pressure of the individual. At block 428, the method 400 ends.
It should be noted that although not explicitly specified, one or more steps, functions, or operations of the methods 300 and 400 described above may include a storing, displaying and/or outputting step as required for a particular application. In other words, any data, records, fields, and/or intermediate results discussed in the methods can be stored, displayed, and/or outputted to another device as required for a particular application. Furthermore, steps, functions, or operations in
As depicted in
It should be noted that the present disclosure can be implemented in software and/or in a combination of software and hardware, e.g., using application specific integrated circuits (ASIC), a programmable logic array (PLA), including a field-programmable gate array (FPGA), or a state machine deployed on a hardware device, a computer or any other hardware equivalents, e.g., computer readable instructions pertaining to the method(s) discussed above can be used to configure a hardware processor to perform the steps, functions and/or operations of the above disclosed methods. In one embodiment, instructions and data for the present module or process 505 for estimating a pulse transmit time (PTT) to calculate a blood pressure (e.g., a software program comprising computer-executable instructions) can be loaded into memory 504 and executed by hardware processor element 502 to implement the steps, functions or operations as discussed above in connection with the exemplary methods 300 and 400. Furthermore, when a hardware processor executes instructions to perform “operations,” this could include the hardware processor performing the operations directly and/or facilitating, directing, or cooperating with another hardware device or component (e.g., a co-processor and the like) to perform the operations.
The processor executing the computer readable or software instructions relating to the above described method(s) can be perceived as a programmed processor or a specialized processor. As such, the present module 505 for estimating a pulse transmit time (PTT) to calculate a blood pressure (including associated data structures) of the present disclosure can be stored on a tangible or physical (broadly non-transitory) computer-readable storage device or medium, e.g., volatile memory, non-volatile memory, ROM memory, RAM memory, magnetic or optical drive, device or diskette and the like. More specifically, the computer-readable storage device may comprise any physical devices that provide the ability to store information such as data and/or instructions to be accessed by a processor or a computing device such as a computer or an application server.
It will be appreciated that variants of the above-disclosed and other features and functions, or alternatives thereof, may be combined into many other different systems or applications. Various presently unforeseen or unanticipated alternatives, modifications, variations, or improvements therein may be subsequently made by those skilled in the art which are also intended to be encompassed by the following claims.
This invention was made with government support under Contract No. 1U01EB018818-01, awarded by the National Institutes of Health (NIH). The government has certain rights in this invention.