Patent Grant
8605961
This application is related to application Ser. No. 12/347,400 entitled “Method and Apparatus for Determining Blood Oxygenation Using a Mobile Communication Device,” filed on Dec. 31, 2008, and commonly assigned to the assignee of the present application, which is hereby incorporated by reference.
1. Field
The present disclosure is directed to a method and apparatus for determining a physiological parameter using a fingerprint sensor on portable electronic device. More particularly, the present disclosure is directed to determining a physiological parameter, such as a heart rate and/or oxygen consumption, from images captured using a fingerprint sensor on a portable electronic device.
2. Introduction
Presently, mobile communication device users can engage in active lifestyle activities and may desire to obtain corresponding physiological parameters, such as heath metrics, blood related physiological parameters, heart rate, oxygen consumption, maximal oxygen consumption (VO2 max), blood oxygenation, or any other physiological parameters. The ability to easily obtain key health metrics provides users with the ability to monitor their health and fitness. Tracking and monitoring data like distance, speed, caloric burn, heart rate, and other key measurements can help users achieve their health and fitness goals. For example, heart rate monitors can be used to measure the heart rate of users with active lifestyles. Heart rate monitors are also useful for patients with respiratory or cardiac issues and useful for athletes whose heart rate may change at high altitudes and/or with intense exercise.
Unfortunately, most heart rate monitors require the use of specialized equipment and are not easy to integrate into existing portable electronic devices because such integration requires extensive hardware modifications to the devices. Also, heart rate monitors do not always meet all end user requirements in all application settings. At the very least, such devices require a user to maintain, keep powered, and carry about an additional device along with their other portable devices, such along with their cellular phone. This can lead to unwanted surprises when the additional device has run out of power during a time of need and/or when a user does not have the additional device during a time of need because the user has not included the device amongst the items that the user carries about.
Thus, there is a need for method and apparatus that determines a physiological parameter using a fingerprint sensor on a portable electronic device.
A method and apparatus that determines a physiological parameter using a fingerprint sensor on a portable electronic device is disclosed. The method can include capturing a plurality of images corresponding to an area beneath a surface of skin using a fingerprint sensor configured to capture a live scan of a fingerprint pattern from a finger on a touch surface on a portable electronic device. The method can include comparing image characteristics corresponding to at least a first image of the plurality of images with image characteristics corresponding to at least a second image of the plurality of images. The method can include determining a physiological parameter based on comparing the image characteristics.
In order to describe the manner in which advantages and features of the disclosure can be obtained, a more particular description of the disclosure briefly described above will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. Understanding that these drawings depict only exemplary embodiments of the disclosure and are not therefore to be considered to be limiting of its scope, the disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:
In operation, a fingerprint sensor on the terminal 110 can operate in a fingerprint reader mode that can include capturing a live scan of a fingerprint pattern from a finger 120 on a touch surface to authenticate a user of the terminal 110. In accordance with some embodiments, fingerprint sensor parameters can be changed to switch the fingerprint sensor from the fingerprint reader mode to a heart rate determination mode. A plurality of images corresponding to an area beneath a surface of skin can then be captured using the fingerprint sensor. Image characteristics corresponding to at least a first image of the plurality of images can be compared with image characteristics corresponding to at least a second image of the plurality of images. A physiological parameter, such as a blood related physiological parameter like heart rate, oxygen consumption, maximal oxygen consumption (VO2 max), blood oxygenation, or any other physiological parameter, can be determined based on comparing the image characteristics. Data corresponding to the physiological parameter can be output from a module in the terminal 110. For example, the data corresponding to a heart rate can be output to memory on the terminal 110, can be output on a display, can be output using an auditory signal, can be output by wirelessly transmitting the data, or can be output in any other manner.
As a further example, a physiological parameter of a subject can be determined using a fingerprint sensor available on a mobile device, such as on the terminal 110. A user can place his/her finger in proximity to the fingerprint sensor and a video or picture sequence can be captured. Image analysis and signal processing techniques can then be used on the captured sequence to process and extract a physiological parameter. The processing can be done either in real-time, such as while the picture sequence is being captured, or can be done off-line after the picture sequence has been captured. An existing fingerprint reader on the terminal 110 can be used for a physiological parameter measurement and the process can be implemented in the terminal 110 without hardware changes. This aspect can be implemented for users involved in fitness activities, for those with special health care needs, for heath care practitioners, and for other users interested in determining a physiological parameter.
As yet a further example, a physiological parameter determination application can be launched in the terminal 110. The physiological parameter determination application can enable a physiological parameter determination mode on the terminal 110. For example, the physiological parameter determination application can switch a fingerprint sensor, such as a fingerprint scanner, from a fingerprint swipe mode to a physiological parameter determination mode that can constantly sense or scan a user's finger that is placed against the fingerprint sensor over a set period of time. A user can then place their finger in front of a fingerprint sensor on the terminal 110. The physiological parameter determination application can then determine the user's physiological parameter and can display the corresponding information on a display of the terminal 110. For example, the physiological parameter determination application can determine the physiological parameter by performing intensity and color based image analysis of the scanned finger. The physiological parameter determination application can display the corresponding data, time stamp the corresponding data, and/or store the corresponding data on the terminal 110 or at a server coupled to the network 130. The physiological parameter determination application can also use the physiological parameter data as an additional form of identification of a user.
The display 240 can be a liquid crystal display (LCD), a light emitting diode (LED) display, a plasma display, or any other means for displaying information. The transceiver 250 can include a transmitter and/or a receiver. The audio input and output circuitry 230 can include a microphone, a speaker, a transducer, or any other audio input and output circuitry. The user interface 260 can include a keypad, buttons, a touch pad, a joystick, an additional display, or any other device useful for providing an interface between a user and an electronic device. The memory 270 may include a random access memory, a read only memory, an optical memory, a subscriber identity module memory, or any other memory that can be coupled to a wireless communication device.
In operation, the controller 220 can be configured to control operations of the apparatus 200. The selective call receiver transceiver 250 can be configured to engage in selective call receiver communications over a long distance wireless wide area network, such as the network 130, and can wirelessly transmit and receive data across the wireless wide area network. The fingerprint sensor 290 can be configured to capture a live scan of a fingerprint pattern from a finger 120 on a touch surface. The image capture module 294 can be configured to capture a plurality of images using the fingerprint sensor 290. The plurality of images can correspond to an area beneath a surface of skin. The image comparison module 296 can be configured to compare image characteristics corresponding to at least a first image of the plurality of images with image characteristics corresponding to at least a second image of the plurality of images. The physiological parameter determination module 298 can be configured to determine a physiological parameter based on comparing the image characteristics. The physiological parameter can be a blood related physiological parameter, such as heart rate, oxygen consumption, maximal oxygen consumption, blood oxygenation, or can be any other physiological parameter.
The controller 220 can be configured to operate the fingerprint sensor 290 in a fingerprint reader mode that can include capturing a live scan of a fingerprint pattern from a finger on the touch surface to authenticate a user of the apparatus 200. The controller 220 can be configured to change fingerprint sensor parameters to switch the fingerprint sensor 290 from the fingerprint reader mode to a physiological parameter determination mode.
At 330, image characteristics corresponding to at least a first image of the plurality of images can be compared with image characteristics corresponding to at least a second image of the plurality of images. Image characteristics can be compared by comparing differences between image characteristics corresponding to at least the first image of the plurality of images and image characteristics corresponding to at least the second image of the plurality of images. Image characteristics can be compared by comparing image characteristics corresponding to a first intensity of at least the first image with image characteristics corresponding to a second intensity of at least the second image, where a difference between the first intensity and the second intensity can correspond to changes in density of blood flow.
At 340, a physiological parameter can be determined based on comparing the image characteristics. The physiological parameter can be a blood related physiological parameter, such as heart rate, oxygen consumption, maximal oxygen consumption, blood oxygenation, or can be any other physiological parameter. At 350, the flowchart 300 can end.
At 420, fingerprint sensor parameters can be changed to switch the fingerprint sensor from the fingerprint reader mode to a physiological parameter determination mode. The physiological parameter determination mode can constantly or continually capture a plurality of images, such as multiple images and/or a video sequence, of a finger pressed in a substantially constant position against the fingerprint sensor. The changed fingerprint sensor parameters can include white balance, exposure time, frame rate, focal point, and other fingerprint sensor parameters that can be changed to allow a fingerprint sensor to determine one or more physiological parameters. For example, the fingerprint sensor can operate in the physiological parameter determination mode by operating the fingerprint sensor at a second focal point different from the fingerprint reader mode first focal point. The second focal point can correspond to an area at a distance beneath the surface of skin to operate the fingerprint sensor as a physiological parameter monitor. The area at a distance beneath the surface of the skin can include one or more capillaries. The fingerprint sensor can also operate in the physiological parameter determination mode by increasing the exposure time from the fingerprint reader mode first exposure time to a second exposure time to operate the fingerprint sensor as a physiological parameter monitor. The fingerprint sensor can also operate in the physiological parameter determination mode by increasing the number of frames per second from the fingerprint reader mode first number of frames per second to a second number of frames per second to operate the fingerprint sensor as a physiological parameter monitor.
At 425, a plurality of images corresponding to an area beneath a surface of skin can be captured using the fingerprint sensor.
At 430, a plurality of binary images can be created based on the plurality of captured images. The plurality of binary images can contain information corresponding to pixels exceeding a threshold. For example, the threshold can be used to convert individual sub-images into corresponding binary images. The plurality of binary images can be created by processing the plurality of captured images, can be created from decomposed sub-images, can be created from a plurality of captured and decomposed images, or can otherwise be based on the plurality of captured images. As a further example, the plurality of binary images can be created by extracting raw Bayer images based on the plurality of captured images.
At 435, dimensional content of the plurality of binary images can be reduced to generate reduced dimension information. Reducing the dimensional content of the plurality of binary images can include summing a number of pixels exceeding the threshold for each image of the plurality of binary images to generate reduced dimension information for each image. The dimensionality reduction can be due to summing of binarized pixel values for each of the sub-images, hence giving the area.
At 440, image characteristics corresponding to at least a first image of the plurality of images can be compared with image characteristics corresponding to at least a second image of the plurality of images. Image characteristics can be compared by comparing image characteristics corresponding to at least a first binary image with image characteristics corresponding to at least a second binary image. Image characteristics can also be compared by comparing image characteristics corresponding to the reduced dimension information based on at least the first binary image with image characteristics corresponding the reduced dimension information based on at least the second binary image. Low frequencies of the reduced dimension information can be filtered out prior to comparing the image characteristics.
At 445, a physiological parameter can be determined based on comparing the image characteristics. The physiological parameter can be a blood related physiological parameter, such as heart rate, oxygen consumption, maximal oxygen consumption, blood oxygenation, or can be any other physiological parameter. At 450, a user of the portable communication device can be identified based on information corresponding to the image characteristics. For example, characteristics corresponding to the physiological parameter can be used as a primary form of authentication or as a secondary form of authentication along with a fingerprint pattern. Some level of authentication can be extracted from a signal used to extract the physiological parameter to provide a primary or secondary form of identification.
At 455, data corresponding to the physiological parameter can be output. The data can be output by displaying the physiological parameter on a display on the portable electronic device. Multiple physiological parameters can be displayed on the display. The data can also be output by storing data corresponding to the physiological parameter. The data can also be output by wirelessly transmitting data corresponding to the physiological parameter across a cellular communications network to a server. For example, the portable electronic device can store the data on the portable electronic device and/or can store it at a server. At 460, the flowchart can end.
The methods of this disclosure are preferably implemented on a programmed processor. However, the operations of the embodiments may also be implemented on a general purpose or special purpose computer, a programmed microprocessor or microcontroller and peripheral integrated circuit elements, an integrated circuit, a hardware electronic or logic circuit such as a discrete element circuit, a programmable logic device, or the like. In general, any device on which resides a finite state machine capable of implementing the operations of the embodiments may be used to implement the processor functions of this disclosure.
While this disclosure has been described with specific embodiments thereof, it is evident that many alternatives, modifications, and variations will be apparent to those skilled in the art. For example, various components of the embodiments may be interchanged, added, or substituted in the other embodiments. Also, all of the elements of each figure are not necessary for operation of the disclosed embodiments. For example, one of ordinary skill in the art of the disclosed embodiments would be enabled to make and use the teachings of the disclosure by simply employing the elements of the independent claims. Accordingly, the preferred embodiments of the disclosure as set forth herein are intended to be illustrative, not limiting. Various changes may be made without departing from the spirit and scope of the disclosure.
In this document, relational terms such as “first,” “second,” and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, relational terms, such as “top,” “bottom,” “front,” “back,” “horizontal,” “vertical,” and the like may be used solely to distinguish a spatial orientation of elements relative to each other and without necessarily implying a spatial orientation relative to any other physical coordinate system. The terms “comprises,” “comprising,” or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by “a,” “an,” or the like does not, without more constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element. Also, the term “another” is defined as at least a second or more. The terms “including,” “having,” and the like, as used herein, are defined as “comprising.”
| Number | Name | Date | Kind |
|---|---|---|---|
| 5148809 | Biegeleisen-Knight | Sep 1992 | A |
| 5719950 | Osten et al. | Feb 1998 | A |
| 6176832 | Habu et al. | Jan 2001 | B1 |
| 6292576 | Brownlee | Sep 2001 | B1 |
| 6483929 | Murakami et al. | Nov 2002 | B1 |
| 6616613 | Goodman | Sep 2003 | B1 |
| 7018338 | Vetter et al. | Mar 2006 | B2 |
| 7254255 | Dennis | Aug 2007 | B2 |
| 7336982 | Yoo | Feb 2008 | B2 |
| 7590269 | Creasey et al. | Sep 2009 | B2 |
| 8229185 | Ennis et al. | Jul 2012 | B2 |
| 20030233051 | Verjus et al. | Dec 2003 | A1 |
| 20040208348 | Baharav et al. | Oct 2004 | A1 |
| 20050089202 | Setlak et al. | Apr 2005 | A1 |
| 20060009700 | Brumfield et al. | Jan 2006 | A1 |
| 20060015021 | Cheng | Jan 2006 | A1 |
| 20060067887 | Guittet et al. | Mar 2006 | A1 |
| 20060115134 | Kozlay | Jun 2006 | A1 |
| 20060211930 | Schaf et al. | Sep 2006 | A1 |
| 20070032725 | Watanabe et al. | Feb 2007 | A1 |
| 20070106183 | Suzuki et al. | May 2007 | A1 |
| 20070160283 | Saphier et al. | Jul 2007 | A1 |
| 20070219439 | Vilser et al. | Sep 2007 | A1 |
| 20080021331 | Grinvald et al. | Jan 2008 | A1 |
| 20080132798 | Hong et al. | Jun 2008 | A1 |
| 20080273768 | Dennis et al. | Nov 2008 | A1 |
| 20090203998 | Klinghult et al. | Aug 2009 | A1 |
| 20090226071 | Schuler et al. | Sep 2009 | A1 |
| 20090281435 | Ahmed et al. | Nov 2009 | A1 |
| 20100166252 | Ahmed et al. | Jul 2010 | A1 |
| 20120078066 | Ahmed et al. | Mar 2012 | A1 |
| 20120078119 | Ahmed et al. | Mar 2012 | A1 |
| 20120078120 | Ahmed et al. | Mar 2012 | A1 |
| 20120078130 | Ahmed et al. | Mar 2012 | A1 |
| Number | Date | Country |
|---|---|---|
| 102007017713 | Oct 2008 | DE |
| 1353292 | Oct 2003 | EP |
| 1988489 | Nov 2008 | EP |
| 0124700 | Apr 2001 | WO |
| 0209049 | Jan 2002 | WO |
| Entry |
|---|
| Imtiaz Ahmed; U.S. Appl. No. 12/347,400, filed Dec. 31, 2008. |
| Francesca Schuler; U.S. Appl. No. 12/043,515, filed Mar. 6, 2008. |
| http://en.wikipedia.org/wiki/VO2. |
| http://en.wikipedia.org/wiki/Fingerprint—reader. |
| http://www.amazon.com/Mark-Fitness-MF-180-Exercise-Monitor/dp/B0002VG510. |
| http://www.amazon.com/Insta-Pulse-105-MonitorInstapulse/dp/B000FJ02PO/ref=sr. |
| http://www.cosycommunications.com/Digitalpulsemonitorcatalogue.htm. |
| http://www.amazon.com/Reebok-Fitwatch-StraplessHeartMonitor/dp/B000G37H0M/ref=sr—1—1?ie=UTF8&s=hpc&qid=1238427454&sr=1-1. |
| International Search Report Issued Jul. 16, 2010, in connection with related PCT Application No. PCT/US2010/026943. |
| D. Maltoni et al; “Handbook of Fingerprint Recognition, Passage” Jan. 1, 2003; Handbook of Fingerprint Recognition; [Springer Professional Computing], Springer Verlag, New York, US, pp. 281-292. |
| Davide Maltoni et al.; “Handbook of Fingerprint Recognition” 2003; Springer Professional Computing; XP002590073; pp. 26-29. |
| Jia Zheng, et al, “Remote Simultaneous Dual Wavelength Imagining Photoplethysmography: A Further Step Towards 3-D Mapping of Skin Blood Microcirculation”, XP-002526642, 2008, 8 pages, United Kingdom. |
| K. Nakajima, et al, “Monitoring of Heart and Respiratory Rates by Photoplethysmography Using a Digital Filtering Technique”, XP-002446273, Jul. 1996, vol. 18, pp. 365-372, Great Britian. |
| Rolando Hong Enriquez, et al, “Analysis of the Photoplethysmographic Signal by Means of the Decomposition in Principal Components”, Jun. 14, 2002, 14 pages, United Kingdom. |
| Chihiro Takano and Yuji Ohta, “Heart Rate Measurement Based on a Time-Lapse Image”, Sep. 19, 2006, 5 pages. |
| F.P. Wieringa, et al., “Contactless Multiple Wavelength Photoplethysmographic Imagining: A First Step Toward SpO2 Camara Technology”, Aug. 2005, vol. 22, No. 8, 8 pages. |
| Wim Verkruysse, et al., “Remote Plethysmographic Imagining Using Ambient Light”, Optic Society of America, Dec. 22, 2008, vol. 16, No. 26, Norway. |
| European Patent Office, “Communication”, for European Pat. Appln. No. 09718414.7, Aug. 3, 2011, 9 pages. |
| Patent Cooperation Treaty, “PCT Search Report and Written Opinion of the International Searching Authority” for International Application No. PCT/US2009/035836, Aug. 24, 2009, 10 pages. |
| United States Patent and Trademark Office, “Non-Final Rejection”, for U.S. Appl. No. 12/043,515 dated Aug. 17, 2011, 8 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20100245553 A1 | Sep 2010 | US |
