Patent Application
20200358925
This disclosure relates to apparatus for assessing a user's health status by analyzing collected data.
Behavior and body motion are key indicators of both mental and physical health. Images of an individual may reveal changes in skin color and areas of unusual pigmentation, changes in body composition, changes in the individual's gait, and behavioral patterns. Examples of conditions which may be identified by assessing color on a patient include cyanosis, anemia, jaundice, rashes, skin lesions, and skin cancers. Examples of conditions which affect movement and behavior include depression, stroke, tremors, joint damage, fatigue, anxiety, and many more. Theoretically, analysis of video and still photos collected in an individual's home or care center could offer the possibility of early diagnosis for many conditions. However, health tracking and monitoring of patients at home by still photo and video poses both security and privacy concerns, particularly when transmitting the data to a remote healthcare provider.
Analysis of still photos and video could be analyzed using telemedicine techniques. A goal of telemedicine is to provide continuous unobtrusive monitoring of users and early alert, detection, and monitoring of progression of health conditions. Machine learning algorithms can recognize users from still images and video and classify behavior, movement, morphology, coloring, and other observations. However, many users may feel uncomfortable with being observed continuously by video in a home, work, or care center environment. In addition, collecting such data may represent a safety concern. Furthermore, video image of users, if intercepted, could pose a threat to security. In an example, video of someone typing a password into their home computer could result in theft. A health-monitoring system is needed which collects visual data of a user in the user's natural environment while safeguarding the user's privacy, safety, and security.
Just a few examples of smart toilets and other bathroom devices can be seen in the following U.S. Patents and Published Applications: U.S. Pat. No. 9,867,513, entitled “Medical Toilet With User Authentication”; U.S. Pat. No. 10,123,784, entitled “In Situ Specimen Collection Receptacle In A Toilet And Being In Communication With A Spectral Analyzer”; U.S. Pat. No. 10,273,674, entitled “Toilet Bowl For Separating Fecal Matter And Urine For Collection And Analysis”; US 2016/0000378, entitled “Human Health Property Monitoring System”; US 2018/0020984, entitled “Method Of Monitoring Health While Using A Toilet”; US 2018/0055488, entitled “Toilet Volatile Organic Compound Analysis System For Urine”; US 2018/0078191, entitled “Medical Toilet For Collecting And Analyzing Multiple Metrics”; US 2018/0140284, entitled “Medical Toilet With User Customized Health Metric Validation System”; and US 2018/0165417, entitled “Bathroom Telemedicine Station.” The disclosures of all these patents and applications are incorporated by reference in their entireties.
We disclose a health-monitoring apparatus which discretely and unobtrusively collects graphic data. This graphic data may be used to assess the user's health status. The apparatus may include a camera which may be positioned such that it may collect graphic data of a user. The apparatus may include a controller which has a memory and a communications port. The camera may be in electronic communication, either wired or wireless, with the controller. A memory within the controller may store instructions for transmitting graphic data to a remote database, for example, a cloud database. The remote database may store non-transitory computer-readable media which includes instructions for applying an algorithm, in some embodiments, a machine-learning algorithm, to the graphic data. The algorithm may create an analysis of the graphic data and an assessment of a user's health status. In some embodiments, the algorithm also prepares a report describing the status of the user's health and transmits it to the user or the user's healthcare provider.
The apparatus may include a light source which may direct light toward the camera's angle of view. More specifically, the light source may direct the light toward the user as the camera collects graphic data including images and video of the user. Depending on the type of health assessment to be made, the light may include a defined range of wavelengths and patterns of light. Examples of ranges of wavelengths and patterns of light include, but are not limited to, visible light, high color temperature light, infrared light, structured light, and modulated light.
A variety of types of cameras may be included in the apparatus. The apparatus may include more than one camera and more than one type of camera. Examples of types of cameras which may be included in the apparatus include, but are not limited to, a 3D time of flight camera, a stereoscopic camera, an infrared thermal imaging camera, a video camera, a structured light 3D scanner, and a still image camera. These different cameras may collect different graphic data which may have different diagnostic uses. For example, the apparatus may use different cameras and different types of light to assess the speed of user movement, user movement patterns, user posture, swelling beneath the user's eyes, swollen lymphatic glands, visual photoplethysmography, heart rate, moles, skin growths, body shape, skin coloration, sclera coloration, hair loss, breathing rate, time in front of the camera, and body shape.
In addition to a fixture, the camera may be disposed within a variety of diagnostic tools. These may include an otoscope, an ophthalmoscope, an endoscope, a laparoscope, a laryngoscope, a colposcope, a hysteroscope, a bronchoscope, a pharyngoscope, a laparoscope, or a dental tool.
In some embodiments, the apparatus includes an auxiliary sensor which detects the presence of a user and actuates the camera, the light source, or both to collect graphic data relating to the user. In some embodiments, the auxiliary sensor is a motion detector. In some embodiments, the auxiliary sensor is a pressure sensor. In some embodiments, the pressure sensor is a floor scale which the user may stand on as the user approaches the camera.
In some embodiments the controller includes a graphics processing unit which includes instructions for performing multiple nonlinear transformations on the graphic data the camera collects. These transformations convert the graphic data to obfuscated graphic data representing obfuscated images which a human view cannot recognize as the image or video originally collected by the camera. However, the obfuscated graphic data retains at least one feature that a machine-learning algorithm can recognize. Non-transitory computer-readable medium stored on the remote database may include instructions for applying a machine-learning algorithm to the transformed graphic data without reconstructing the original image or video. The machine-learning algorithm may create an analysis of the obfuscated graphic data and an assessment of the user's health status.
Examples of techniques which may be used to transform the graphic data collected by the camera into obfuscated graphic data include deep convolution, compressed sensing obfuscation in which a truncated sparse basis expansion is used, and block-chain based obfuscation. In the block-chain based obfuscation technique, difference hashing between frames may be secured via blockchain.
By transmitting the data to the remote server as obfuscated graphic data, the user's privacy is preserved. In addition, the safety and security of the transmitted graphic data is enhanced relative to transmitting non-transformed graphic data.
Non-limiting and non-exhaustive implementations of the disclosure are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified. Advantages of the disclosure will become better understood regarding the following description and accompanying drawings where:
The following description recites various aspects and embodiments of the inventions disclosed herein. No particular embodiment is intended to define the scope of the invention. Rather, the embodiments provide non-limiting examples of various compositions, and methods that are included within the scope of the claimed inventions. The description is to be read from the perspective of one of ordinary skill in the art. Therefore, information that is well known to the ordinarily skilled artisan is not necessarily included.
The following terms and phrases have the meanings indicated below, unless otherwise provided herein. This disclosure may employ other terms and phrases not expressly defined herein. Such other terms and phrases shall have the meanings that they would possess within the context of this disclosure to those of ordinary skill in the art. In some instances, a term or phrase may be defined in the singular or plural. In such instances, it is understood that any term in the singular may include its plural counterpart and vice versa, unless expressly indicated to the contrary.
As used herein, the singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. For example, reference to “a substituent” encompasses a single substituent as well as two or more substituents, and the like.
As used herein, “for example,” “for instance,” “such as,” or “including” are meant to introduce examples that further clarify more general subject matter. Unless otherwise expressly indicated, such examples are provided only as an aid for understanding embodiments illustrated in the present disclosure, and are not meant to be limiting in any fashion. Nor do these phrases indicate any kind of preference for the disclosed embodiment.
As used herein, “safe” means that data which could reasonably be exploited by actors with malicious intent is prevented from being accessed by such actors or their agents.
As used herein, “secure” means only authorized parties receive data.
As used herein, “private” means the data is kept exclusively within the health monitoring system to which the user has subscribed.
As used herein, “user” means the individual from whom the disclosed system is collecting graphical data.
As used herein, “electronic” means either wired or wireless. For example, the phase “in electronic communication” means either a wired communication between two devices or could mean a wireless communication between devices, such as WiFi.
While this invention is susceptible of embodiment in many different forms, there are shown in the drawings, which will herein be described in detail, several specific embodiments with the understanding that the present disclosure is to be considered as an exemplification of the principals of the invention and is not intended to limit the invention to the illustrated embodiments.
We disclose a system for discrete and unobtrusive health monitoring using video, still photos, or both. The system may be used in the home, care facility, or clinical setting. The system may include one or more cameras and a controller. The controller may include a memory and be in electronic communication with the camera. The electronic communication may be either wired or wireless. The one or more cameras may collect graphic data and transmit the graphic data to the controller.
The controller may include a memory which may house non-transitory computer-readable medium. The non-transitory computer-readable medium may include instructions for applying an algorithm, which in some embodiments is a machine-learning algorithm, to create an analysis of the graphic data. The algorithm may further convert the analysis of the graphical data to an assessment of the user's health status. Alternatively, separate algorithms may perform the tasks of directly analyzing the graphic data and creating an assessment of the user's health. An algorithm may also create a report of the assessment of the user's health and transmit the report to the user or the user's healthcare provider.
The controller may also include a communication port which is in electronic communication with the controller, either wired or wirelessly. The controller may include non-transitory computer-readable medium which includes instructions to transmit the graphic data through the communications port. The communication port may transmit the graphic data to a remote server, for example, a cloud computing resource. Graphic data may be stored on the remote server. The remote server may store and implement non-transitory computer-readable medium which includes instructions for applying one or more machine-learning algorithms to create an analysis of the graphic data and to create an assessment of a user's health status. In some embodiments, the remote server is secure, safe, private, or a combination thereof.
Some embodiments may include a light source and a light source modulator. The light source modulator may direct light toward an angle of view of the camera. The light provided by the light source may enable the camera to collect better quality still or video images which are more readily analyzable. The modulator may select the wavelength of light or other characteristics of the light provided by the light source. For example, the light source may provide one or more of the following types of light: visible light, high color temperature light, infrared light, structured light, and modulated light. In an example, visible light may be useful to illuminate the subject to record movement and behavior. Controlled wavelength illumination may allow a degree of depth profiling. Infrared illumination may reveal vascular structure deeper in the skin. High color temperature (blue-tint) light may be useful to detect features on the skin surface. Infrared illumination may be used to record the user's motion in an otherwise unlit environment. Stereoscopy, time of flight, or structured lighting may be used to collect 3D information about the user.
The camera may be a single camera. Alternatively, the system may include a plurality of cameras. Each camera may include, but is not limited to, one or more of the following types of cameras: a 3D time of flight camera, a stereoscopic camera, an infrared thermal imaging camera, a video camera, a structured light 3D scanner, and a still image camera.
In an example, a camera is disposed within an otoscope, an ophthalmoscope, an endoscope, a laparoscope, a laryngoscope, a colposcope, a hysteroscope, a bronchoscope, a pharyngoscope, a laparoscope, or a dental tool. In some examples, the camera in these devices is a perspective camera.
In some embodiments, the camera may be disposed within a fixture within the room. By disposing the camera within a fixture, the camera is unobtrusive to the user and therefore, unlikely to cause the user to modify his or her behavior due to self-consciousness when the camera is collecting graphic data. In an example, the fixture may be placed in a bathroom, a kitchen, a hallway, an entryway, an office, or a living room.
In an example, the fixture includes a partially silvered mirror. The mirror may be disposed between the user and the lens of the camera (in front of the lens of the camera). The user may view himself or herself in the mirror in a traditional manner while the camera collects graphic data from the user. The mirror may appear no differently to a user than a traditional mirror. This characteristic may prevent the user from behaving differently because the user is focusing on the fact that the camera is collecting graphical data. In another example, a small window is disposed in the fixture and over the camera lens so that the camera may collect graphic data on the other side of the fixture.
In addition to the one or more cameras, the system may include one or more auxiliary sensors, each which may be in electronic communication with the controller. The non-transitory computer-readable medium on the controller may include instructions to actuate the health-monitoring apparatus when the controller receives a signal from an auxiliary sensor. In an example, an auxiliary sensor may be a pressure sensor. In a more specific example, the pressure sensor may be placed on or within flooring in the environment surrounding the one or more cameras. In some embodiments, the pressure sensor may be a floor scale or pressure sensitive floor mat. When a user enters the environment and crosses over the pressure sensor, the pressure sensor may detect the user's mass and send a signal to the controller. In another example, the auxiliary sensor may be a motion detector which sends a signal to the controller when it detects the presence of a user in the environment. In the embodiments which include an auxiliary sensor, the user does not need to consciously actuate the system. The system's presence may, therefore, be less noticeable. Consequently, the user may be less likely to modify his or her behavior due to self-consciousness about being recorded by the camera.
In some embodiments, the system may include an interactive display for the user. In the embodiment which includes a mirror, the display may appear on the mirror. The display may provide health data to the user, as measured in real time (such as the measured weight from the floor scale), as well as health data which is the result of the analysis performed by the system. The display may also include indicators of when images are being captured and when the process has completed. The display may also be connected to other information sources, such as a smart home system, and may be used to display non-health related data, such as weather, schedule or news.
The one or more cameras may record a plurality of clinically relevant observations. These may include behaviors and movements, for example, the speed of the user's movement, the patterns of the user's movements, the user's posture, the time of day that the user appears in front of the camera, uncharacteristic use of the non-dominant hand, and the amount time the user spends in front of the camera. Observations relating to behavior and movement may assist in the diagnosis of psychological, neurological, skeletal, and motor aberrations.
The one or more cameras may also record physiological changes in a user including, but not limited to, edema including swelling beneath the user's eyes, swollen lymphatic glands, visual photoplethysmography, heart rate, moles, skin growths, skin coloration, sclera coloration, mucous membrane coloration, coloration of the ear canal, hair loss, breathing rate, and body shape.
In an example, the camera may be disposed within an otoscope and collect graphic data which may be used to identify inflammation within the external auditory canal.
In another example, the camera may collect graphic data which may be used to assess the transparency of the lens of the user's eye. If the lens is relatively opaque, the assessment of the user's health status may indicate a possible cataract. Scratches and abrasions on the user's eye may also be detected. In such embodiments, the camera may be disposed within an ophthalmoscope or behind a mirror. In the example in which the camera is disposed behind a mirror, the mirror may be a bathroom mirror which the user spends time standing directly in front of and facing while performing toiletry tasks, for example, hair and teeth brushing. Consequently, the camera may be able to collect images of the user's eyes as the user peers directly into the mirror.
In an example in which the camera is disposed within a fixture, for example, behind a mirror, the camera may detect coloration in the user's skin. A yellow coloration may indicate jaundice while a blue coloration may indicate cyanosis. A pale coloration may indicate pallor which is a symptom of emotional stress, anemia, and other pathologies. The camera may measure the color and shape of a mole or skin growth leading to a dermatological diagnosis. The camera may record movement in the user's chest which may be used to calculate the user's breathing rate.
In some embodiments, the controller further includes a graphic processing unit (hereinafter, “GPU”). The GPU may include non-transitory computer-readable medium which includes instructions for performing algorithms which transform and obfuscate the graphic data the camera collects. The algorithms may perform multiple convolutions or nonlinear transformations on the graphic data which convert the graphic data to a form which is unrecognizable by a human viewer, but which also retains a feature recognizable by the machine-learning algorithm. The graphic data may be transformed into several kinds of transformed data by several different transformations, each transformation preserving a feature recognizable or useful for a machine-learning or computer classification algorithm. The graphic data is now obfuscated graphic data. In addition to being unrecognizable by humans, the obfuscated graphic data may not be converted back to the original graphic data by mathematical inversion or available computational methods. Examples of techniques which may be used to transform the graphic data include deep convolution, compressed sensing obfuscation using sparse basis expansion and discarding basis functions; and block-chain based obfuscation. In the block-chain obfuscation technique, difference hashing between frames may be secured via blockchain.
Obfuscation may be used to preserve safety, security, and user privacy. After the graphic data has been transformed, it may be transmitted to a remote processor which may apply algorithms to assess the status of the user's health. Because the graphic data has been transformed as described herein, the system maintains the user's privacy and the safety and security of the data is enhanced relative to graphic data which has not been transformed.
An example of an obfuscating, non-linear transformation is a multi-layer convolution network with non-linear pooling layers. A series of 2D kernels may be applied to each 2D frame of a video or still image collection using convolution, possible varying the padding and step size. This may produce a set of convolved images for each image (one for each kernel). At this point, it is still possible for an inverse convolution to reconstruct the original image. However, a pooling layer is then applied whereby the size of each convolved image is reduced by taking the average or maximum absolute value of all pixels within a neighborhood; for example, a group of 4 pixels may be replaced by a single pixel that has the maximum value of any of the original 4 pixels. This step is non-linear and cannot be exactly inverted. Typical implementations may have an alternating succession of 2D convolution and pooling layers to produce a final set of convolved images that are flattened to a single vector. This vector may encode information that may be useful to a machine-learning algorithm, but will appear to be random noise to a human viewer.
Another approach to obfuscating the video stream involves expanding each video frame using a sparse representation (for example, wavelet or discrete cosine) using an algorithm which may include compressed sensing techniques. These sparse regression algorithms penalize solutions to have a minimal l_1 norm, which preferentially selects sparse solutions. Then, machine-learning algorithms may be given access to data in the transformed space to train on. Only those basis functions that are relevant to the machine learning model are selected, allowing a large number of basis functions to be discarded. This loss of basis function renders the images unrecognizable.
Another way to obfuscate graphic data collected by the camera within the disclosed system is using a blockchain-based system. In this example, the user and server exchange public keys. By giving the server the user's public key, the user is granting permission for the server to access the obfuscated video stream. A difference hash is computed for each frame in the video relative to its neighboring frames. This hash is then combined with the user's private key and the server public key to generate a transaction in a blockchain ledger. Each frame captured and sent to the server is traceable to a transaction that was approved by the user for use by the server. Only the user or the server possess the keys to decrypt the difference hash in the video frames.
Referring now to the drawings,
Should the image in
While specific embodiments have been illustrated and described above, it is to be understood that the disclosure provided is not limited to the precise configuration, steps, and components disclosed. Various modifications, changes, and variations apparent to those of skill in the art may be made in the arrangement, operation, and details of the methods and systems disclosed, with the aid of the present disclosure.
Without further elaboration, it is believed that one skilled in the art can use the preceding description to utilize the present disclosure to its fullest extent. The examples and embodiments disclosed herein are to be construed as merely illustrative and exemplary and not a limitation of the scope of the present disclosure in any way. It will be apparent to those having skill in the art that changes may be made to the details of the above-described embodiments without departing from the underlying principles of the disclosure herein.
This application claims priority to PCT International Patent Application Number PCT/US2018/13836 titled “Health Monitoring System Including Privacy-Ensuring Obfuscated Camera Images” filed on 16 Jan. 2018, which is incorporated herein by reference in its entirety.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2018/013836 | 1/16/2018 | WO | 00 |
