Patent Application
20200138337
Various embodiments of the present technology generally relate to systems and methods for monitoring breathing activity and identifying various pulmonary conditions. More specifically, some embodiments of the present technology relate to non-contact breathing activity monitoring and analyzing system through thermal on projection medium imaging.
Respiration monitoring techniques that are both accurate and comfortable for the patient are sorely lacking from the medical field. The most accurate methods of respiration monitoring may require placing ECG electrodes on the patient's body, putting thermistors in the patient's nose, having the patient wear an abdominal strain-gauge transducer, having the patient breathe through a tube while wearing a nose clip, or some combination of these techniques that monitor multiple biophysiological parameters concurrently. These traditional solutions all involve placing sensors directly on the patient's body. These direct measurements have a high rate of accuracy, but cause discomfort and alter the natural breathing of the patient.
Pulmonologists employ a variety of respiratory monitoring tools to accurately assess the health of a patient, such as spirometers, plethysmographs, and polysomnography. Termed “contact methods”, these techniques require placing sensors directly on the body of the subject or having the subject breath through a tube-based device. Though these methods boast high accuracy and utility, these methods often inflict physical and psychological discomfort, and interfere with a patient's natural breathing behaviors, limiting their application for long-term monitoring. Additionally, cumbersome equipment and labor-intensive setups make these techniques difficult to utilize in out-patient clinics.
Many proposed non-contact measurement methods use remote sensors such as thermal cameras, RGB cameras, depth sensors, ultrasonic sensors, among other imaging devices. Though inherently comfortable, they are not currently used due to their lowered accuracy, sensitivity to bodily characteristics, and limited utility in practical clinical settings. Thermal imaging methods that monitor breathing in open air have reduced accuracy due to fast heat dissipation, and methods that measure skin temperature changes are unable to provide detailed breathing behavior information. As such, there are a number of challenges and inefficiencies created in traditional respiratory analysis systems. It is with respect to these and other problems that embodiments of the present technology have been made.
Various embodiments of the present technology generally relate to systems and methods for monitoring breathing activity. More specifically, some embodiments of the present technology relate to non-contact breathing activity monitoring and analyzing through thermal imaging of thin-medium surfaces. Thin-medium thermal imaging refers to the visual collection of exhaled heat signatures obtained on a thin-medium imaged by a thermal camera. Various embodiments provide for systems and methods based on this premise for the extraction of clinically meaningful metrics for respiratory analysis. This includes, but is not limited to, the reconstruction of exhale flows to obtain volumetric estimates of exhaled breath, the identification of the separation between nose (each nostril) and mouth exhale flows to measure distribution, and both the velocity and strength of exhale flows.
Various embodiments of the present technology provide for a system comprising an infrared (IR) thermal camera. The IR thermal camera can be combined with a laser transmitter (Tx) and receiver (Rx) which can be used to measure the distance between the thermal camera and the thin-medium (e.g., formed in a planar or a curved surface). In some embodiments the IR thermal camera can be configured to image a subset interval of the electromagnetic spectrum within the infrared wavelengths and generate grayscale intensity values within a dense pixel array that represent the thermal intensity distribution of exhale behaviors within the thin-medium.
The system can define a configuration where exhale characteristics and behaviors are identified through the visualization of heated exhale interacting with the thin medium (e.g., plastics, polymers, metals, fibers, or other synthetic materials) which is imaged from the opposing side or from surrounding directions of the thin-medium. In some embodiments, a chemical coating may be applied to the thin-medium to enhance the thermal residual and dissipative characteristics of the medium to modify the thermal signature imposed on the material from external heat sources. The chemical coating may alter the material's ability to maintain heat residuals or alter the material's dissipative characteristics. In some embodiments, the thin medium can be selected to maintain a thermal radiance above ambient temperature for a given period of time.
In some embodiments, the IR thermal camera, distance laser, and thin-medium can be combined into a mobile or wearable device. The camera can be placed to allow imaging of a large portion (e.g., 60% or more) of the thin-medium for recording the thermal changes in the thin-medium. In other embodiments, the thermal camera, laser distance system, and thin-medium may be mounted within an adjustable mount that provides a targetable vision system that tracks thermal changes in the thin-medium.
Various embodiments may also include processing units, processors, or computers to execute image processing and respiratory metric algorithms. In some embodiments, the computing resources can be integrated within the device or may be external to the device and communicably coupled via a wireless link.
In some embodiments, a system can include a thermal camera that images projections of an exhale cross-section on a projection-medium, a communication interface coupled to the thermal camera to receive a sequence of thermal images depicting the thermal distribution of the exhale on the thin-medium, a processor, and a set of instructions stored thereon that when executed by the processor, cause the processor to generate a representation or model of the exhale of the subject. Various metrics of the respiratory behavior, two- and three-dimensional representations of the exhale cross-sections, 3D reconstruction, a geometrically consistent estimation of the approximate flow volume via a reconstruction on a per-frame basis to provide an estimate of the subject's tidal volume, and/or the like can also be generated.
In some embodiments, a method of analysis for monitoring and extracting metrics related to pulmonary function can include taking thermal images of a thin-medium while the user breathes onto the surface of the medium. A temporary representation of the exhale signal on the medium that represents the exhale behaviors of the user can be created. Then, for each intensity value range within the image, iso-lines to form an iso-line map of gradients and exhale boundary captured over time can be created to form a three-dimensional representation of the exhale. Respiratory information can be derived through applying image processing techniques to the thermal gradient images. In addition, a correlation between the intensity values and exhale behavior over time with objective metrics may also be derived through machine learning.
Embodiments of the present technology also include computer-readable storage media containing sets of instructions to cause one or more processors to perform the methods, variations of the methods, and other operations described herein.
In some embodiments, the system can generate metrics that describe pulmonological function that include: reconstructions of exhale flow behavior by (1) imaging the thin-medium using the thermal camera to produce a video stream composing a sequence of images, (2) extracting 2D cross-sectional regions of the exhale behavior within the medium for each image, (3) generating a model based on these cross sections over time. This model can then be used to estimate various pulmonological traits including: 3D reconstructions of the exhale volume used to obtain tidal volume estimates, identifying the separation between nose, mouth, and nostrils, and exhale strength over time. In some embodiments, the image sequence produced by the thermal camera can be used to generate a sequence of contours that are extracted from the 2D intensity images. Each contour can be identified temporally by a time-stamp that can be used to define these 2D contours over time. The sequence of contours is then projected over time to define a 3D volume. The 3D volumetric model can then be used to provide estimates of exhale behaviors and tidal volume. Exhale can be separated into a distribution between each nostril and the mouth. This distribution is represented as a percentage (%) relationship. The behaviors of the contours and thermal distribution within the thin-medium are used to identify a relational proportion between each nostril and the mouth. The duration and thermal distribution of an exhale can be analyzed to extract the strength of the exhale. Strength may be defined as a relationship between velocity, force, and duration of the exhale.
Some embodiments may include an additional thermal camera to capture a side profile of the exhale. The additional thermal camera may include a wavelength filter tuned to a spectral range subset of the infrared spectrum used to visualize CO2 from exhaled airflows from a side profile. This is used as a ground-truth device with which to validate metrics collected from the medium. The thermal CO2 camera may be positioned perpendicular to the linear setup of the thermal camera and thin-medium. This can provide a method for visualizing the exhale as it exits the nose/mouth and makes contact with the thin-medium, imparting thermal energy into the medium. The flow delay and nose/mouth distribution ground-truth values can then be correlated with the behaviors identified on the thin-medium. Measurements obtained from the side view using the thermal CO2 camera are used to improve the accuracy of the apparent behavior identified in the thin medium using the thermal camera. This correlation can be established in some embodiments by training a deep neural network with the relationship of the exhale metrics between those obtained with the thermal CO2 camera and the behaviors visualized on the thin-medium.
While multiple embodiments are disclosed, still other embodiments of the present technology will become apparent to those skilled in the art from the following detailed description, which shows and describes illustrative embodiments of the technology. As will be realized, the technology is capable of modifications in various aspects, all without departing from the scope of the present technology. Accordingly, the drawings and detailed description are to be regarded as illustrative in nature and not restrictive.
Embodiments of the present technology will be described and explained through the use of the accompanying drawings in which:
The drawings have not necessarily been drawn to scale. Similarly, some components and/or operations may be separated into different blocks or combined into a single block for the purposes of discussion of some of the embodiments of the present technology. Moreover, while the technology is amenable to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and are described in detail below. The intention, however, is not to limit the technology to the particular embodiments described. On the contrary, the technology is intended to cover all modifications, equivalents, and alternatives falling within the scope of the technology as defined by the appended claims.
Various embodiments of the present technology generally relate to systems and methods for monitoring breathing activity. More specifically, some embodiments of the present technology relate to non-contact breathing activity monitoring and analyzing through Thin-Medium Thermal Imaging (TMTI). In this respiratory monitoring and diagnosis system, breathing behaviors are captured as they contact a thin planar surface that is then imaged using a thermal camera. The heat distribution imposed on the thin-medium is then used to extract respiratory metrics. Conventional breath monitoring systems need to place devices on patients, which causes discomfort and may not be applicable for long term monitoring. Moreover, the primary evaluation criteria within respiratory analysis revolves around the collection of a limited set of quantitative metrics such as breathing rate, flow analysis, and tidal volume estimates.
Extensive research has culminated numerous contact and non-contact methods that obtain these metrics with promising levels of accuracy. However, all current non-contact respiratory evaluation is performed using indirect methods, that is, they infer measurements through secondary signals such as visible chest movements, vibration, pressure, acceleration, or sound. Prior methods using spectral analysis for CO2 visualization measure and model the exhale flow consisting of the visualized thermal signature of the CO2 waveform. While prior methods only provide a breathing rate evaluation, this form of visualization has the ability to generate numerous additional metrics such as nose/mouth distribution, velocity, dissipation, behavioral characteristics and even insight into lung efficiency in controlled environments.
In contrast, some embodiments provide for systems and methods of respiratory analysis that are non-contact, but also measure the exhaled air of a human subject directly through a medium-based exhale visualization technique. In some embodiments, a thin medium can be placed perpendicular to the exhaled airflow of an individual, and thermal camera can be used to record the heat signature from the exhaled breath on the opposite side of the material. Breathing rate and respiratory behaviors can be extracted from the thermal data in real-time. Some embodiments of the respiration monitoring technique accurately report breathing rate and provide other information not obtainable through other non-contact methods. Various embodiments can be implemented as a small low-cost device for ease of use in a clinical environment or within an at-home deployment.
In some embodiments, the thin (or projection) medium material can have thermally conductive properties that reflect the temperature changes from the exhale but also allow for rapid dissipation of the heat between breaths. The material may also be as thin as possible to promote this dissipation process within the material as quickly as possible and may have a high emissivity so that changes in temperature can be seen with the thermal camera. In accordance with various embodiments, when selecting the material one or more of the following considerations may be made: (1) thermally conductivity of the material, (2) ability of the material to allow for rapid thermal dissipation, (3) whether the material is thermally opaque, (4) ability of the material to retain heat signatures long enough to capture (e.g. 5 [Hz] or higher), (5) and the ability of the material to not introduce material composition patterns into the resulting images.
Some embodiments use a specialty thermal and CO2 camera to capture images and videos of human exhales and to extract clinically valuable information in a non-invasive way. As a result, various embodiments allow for non-contact analysis of various breathing activities including breathing rate, speed, strength, tidal volume, nose/mouth distribution, and CO2 concentration out of exhale, lung efficiency, and obstructive breathing which can be used for various breathing/pulmonary related diseases diagnosis.
Some embodiments provide for Thin-Medium Thermal Imaging (TMTI) which is an innovative non-contact respiration sensing method that strives to address the problems of existing methods by monitoring respiration directly, but without touching the patient. In accordance with various embodiments, a patient can breathe onto a thin medium while a thermal camera records images of the opposite side of the medium. The thin medium can accurately capture the heat signature of the breath, retaining the temperature gradient long enough to be recorded by a thermal camera, but dissipating the heat quickly between breaths. These images can then be processed using signal processing or machine learning which can convert the thermal signatures on the medium into clinically important metrics such as respiratory rate and volume, and provides additional breathing behavior information to clinicians for a comprehensive view of the respiratory functioning of a patient. Some embodiments use small, low-cost equipment, and works for a variety of patient populations.
Some embodiments can use a variety of techniques to generate respiratory metrics and behavior information. For example, some embodiments may use estimated values and self-reported information from the subject to determine unseen activity behind the medium. Metrics such as the distance between the person's face and the medium, the delay between the start of a breath and when the breath hits the medium, and the person's breathing mode (e.g., whether the person is breathing through their nose, mouth or both) may be unknown and/or subject to fluctuation throughout measurement. Data taken by having a person breathe through an intermediary respiratory measurement device (such as a spirometer) onto the medium can provide accurate timing and exhale information, but the resulting heat signatures may not represent natural breathing. A ground-truth device that does not interfere with natural breathing that also removes uncertainty about activities occurring behind the medium is required to improve some embodiments of the present technology.
Various embodiments of the present technology employ a reinforced hybrid breathing model that uses an additional thermal camera with a spectral filter in the 3-5 [μm] range. This camera acts as a CO2 particle sensor, visualizing turbulent airflows as they exit a person's mouth or nose and collide with the medium. By collecting images from this camera from a side-profile while also collecting thermal images of the medium, human and airflow behaviors can be identified that contribute to the thermal signatures on the medium. These synchronized image sets can be used in some embodiments to train a Convolutional Neural Network (CNN) to identify breathing activities from medium images from an inexpensive device.
In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of embodiments of the present technology. It will be apparent, however, to one skilled in the art that embodiments of the present technology may be practiced without some of these specific details.
The techniques introduced here can be embodied as special-purpose hardware (e.g., circuitry), as programmable circuitry appropriately programmed with software and/or firmware, or as a combination of special-purpose and programmable circuitry. Hence, embodiments may include a machine-readable medium having stored thereon instructions which may be used to program a computer (or other electronic devices) to perform a process. The machine-readable medium may include, but is not limited to, floppy diskettes, optical disks, compact disc read-only memories (CD-ROMs), magneto-optical disks, ROMs, random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or other type of media/machine-readable medium suitable for storing electronic instructions.
The phrases “in some embodiments,” “according to some embodiments,” “in the embodiments shown,” “in other embodiments,” and the like generally mean the particular feature, structure, or characteristic following the phrase is included in at least one implementation of the present technology, and may be included in more than one implementation. In addition, such phrases do not necessarily refer to the same embodiments or different embodiments.
The exhale medium 110 can be composed of any natural or composite material. Typically, it is desirable that the material be highly emissive, very thin and thermally opaque, and have specific thermal properties that retain heat long enough for the camera to capture the image, but allow for dissipation of the heat between breaths. Examples include, but are not limited to, thermochromatic liquid crystal films, a piece of paper, plastics, polymers, metals, fibers, or other synthetic materials.
In some embodiments, thermal camera 130 may be pointed at the opposite side of medium 110 relative to the patient. The exhale from the patient imparts a thermal signature on exhale (or thin) medium 110 that can then be visualized using thermal camera 130. Based on this, some embodiments can track this change over time as a cross-sectional sequence of the exhale flow. A computer system can then use the sequence to form a 3D volume over time. This volume is then calibrated to the tidal volume read from a spirometer or directly measured using the cross-sectional area over time to form a measured enclosed volume.
Recording the sequence of thermal images over time allows a computer system (not shown in
Some embodiments provide an effective method for estimating the tidal volume using only a small thermal camera and the Thin-Medium Thermal Imaging technique. This technique processes the sequence of thermal images, extracts the thermal distribution from them, and produces a cross-sectional area estimation over time to generate a volume. Then, some embodiments correlate this volume with an established ground truth (gold standard) method of measuring tidal volume with a spirometer. This correlation is then used to train a neural network to provide an estimate of the patient's tidal volume directly from the thermal image sequence.
An example of an image sequence is shown in
The thermal signature on the medium can be used to determine whether a patient is breathing nasally, orally, or oronasally. Each of these modes of respiration show a unique thermal signature pattern on the medium that can be identified by their size, shape, location, and flow direction. The thermal signature can be segmented into separate exhale sources by filtering out the regional maxima, threshold the image, and then using pixel clustering. After identifying individual exhale sources from the thermal signature, the contribution from the mouth and each nostril to the surface area of the heat signature can be determined by the pixel sum per thermal region. From this information, various embodiments can calculate the nose-mouth distribution ratio or between each nostril. The strength of an individual exhale can be estimated by the rate of expansion of the thermal signature on the medium. Stronger exhale heat expands farther after hitting the medium surface than that from a less forceful exhale. To estimate exhale strength, some embodiments can use optical flow to estimate surface heat flow across the medium.
This correlation does not need to be patient-specific because the difference between exhale patterns from different individuals can be handled by the complexity of the neural network. Thus, some embodiments can offer a solution that does not require the training process for every individual. Based on the training or trained network provided, the monitoring stage 350 can be defined by having the patient 360 breathe onto the thin medium 370 without the spirometer. This will result in a natural breathing pattern within the thin medium from which some embodiments can then extract metrics.
Some embodiments of the monitoring system can describe the volume extracted different ways. For example, in some embodiments, a cumulative volume over time can be computed that represents how much exhaled air the patient breathes out during normal breathing. The result of this metric can then be used to form a volume per exhale estimate that represents the change from the last valley to the next peak in volume. As another example, some embodiments can measure the flow, or volume over time, that is described in Liters per second. The results of this are shown in the plots 400 within
A breathing strength mode can identify the strength of an individual exhale. In accordance with various embodiments, the system can estimate the breathing strength by the rate of expansion of the thermal signature on the medium. Stronger exhale heat expands farther after hitting the medium surface than that from a less forceful exhale. To estimate exhale strength, some embodiments use optical flow to estimate surface heat flow across the medium.
Some embodiments aim to deliver higher level respiratory metrics than traditional monitoring solutions. For example, some embodiments can measure or estimate nose/mouth distribution and tidal volume, both of which are not obtainable using existing methods. Therefore, the ability to extract this information from the medium-based method is useful.
Some embodiments introduce the notion of controlled respiratory analysis that incorporates two factors: (1) patient distraction and (2) active stimulation. For the first factor, if a patient is aware that they are being monitored, their breathing may not be natural or normal. Therefore, to prevent this observation bias, some embodiments introduce methods of distracting the patient and taking their mind off of the fact that they are being observed. This will result in much more natural recordings, even if they are still wearing the device.
The second factor is an extension of this concept to active stimulation. A multimedia scene with a specific content is chosen and presented to a patient to promote a particular reaction or emotion within the patient (e.g., happy, relaxed, fear, etc.) so that different breathing patterns under an active stimulation can be observed. This provides a differential factor for the types of natural breathing that can be recorded and analyzed based on the observed environment. Some embodiments provide for both a traditional view (e.g., media, movie, animation, image sequence, etc.) and also within a Virtual Reality (VR) environment. This provides an immersive and controlled environment in which some embodiments can invoke natural changes within the patient's breathing which may contribute to identifying the exaggerated effects of some pulmonary conditions.
The distance of the thin medium 610 from the subject can directly affect the performance of the system and the exhale characteristics of the subject which are identified. For example, if the medium is too far away from the subject's face, the exhale may have mostly dissipated before making contact with the medium. This situation could be overcome by a strong exhale, but is exacerbated by light exhales. Conversely, if the medium is close to the subject's face and the subject forcefully exhales, the airflow will hit the surface of the medium and spread in multiple directions. As such, various embodiments use a distance between one and eight inches between the subject's face and the medium. These ranges typically ensure a strong signal, but enough distance that it minimizes the impact of turbulent flows, and is still comfortable for the patient. Other embodiments may position the medium further or closer to the patient.
As the patient begins the monitoring process, they place their chin on the chinrest 620 and breathe normally onto the thin medium 610. The thermal image will then capture the exhale thermal distribution on the thin medium 610. These embodiments provide a practical and simple setup for using the thin medium within a clinical setup for short-term monitoring sessions, at the cost of asking the patient to provide a steady posture during the monitoring process.
In the embodiments illustrated in
One of the primary challenges associated with developing a non-contact method is the movement and orientation of the patient's face. This problem is further exacerbated by the orientation of the medium which may lead to incomplete or lost thermal signatures based on the patient's breathing pattern or facial rotation. Therefore, the method can be transformed into a semi-contact solution that has the patient wear the thin-medium as a mask, separated from the face in some embodiments. The primary difference between this solution and a traditional mask is the separation distance between the patient's face and the thin-medium. This provides natural airflow during inhale and provides a constant distance at which the surface will accurately represent the patient's exhale.
As illustrated in
Respiratory analysis can be significantly affected by changes in a patient's natural breathing behavior due to the monitoring method, their movement, and their focus or concentration. These factors can significantly alter the results of the breathing analysis and should be minimized during the monitoring session. The solution is to disrupt the patient's concentration away from their breathing and the monitoring process and provide a sufficient distraction so that their breathing behaviors will return to normal. This will provide a more accurate method for recording natural behaviors.
To achieve this, some embodiments provide an external stimulus that will draw the patient's attention away from the monitoring process. Two primary methods for doing this that some embodiments introduce are: (1) media-based and (2) Augmented Reality (AR) and/or Virtual Reality (VR) based distractions. The illustration of how the medium and the entertainment source are configured with respect to the thin-medium are shown in
Some embodiments incorporate the concept of the mask and the VR distraction methodology for creating controlled stimulus during the monitoring period. To build this setup, some embodiments combine both the mask design and the VR setup. This will result in the patient wearing the Head Mounted Display (HMD) 910 as they normally would for VR with an additional thin-medium 820 and thermal camera 930 attached. Due to the weight of the HMD itself, the additional attachments will not significantly burden the patient. The illustration of this design 900 is shown in
While a thermal camera provides a viable method for precisely generating a dense representation of the patient's exhale as an image, it requires the camera to be placed on the opposite side of the medium, which limits the mobility of the system. The solution to this problem would be to eliminate the camera from the design, while keeping the thin medium as the primary detection mechanism used for monitoring exhale patterns. Some embodiments use an electrically conductive mesh from which slight changes in voltage across the medium can be interpreted as thermal distributions. Based on the airflow through the permeable mesh, the temperature changes will modify the resistance of the mesh over its surface. These changes and the small fluctuations within the monitored voltages will provide a basis for identifying the thermal distribution of the exhale.
Smart Thin-Medium with Side Camera
Various embodiments of the present technology provide for a data-driven correlation between exhale behaviors on a thin medium with the actual CO2 exhale of the user. The purpose of this model is to improve the accuracy of data extracted from using medium-view images collected from a mobile thermal camera. Some embodiments use the medium-view images and data extracted from the CO2 camera images to train a model that correlates breathing behaviors with thin medium thermal images to improve the accuracy of the TMTI method.
Various embodiments of the present technology may use local and/or remote computing resources to train and/or use a model correlating respiratory activity and recorded data.
Images from the CO2 camera 1030 show a detailed view of breathing behavior, such as when a breath starts and stops, how the person is breathing, and how the exhaled air collides with and spreads across the medium. Information gathered from this camera informs our understanding of the circumstances occurring for each frame from the medium-view camera. Breathing mode is one metric of interest to pulmonologists that can be obtained from the CO2 camera images. Healthy individuals tend to breathe through their nose unless the nasal passage is obstructed, at which point the individual breathes either completely through their mouth, or through their nose and mouth simultaneously. By extracting breathing mode from the CO2 camera images, various embodiments can label each medium-view image and use this data to train a CNN to classify the breathing mode of medium-view images 1060.
To extract information from the side-view CO2 camera images 1070, some embodiments can use various software (e.g., OpenCV library) to find the outline of the person's face through a combination of thresholding and morphological transformations. Various embodiments can process this data (e.g., with a NumPy Python module) to find facial landmarks. The tip of the nose is the left-most pixel coordinate of the face, and the chinrest can be identified by taking the difference of the x-values of the pixel locations and finding the greatest peak.
The chin and mouth are located between the tip of the nose and the chinrest and can be identified in a similar manner to the chinrest. These landmarks can be used in some embodiments to mask unnecessary information and to extract the person's breathing mode. Breathing mode can be determined by processing the pixels along the outline of the face (e.g., using the SciPy Python module), and looking for peaks in the data near the nose and mouth. No prominent peaks in the data indicate that the person is inhaling. This information can be used to label each medium-view thermal image as one of four breathing states: not exhaling, exhaling through the nose, exhaling through the mouth, or exhaling through the nose and mouth.
The medium-view images and the labels extracted from the CO2 camera images can be used to train a machine learning model that predicts breathing mode from the medium-view images. As an image classification problem, an Artificial Neural Network (ANN) model is one well-suited machine learning model for this task, but would likely benefit from some temporal context, as it is difficult to determine if the thermal signature on the medium is increasing or decreasing in temperature by examining a single image. To preserve some temporal context, some embodiments feed both the original image and the previous image subtracted from the current image to the model, which indicates whether the medium temperature is increasing (exhale) or decreasing (inhale).
An implementation based on a CNN model can be built using the Python Keras Python library with a TensorFlow backend. The CNN may consist, in some embodiments, of a 2D convolutional layer with a Tan h activation map, a 2D max pooling layer with a (2×2) pool size, a flattening layer, and then a dense layer. The dense layer uses a softmax activation map to return label probabilities for classification of exhale behaviors.
During data collection, a test subject places their chin on the chinrest and breathes onto the medium (see, e.g.,
Some embodiments may use paper as the medium material because it retains enough heat to be recorded by a thermal camera at a low framerate but dissipates heat between breaths. As one potential material, paper is also inexpensive, easy to find, and standardized. Test subjects that participated in this research were asked to provide six different samples of breathing data, approximately 60[s] each in length. They were asked to first provide the following four breathing samples while at a normal heart rate: (a) nose breathing, (b) normal mouth breathing, (c) breathing through a small mouth opening, and (d) breathing through a large mouth opening. Participants were then asked to provide nose and mouth breathing samples at a slightly elevated heartrate.
Approximately 4000 medium images were collected from 4 individuals. All test subjects were healthy individuals with no congestion due to illness and no other breathing obstructions. Of the collected data, medium-view images that were matched with ambiguous side-view images, and abnormal images resulting from camera calibrations were excluded. This resulted in 2331 medium-view images and their associated labels and measurements.
The collected medium-view images were composed of 1245 images where the individual is inhaling, 422 images of nose breathing, 587 images of mouth breathing, and 77 images where the subject is breathing through both their nose and mouth simultaneously. The collected data was iteratively separated into training data and test data, using a 75% to 25% split. The performance of the CNN was evaluated by running the evaluation 4 times with a different set of training and test data for each, and then calculating the accuracy of each prediction.
Breathing mode prediction from the experimental results can be performed with reasonable accuracy in various embodiments of the present technology. Interestingly, the classification with the highest accuracy rate was the mouth, which is surprising due to the unique thermal signatures mouth breathing tends to produce compared with nose breathing. The CNN would likely be improved with additional training. Additional training data is expected to continue to improve the accuracy of the system and may also be used in a real-time manner in some embodiments. Some embodiments may apply machine learning techniques to other information extracted from the side-view images, such as determining the area of the medium that is directly heated by the initial contact with exhaled air instead of heated by the spread of the exhaled air after colliding with the medium.
As mentioned in and shown in
In some embodiments, breathing rate can be extracted from a sequence of medium images by plotting the average intensity values of the medium images for a window of time and performing a Fast Fourier Transform (FFT) of the data. The FFT transforms the data from the time domain to the frequency domain. The overarching frequency is the breaths per minute (BPM) for that window of time.
In some embodiments, optical flow image processing techniques can be applied to consecutive thin-medium thermal images to highlight the spread of heat across the medium. This results in a dense field of flow vectors across the image. The length of each individual vector indicates the flow of heat between frames, where longer vectors denote a faster flow of exhaled air across the surface of the medium, and shorter vectors denote a slower flow of air across the medium.
The positive difference image values over time from the breathing rate calculation can provide useful insight into the exhale patterns of the individual. Breathing pattern abnormalities, such as several breaths in rapid succession over a short period of time, are lost after condensing the data into a single breathing rate value. However, these abnormalities are made visible as a plot of increasing difference sums over time.
Aspects and implementations of the breathing analysis system of the disclosure have been described in the general context of various steps and operations. A variety of these steps and operations may be performed by hardware components or may be embodied in computer-executable instructions, which may be used to cause a general-purpose or special-purpose processor (e.g., in a computer, server, or other computing device) programmed with the instructions to perform the steps or operations. For example, the steps or operations may be performed by a combination of hardware, software, and/or firmware.
Computers may employ central processing unit (CPU) or processor to process information. Processors may include programmable general-purpose or special-purpose microprocessors, programmable controllers, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), embedded components, combination of such devices and the like. Processors execute program components in response to user and/or system-generated requests. One or more of these components may be implemented in software, hardware or both hardware and software. Processors pass instructions (e.g., operational and data instructions) to enable various operations.
The controller 1900 may include clock 1965, CPU 1970, memory such as read only memory (ROM) 1985 and random access memory (RAM) 1980 and co-processor 1975 among others. These controller components may be connected to a system bus 1960, and through the system bus 1960 to an interface bus 1935. Further, user input devices 1905, peripheral devices 1910, co-processor devices 1915, and the like, may be connected through the interface bus 1935 to the system bus 1960. The interface bus 1935 may be connected to a number of interface adapters such as processor interface 1940, input output interfaces (I/O) 1945, network interfaces 1950, storage interfaces 1955, and the like.
Processor interface 1940 may facilitate communication between co-processor devices 1915 and co-processor 1975. In one implementation, processor interface 1940 may expedite encryption and decryption of requests or data. Input output interfaces (I/O) 1945 facilitate communication between user input devices 1905, peripheral devices 1910, co-processor devices 1915, and/or the like and components of the controller 1900 using protocols such as those for handling audio, data, video interface, wireless transceivers, or the like (e.g., Bluetooth, IEEE 1394a-b, serial, universal serial bus (USB), Digital Visual Interface (DVI), 802.11a/b/g/n/x, cellular, etc.). Network interfaces 1950 may be in communication with the network 1930. Through the network 1930, the controller 1900 may be accessible to remote terminal devices 1920. Network interfaces 1950 may use various wired and wireless connection protocols such as, direct connect, Ethernet, wireless connection such as IEEE 802.11a-x, and the like.
Examples of network 1930 include the Internet, Local Area Network (LAN), Metropolitan Area Network (MAN), a Wide Area Network (WAN), wireless network (e.g., using Wireless Application Protocol WAP), a secured custom connection, and the like. The network interfaces 1950 can include a firewall which can, in some aspects, govern and/or manage permission to access/proxy data in a computer network, and track varying levels of trust between different machines and/or applications. The firewall can be any number of modules having any combination of hardware and/or software components able to enforce a predetermined set of access rights between a particular set of machines and applications, machines and machines, and/or applications and applications, for example, to regulate the flow of traffic and resource sharing between these varying entities. The firewall may additionally manage and/or have access to an access control list which details permissions including, for example, the access and operation rights of an object by an individual, a machine, and/or an application, and the circumstances under which the permission rights stand. Other network security functions performed or included in the functions of the firewall, can be, for example, but are not limited to, intrusion-prevention, intrusion detection, next-generation firewall, personal firewall, etc., without deviating from the novel art of this disclosure.
Storage interfaces 1955 may be in communication with a number of storage devices such as, storage devices 1990, removable disc devices, and the like. The storage interfaces 1955 may use various connection protocols such as Serial Advanced Technology Attachment (SATA), IEEE 1394, Ethernet, Universal Serial Bus (USB), and the like.
User input devices 1905 and peripheral devices 1910 may be connected to I/O interface 1945 and potentially other interfaces, buses and/or components. User input devices 1905 may include card readers, finger print readers, joysticks, keyboards, microphones, mouse, remote controls, retina readers, touch screens, sensors, and/or the like. Peripheral devices 1910 may include antenna, audio devices (e.g., microphone, speakers, etc.), cameras, external processors, communication devices, radio frequency identifiers (RFIDs), scanners, printers, storage devices, transceivers, and/or the like. Co-processor devices 1915 may be connected to the controller 1900 through interface bus 1935, and may include microcontrollers, processors, interfaces or other devices.
Computer executable instructions and data may be stored in memory (e.g., registers, cache memory, random access memory, flash, etc.) which is accessible by processors. These stored instruction codes (e.g., programs) may engage the processor components, motherboard and/or other system components to perform desired operations. The controller 1900 may employ various forms of memory including on-chip CPU memory (e.g., registers), RAM 1980, ROM 1985, and storage devices 1990. Storage devices 1990 may employ any number of tangible, non-transitory storage devices or systems such as fixed or removable magnetic disk drive, an optical drive, solid state memory devices and other processor-readable storage media. Computer-executable instructions stored in the memory may include one or more program modules such as routines, programs, objects, components, data structures, and so on that perform particular tasks or implement particular abstract data types. For example, the memory may contain operating system (OS) component 1995, modules and other components, database tables, and the like. These modules/components may be stored and accessed from the storage devices, including from external storage devices accessible through an interface bus.
The database components can store programs executed by the processor to process the stored data. The database components may be implemented in the form of a database that is relational, scalable and secure. Examples of such database include DB2, MySQL, Oracle, Sybase, and the like. Alternatively, the database may be implemented using various standard data-structures, such as an array, hash, list, stack, structured text file (e.g., XML), table, and/or the like. Such data-structures may be stored in memory and/or in structured files.
The controller 1900 may be implemented in distributed computing environments, where tasks or modules are performed by remote processing devices, which are linked through a communications network, such as a Local Area Network (“LAN”), Wide Area Network (“WAN”), the Internet, and the like. In a distributed computing environment, program modules or subroutines may be located in both local and remote memory storage devices. Distributed computing may be employed to load balance and/or aggregate resources for processing. Alternatively, aspects of the controller 1900 may be distributed electronically over the Internet or over other networks (including wireless networks). Those skilled in the relevant art(s) will recognize that portions of the breathing analysis system may reside on a server computer, while corresponding portions reside on a client computer. Data structures and transmission of data particular to aspects of the controller 1900 are also encompassed within the scope of the disclosure.
Unless the context clearly requires otherwise, throughout the description and the claims, the words “comprise,” “comprising,” and the like are to be construed in an inclusive sense, as opposed to an exclusive or exhaustive sense; that is to say, in the sense of “including, but not limited to.” As used herein, the terms “connected,” “coupled,” or any variant thereof means any connection or coupling, either direct or indirect, between two or more elements; the coupling or connection between the elements can be physical, logical, or a combination thereof. Additionally, the words “herein,” “above,” “below,” and words of similar import, when used in this application, refer to this application as a whole and not to any particular portions of this application. Where the context permits, words in the above Detailed Description using the singular or plural number may also include the plural or singular number respectively. The word “or,” in reference to a list of two or more items, covers all of the following interpretations of the word: any of the items in the list, all of the items in the list, and any combination of the items in the list.
The above Detailed Description of examples of the technology is not intended to be exhaustive or to limit the technology to the precise form disclosed above. While specific examples for the technology are described above for illustrative purposes, various equivalent modifications are possible within the scope of the technology, as those skilled in the relevant art will recognize. For example, while processes or blocks are presented in a given order, alternative implementations may perform routines having steps, or employ systems having blocks, in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and/or modified to provide alternative or sub-combinations. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks are at times shown as being performed in series, these processes or blocks may instead be performed or implemented in parallel, or may be performed at different times. Further any specific numbers noted herein are only examples: alternative implementations may employ differing values or ranges.
The teachings of the technology provided herein can be applied to other systems, not necessarily the system described above. The elements and acts of the various examples described above can be combined to provide further implementations of the technology. Some alternative implementations of the technology may include not only additional elements to those implementations noted above, but also may include fewer elements.
These and other changes can be made to the technology in light of the above Detailed Description. While the above description describes certain examples of the technology, and describes the best mode contemplated, no matter how detailed the above appears in text, the technology can be practiced in many ways. Details of the system may vary considerably in its specific implementation, while still being encompassed by the technology disclosed herein. As noted above, particular terminology used when describing certain features or aspects of the technology should not be taken to imply that the terminology is being redefined herein to be restricted to any specific characteristics, features, or aspects of the technology with which that terminology is associated. In general, the terms used in the following claims should not be construed to limit the technology to the specific examples disclosed in the specification, unless the above Detailed Description section explicitly defines such terms. Accordingly, the actual scope of the technology encompasses not only the disclosed examples, but also all equivalent ways of practicing or implementing the technology under the claims.
To reduce the number of claims, certain aspects of the technology are presented below in certain claim forms, but the applicant contemplates the various aspects of the technology in any number of claim forms. For example, while only one aspect of the technology is recited as a computer-readable medium claim, other aspects may likewise be embodied as a computer-readable medium claim, or in other forms, such as being embodied in a means-plus-function claim. Any claims intended to be treated under 35 U.S.C. § 112(f) will begin with the words “means for”, but use of the term “for” in any other context is not intended to invoke treatment under 35 U.S.C. § 112(f). Accordingly, the applicant reserves the right to pursue additional claims after filing this application to pursue such additional claim forms, in either this application or in a continuing application.
This application claims priority to U.S. Provisional Application Ser. No. 62/756,501 filed Nov. 6, 2018, which is incorporated herein by reference in its entirety for all purposes.
| Number | Date | Country | |
|---|---|---|---|
| 62756501 | Nov 2018 | US |
