Patent Application
20220211310
The present invention generally relates to an ocular system for monitoring mental health. More particularly, the present invention relates to an ocular system that can visually scan a user's (patient's) eye movements (i.e., gaze) combined with ocular activity in the eye (i.e., pupil dilation, iris dilator and sphincter muscle dilation and constriction) to diagnose a mental health condition that can be displayed to the user or a mental health professional.
The prevalence of posttraumatic stress disorder (PTSD) has been estimated to be as high as 23% in veterans returning from Iraq and Afghanistan. The lifetime incidence of PTSD for all US adults is estimated at 6.8%. However, PTSD diagnosis requires a structured clinical interview with a mental health clinician and incorporating screening tools such as the Clinician-Administered PTSD Scale for DSM-5 (CAPS-5), which is time-consuming and labor-intensive, and heavily relies on subjective self-reporting from the patient. Given the prevalence of PTSD and the need for a quick, effective, objective and accurate diagnostic tool (particularly in the high-risk population such as military personnel) Senseye has developed a Machine Learning powered software as a medical device (SAMD) to quantitatively assess the presence and severity of PTSD symptoms measured through computer vision and analytic techniques.
PTSD is associated with adverse aggressive behaviors, emotional constrictions, and social withdrawal, with evidence of impaired fear extinction and neuroplasticity, and is linked with impaired eye reactivity, autonomous nervous system (ANS) reactivity, and increased activity, neurovascular inflammation, sleep disturbances, suicidality, and major cardiovascular events. (9-18) In fact, prior research has demonstrated that PTSD patients could be accurately discriminated from control participants based on their pupil reactivity to visual and auditory threat stimuli. (19) This atypical reactivity may also be manifested in a simple reflexive response, as sympathetic overdrive would result in reduced constriction velocity and amplitude to light because the dilator is overactive. (19-22)
Prior studies have shown that impaired oculomotor reactions measured by eye-tracking and impaired ANS reactivity measured by pupil light reflex in response to threat-relevant stimuli can directly assess PTSD's severity of symptoms. (19-22)
Clinician-Administered PTSD Scale for DSM-5 (CAPS-5), UCLA PTSD reaction index (RI), as gold standards tools in the diagnosis of PTSD, have been extensively validated against standardized structural clinical interview across sex, age groups, and cultures with high feasibility, acceptability to assess core PTSD symptoms, and facilitating risk stratification and outcome prediction in individuals at risk for PTSD. (23, 24)
Deep machine learning and artificial intelligence (AI) can detect eye reactivity, sensory perception, and engagement. (9-15) AI evaluates an individual's response to digitally created scenarios threat and neutral stimuli into the real-world environment; it provides the unique opportunity for real-time detection of PTSD. (9-15) The lack of a scalable real-time operator-independent tool to assess the presence and severity of PTSD and monitoring response to intervention significantly limit the early identification and management of individuals at risk for PTSD. (17, 25-28) Senseye's Operator-independent Ocular Brain-Computer Interface (OBCI) can eliminate these limitations and add a safe, viable adjunct to standardized structured clinical interviews to assess the real-time presence and severity of PTSD and monitor response to interventions. (29) With the emergence of deep machine learning technology that is now possible to detect and monitor PTSD in real-time with Senseye's CV and Machine Learning Algorithms, we propose to utilize a machine learning-powered software as a diagnostic device to quantitatively assess the presence and severity of PTSD symptoms measured through computer vision and proprietary analytic techniques developed by Senseye.
A method of measuring non-invasive ocular metrics to diagnose a mental health state of a patient comprises the steps of: providing a video camera, an electronic display screen, a hardware system and a software configured to run on the hardware system, wherein the video camera and the electronic display screen are connected to the hardware system and controlled by the software; providing access to the patient to the electronic display screen to interact with the software, wherein the video camera is located near or as part of the electronic display screen configured to non-invasively record at least one eye of the patient when viewing the electronic display screen; presenting a stimuli on the electronic display screen by the software; during presenting the stimuli, recording a video of the at least one eye of the patient by the video camera; wherein the stimuli comprises an oculomotor task or oculomotor stimuli configured to elicit a change in at least one ocular signal of the at least one eye of the patient, the stimuli comprising a stimuli image, a series of stimuli images or a stimuli video for passive watching by the patient configured to elicit the change in the at least one ocular signal; wherein the at least one ocular signal is selected from the following group of a(n): eye movement, gaze location X, gaze location Y; saccade rate, saccade peak velocity, saccade average velocity, saccade amplitude, fixation duration, fixation entropy (spatial), gaze deviation (polar angle), gaze deviation (eccentricity), re-fixation, smooth pursuit, smooth pursuit duration, smooth pursuit average velocity, smooth pursuit amplitude, scan path (gaze trajectory over time), pupil diameter, pupil area, pupil symmetry, velocity (change in pupil diameter), acceleration (change in velocity), jerk (pupil change acceleration), pupillary fluctuation trace, pupil area constriction latency, pupil area constriction velocity, pupil area dilation duration, spectral features, iris muscle features, iris muscle group identification, iris muscle fiber contractions, iris sphincter identification, iris dilator identification, iris sphincter symmetry, pupil and iris centration vectors, blink rate, blink duration, blink latency, blink velocity, partial blink rate, partial blink duration, blink entropy (deviation from periodicity), sclera segmentation, iris segmentation, pupil segmentation, stroma change detection, percent eyes closed, eyeball area (squinting), iridea changes; wherein the hardware system comprises a processor configured to run a machine learning classification model and a computer vision model; processing, by the computer vision model, image frames of the video of the at least one ocular signal through a series of optimized algorithms configured to isolate and quantify the at least one ocular signal by applying an image mask isolating components of the at least one eye of the patient; estimating, by an algorithm run by the machine learning classification model, a probability from the at least one ocular signal that it represents the mental health state; and displaying, after the processing, the mental health state estimated by the software of the patient to the patient, or, sending the mental health state to a mental health professional via an electronic communication.
The mental health state may comprise a mental health disorder, a substance abuse disorder, a post-traumatic stress disorder, an anxiety disorder, a depressive disorder, an acute stress disorder or an acute stress reaction.
The at least one ocular signal may comprise at least two ocular signals or at least three ocular signals.
The method may be repeated after an initial diagnosis to measure a severity of the mental health disorder over a period of time.
The method may be repeated after an initial diagnosis to measure a severity of the mental health disorder over a period of time while the patient is receiving treatment in order to measure a treatment efficacy.
The method may include storing the mental health state of the patient in a retrievable data retention system.
The video camera, the electronic display screen, the hardware system and the software may be configured to run on the hardware system which are all part of an electronic mobile device, a tablet, a desktop computer or a laptop computer.
The video camera and electronic display screen may be remotely disposed in relation to the hardware system and software configured to run the hardware system. For example, the hardware system and software may comprise a cloud-based system.
The video camera may be a webcam, a cell phone camera, or any other video camera with sufficient resolution and frame rate. The sufficient frame rate may be 30 frames per seconds and the sufficient resolution may be 100 pixels per inch.
The method may include the step of measuring heart rate, wherein the estimating, by the algorithm run by the machine learning classification model, of the probability includes information from both the at least one ocular signal and the heart rate.
The method may include the step of measuring respiration, wherein the estimating, by the algorithm run by the machine learning classification model, of the probability includes information from both the at least one ocular signal and the respiration.
Other features and advantages of the present invention will become apparent from the following more detailed description, when taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the invention.
The accompanying drawings illustrate the invention. In such drawings:
Ocular System for Monitoring Mental Health
Overview: Senseye Mental Health Monitoring (SMHM) operates at the intersection of mental health therapies and technology. It provides a new, objective, method of quantifying mental health states and the impacts of therapeutic techniques. The system uses non-invasive ocular measures, to measure the Sympathetic and Parasympathetic Nervous systems to identify and track occurrences of mental health disorders manifesting themselves in disruptions of the sympathetic nervous system (such as Anxiety, Depression and PTSD). SMHM algorithms monitor and classify these mental states on an individual basis. SMHM algorithms are not only able to identify mental health disorders, but are also able to track mental health status over time. SHMH can aid in adapting therapeutic interventions, from talk therapy to microdosing, to an individual's unique mental state. This level of adaptive therapy and monitoring provides accelerated treatment while ensuring the compliance and utility of the intervention.
Product Function: The Senseye system is designed to run on a variety of hardware options. The eye video can be acquired by a webcam, cell phone camera, or any other video camera with sufficient resolution and frame rate. For example, a sufficient frame rate is 30 or 60 fps but could be lower over time with the improvement in technology. Also, the sufficient resolution is 240 by 240-pixel box over the eyes, but could be as low as a 100 pixels per inch. The stimuli can be presented on a cell phone, tablet, or laptop screen or a standard computer monitor. The necessary hardware to run the software is neural-network-capable fpgas (Field Programmable Gate Array), asics (application-specific integrated circuit) or accelerated hardware, either within the device or on a server accessed through an API.
The Senseye assessment begins with the user initiating the process by logging in to the system. This can be achieved by typing a username and password issued to them by their HCP (HealthCare Provider). In one embodiment, the user is presented with a series of oculomotor tasks and or stimuli. In another embodiment, the scan is designed to be more passive, so the user's eyes are recorded while they passively view a screen.
Signals: Senseye Mental Health Monitoring detection relies on ocular signals to make its classifications. These include:
The signals are acquired using a multistep process designed to extract nuanced information from the eye. Image frames from video data are processed through a series of optimized algorithms designed to isolate and quantify structures of interest. These isolated data are further processed using a mixture of automatically optimized, hand parameterized, and non-parametric transformations and algorithms.
Disorder Detection: The SMHM software is capable of working on any device with a front facing camera (tablet, phone, computer, etc.). The SMHM software draws on previous scientific findings (D'Hondt et al., 2014; Ferneyhough et al., 2013; Kattoulas et al., 2011; Laretzaki et al., 2011; Nagai et al., 2002; Quigley et al., 2012; Strollstorf et al., 2013; Young et al., 2012) and uses anatomical and physiological signals extracted from images to predict different mental states through optimized algorithms. The algorithms provide an estimated probability that the input data represents a particular disordered mental state and may identify the presence of one or more states. Image signals are run through a series of data processing operations to extract signals and estimations. Multiple image masks are first applied, isolating components of the eyes as well as facial features allowing various metrics to be extracted from the image in real-time. From the image filters, pertinent signals are extracted through transformation algorithms supporting the final estimation of mental states. Multiple data streams and estimations can be made in a single calculation, and mental state signals may stem from combinations of multiple unique processing and estimation algorithms. The mental state output is directly linked to the stimulus (video and/or images and/or blank screen shown) by relating processing signals during the stimulus. The software can display, immediately after the screening, the mental state of the individual.
The SHMH software can operate on a longitudinal basis as well. As users continue to check in with the software, their states over time are monitored for information as to how frequently a user experiences disordered mental states. The system stores this information unique to each user. This provides additional information to users and treatment specialists.
Therapeutic effectiveness and intervention: The capability to track a user longitudinally and remotely allows for analysis of the effectiveness of therapeutic interventions. As a user undergoes therapy the system continues to output information about mental states stored longitudinally for each user. This allows the user and other stakeholders to objectively monitor improvements in condition via changes in ocular signals. Therapeutic interventions are not limited and may include traditional therapeutic methods as well as analysis of patient response to smart dosing. Ocular metrics can be taken at different levels of dosing and help treatment specialists converge quicky on effective treatment levels.
Detecting, Diagnosing and Monitoring Substance Use Disorders in an Objective and Noninvasive Manner
Overview: Senseye Substance Use Disorder Diagnosis (SSUDD) uses non-invasive ocular measures of brain state and physiology to identify and track substance use disorders. It is able to differentiate between different substances, specifically between substances of abuse and those used for therapeutic intervention, and thus serve as a therapeutic monitoring tool. By monitoring ocular metrics throughout different levels of drug based therapeutic intervention, SSUDD aids in adapting the interventions to an individual's unique case. This level of monitoring provides accelerated treatment while ensuring the compliance and utility of the intervention.
Product Function: The Senseye system is designed to run on a variety of hardware options. The eye video can be acquired by a webcam, cell phone camera, or any other video camera with sufficient resolution and frame rate. The stimuli can be presented on a cell phone, tablet, or laptop screen or a standard computer monitor. The necessary hardware to run the software is neural-network-capable fpgas, asics or accelerated hardware; either within the device or on a server accessed through an API.
The Senseye assessment begins with the user initiating the process by logging in to the system. This can be achieved by typing a username and password, or using facial recognition. In one embodiment, the user is presented with a series of oculomotor tasks and or stimuli. In another embodiment, the scan is designed to be more passive, so the user's eyes are recorded while they passively view a screen.
Signals: Senseye Substance Use Disorder Detection relies on ocular signals to make its classifications. These include:
The signals are acquired using a multistep process designed to extract nuanced information from the eye. Image frames from video data are processed through a series of optimized computer vision algorithms designed to isolate and quantify structures of interest. These isolated data are further processed using a mixture of automatically optimized, hand parameterized, and non-parametric transformations and algorithms.
Substance use detection: The SSUDD software is capable of working on any device with a front facing camera (tablet, phone, computer, etc.). The SSUDD software uses anatomical signals extracted from images to predict the levels of different substances present in a user through optimized algorithms. The algorithms provide an estimated probability that the input data represents presences of a particular substance and may identify the presence of one or more substances. Image signals are run through a series of data processing operations to extract signals and estimations. Multiple image masks are first applied, isolating components of the eyes as well as facial features allowing various metrics to be extracted from the image in real-time. From the image filters, pertinent signals are extracted through transformation algorithms supporting the final estimation of substance levels. Multiple data streams and estimations can be made in a single calculation, and substance presence signals may stem from combinations of multiple unique processing and estimation algorithms. Previous scientific research has shown links between ocular physiology and substances present in a person's system (Dhingra, Kaur, & Ram 2019; Fazari, 2011; Kaut, Oliver, Kornblum, & Cornelia, 2010; Merlin, 2008; Murillo, Crucilla, Schmittner, Hotchkiss, & Pickworth, 2004; Rottach, Wohlgemuth, Dzaja, Eggert, & Straube, 2002). In SSUDD, the substance level output is directly linked to the stimulus (video and/or images and/or blank screen shown) through analysis of ocular signals. The software can display, immediately after the screening, the presence or absence of opioids, alcohol or other substances of abuse.
Therapeutic effectiveness and intervention: Because the SSUDD is able to differentiate between substances, specifically substances of abuse and therapeutic substances, this allows for the application to be used to track compliance with therapeutic interventions. Not only are the readings informative, but the rate at which the user deviates from a set check-in schedule can provide information about their compliance with a therapeutic program. As a user undergoes therapy the system continues to output information about substance use, including use of therapeutic substances, and this is stored longitudinally for each user. This allows the user and other stakeholders, such as doctors and other therapists, to objectively monitor improvements in condition via changes in ocular signals.
An Objective Diagnostic and for Post-Traumatic Stress Disorder
Overview: The Senseye PTSD Diagnostic provides a new, objective, method of quantifying mental health states and the impacts of therapeutic techniques. It is the first of its kind tool allowing for the objective diagnosis of and continuous monitoring of PTSD. The tool can both diagnose PTSD as well as continuously monitor the patient via recurring scans it in order to monitor treatment response, changing severity and too predict treatment responses.
The system records video of the user's eyes while they perform various oculomotor tasks and/or passively view a screen. The ORM system also includes the software that presents the stimuli to the user. The system uses computer vision to segment the eyes and quantify a variety of ocular features. The ocular metrics then become inputs to a machine learning algorithm designed to diagnose the condition and report on its severity. The product's algorithms are not only able to identify anxiety-related mental health disorders, but are also able to track mental health status over time. SHMH can aid in adapting therapeutic interventions, from talk therapy to microdosing, to an individual's unique mental state. This level of adaptive therapy and monitoring provides accelerated treatment while ensuring the compliance and utility of the intervention.
Inputs and outputs: The primary input the Senseye system is video footage of the eyes of the user while they perform the oculomotor tasks presented by the system. The location and identity of visible anatomical features from the open eye (i.e., sclera, iris, and pupil) are classified in digital images in a pixel-wise manner via convolutional neural networks originally developed for medical image segmentation. Based on the output of the convolutional neural network, numerous ocular features are produced. These ocular metrics are combined with event data from the oculomotor tasks which provide context and labels. The ocular metrics and event data are provided to the machine learning algorithms which then return a result of a diagnosis or lack of, or “more information needed.” This is achieved by quantifying the pupil and iris dynamics throughout the oculomotor tasks.
Signals: Senseye Mental Health Monitoring detection relies on ocular signals to make its classifications. These include:
The signals are acquired using a multistep process designed to extract nuanced information from the eye. Image frames from video data are processed through a series of optimized algorithms designed to isolate and quantify structures of interest. These isolated data are further processed using a mixture of automatically optimized, hand parameterized, and non-parametric transformations and algorithms.
Product Function: The Senseye PTSD system is designed to run on a variety of hardware options. The software is capable of working on any device with a front facing camera (tablet, phone, computer, etc.). The SMHM software draws on previous scientific findings (D'Hondt et al., 2014; Ferneyhough et al., 2013; Kattoulas et al., 2011; Laretzaki et al., 2011; Nagai et al., 2002; Quigley et al., 2012; Strollstorf et al., 2013; Young et al., 2012) and uses anatomical and physiological signals extracted from images to predict different mental states through optimized algorithms. The algorithms provide an estimated probability that the input data represents a particular disordered mental state and may identify the presence of one or more states. Image signals are run through a series of data processing operations to extract signals and estimations. Multiple image masks are first applied, isolating components of the eyes as well as facial features allowing various metrics to be extracted from the image in real-time. From the image filters, pertinent signals are extracted through transformation algorithms supporting the final estimation of mental states. Multiple data streams and estimations can be made in a single calculation, and mental state signals may stem from combinations of multiple unique processing and estimation algorithms. The mental state output is directly linked to the stimulus (video and/or images and/or blank screen shown) by relating processing signals during the stimulus. The software can display, immediately after the screening, the mental state of the individual.
The software can operate on a longitudinal basis as well. As users continue to check in with the software, their states over time are monitored for information as to how frequently a user experiences disordered mental states. The system stores this information unique to each user. This provides additional information to users and treatment specialists.
Therapeutic effectiveness and intervention: The capability to track a user longitudinally and remotely allows for analysis of the effectiveness of therapeutic interventions. As a user undergoes therapy the system continues to output information about mental states stored longitudinally for each user. This allows the user and other stakeholders to objectively monitor improvements in condition via changes in ocular signals. Therapeutic interventions are not limited and may include traditional therapeutic methods as well as analysis of patient response to smart dosing. Ocular metrics can be taken at different levels of dosing and help treatment specialists converge quicky on effective treatment levels.
Description: The Senseye Device is an AI/MI based Software as a Medical Device. A patient views a series of stimuli in the form of ocular tasks on a mobile phone while we track their ocular movements in response to such stimuli. The methods described here are intended to provide the high-level composition of ocular screening tasks that form the basis of each experimental session. Final task composition and duration will likely be modified.
Ocular tasks known to elicit pupillary and eye movement dynamics of interest will be used. See
Other tasks include a task requiring participants to make saccadic eye movements toward randomly appearing targets on the screen, a task requiring free viewing of neutral and aversive images, and tasks measuring alertness or reaction time.
All of these tasks are short in duration (less than 1 minute), but may be repeated multiple times within an experimental session, thereby requiring an onsite time commitment from participants of 5-30 minutes. The tasks can easily be deployed on mobile devices that the participants can take home for regular check-ins (5-10 minutes) throughout the day at specified intervals if required. Senseye intends to initially deploy the product with 10-15 ocular tasks in clinical trials to identify which 3-5 are the most accurate in PTSD diagnostics.
Hardware: Onsite high-resolution data is collected using mobile phones with either their built-in cameras or an external camera plugged into the phone, or with cameras plugged into laptop computers. To utilize the features we have developed and optimize the performance of the device, we have picked a minimum list of requirements for use with the Senseye application as shown in
It has been shown that pupil diameter changes in response to images differently if a patient has PTSD. However, to the inventor's knowledge, and based on the FDA's De Novo Classification for the device nobody has ever been able to build a product that works based on changes in pupil diameter. The present invention works because it is measuring all those things beyond just pupil size. Therefore, a system must be able to measure at least 2, 3, 4, 5, 10, 15, 20 or any “n” number ocular metrics beyond pupil size. While it is positive to use just one ocular metric to determine a mental health state, this may lead to a false positive such that the rate for a false determination would be too high. Thus, the inventors prefer to use a combination of ocular metrics to provide a more reliable determination of mental health state.
The inventors have developed computer vision algorithms capable of using normal cameras for the present invention. Accordingly, the entire contents of the following list of patent applications by the inventors are fully incorporated herein with this reference: application Ser. No. 17/247,634 filed Dec. 18, 2020; application Ser. No. 17/247,635 filed Dec. 18, 2020; application Ser. No. 17/247,636 filed Dec. 18, 2020; application Ser. No. 17/247,637 filed Dec. 18, 2020, and PCT application PCT/US20/70939 filed on Dec. 19, 2020. More specifically, these prior applications taught a method for generating NIR images from RGB cameras using generative adversarial networks and a combination of visible and IR light. Thus, the relevant text from those applications is repeated herein for convenience.
Continuing the theme of creating a mapping between subsurface iris structures visible in IR light onto surface structures seen in visible light, Senseye has developed a method of projecting iris masks formed on IR images onto the data extracted from visible light. This technique uses a generative adversarial network (GAN) to predict the IR image of an input image captured under visible light (see
Part of this method is generating a training set of images on which the GAN learns to predict IR images from visible light images (see
Creating optically and temporally aligned visible and NIR datasets with low error allows Senseye to create enormous and varied datasets that do not need to be labelled. Instead of manual labelling, the alignment allows Senseye to use the NIR images as reference to train the color images against. Pre-existing networks already have the ability to classify and segment the eye into sclera, iris, pupil, and more, giving us the ability to use their outputs as training labels. Additionally, unsupervised techniques like pix-to-pix GANs utilize this framework to model similarities and differences between the image types. These data are used to create surface-to-surface, and/or surface-to-subsurface mapping of visible and invisible iris features.
Other methods being considered to properly filter the RGB spectrum so it resembles the NIR images, is the use of a simulation of the eye so that rendered images resembles both natural light and that in NIR light spectrum. The neural network structures would be similar to those listed previously (pix-to-pix) and the objective would be to allow for the sub cornea structures (iris and pupil) to be recovered and segmented properly despite the reflections or other artifacts caused by the interaction of the natural light spectrum (360 to 730 nm) with the particular eye.
The utility of the GAN is to learn a function that is able to generate NIR images from RGB images. The issues with RGB images derive from the degradation of contrast between pupil and iris specifically for darker eyes. What this means is that if there isn't enough light flooding the eye, the border of a brown iris and the pupil hole are indistinguishable due to their proximity in the color spectrum. In RGB space, because we do not control for a particular spectrum of light, we are at the mercy of another property of the eye which is that it acts as a mirror. This property allows for any object to appear as a transparent film on top of the pupil/iris. An example of this is you can make out a smaller version of a bright monitor on your eye given an rgb image. So the GAN acts as a filter. It filters out the reflections, sharpens boundaries, and due to its learned embedding, it is capable of restoring the true boundary of iris and pupil.
In furtherance of improving the present invention, the inventors have been able to make the present invention work with just a normal camera without use of the GAN. However, sometimes use of the GAN is still needed, but not always. Again, this is an area of constant improvement by the inventors of the instant application.
Although several embodiments have been described in detail for purposes of illustration, various modifications may be made to each without departing from the scope and spirit of the invention. Accordingly, the invention is not to be limited, except as by the appended claims.
This continuation-in-part application claims priority to provisional application 63/200,696 filed Mar. 23, 2021, the entire contents of which are hereby incorporated in full by this reference. This continuation-in-part application also claims priority to application Ser. No. 17/247,634 filed Dec. 18, 2020; application Ser. No. 17/247,635 filed Dec. 18, 2020; application Ser. No. 17/247,636 filed Dec. 18, 2020; and application Ser. No. 17/247,637 filed Dec. 18, 2020, the entire contents of which are hereby incorporated in full by this reference.
| Number | Date | Country | |
|---|---|---|---|
| 63200696 | Mar 2021 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17247634 | Dec 2020 | US |
| Child | 17655977 | US | |
| Parent | 17247635 | Dec 2020 | US |
| Child | 17247634 | US | |
| Parent | 17247636 | Dec 2020 | US |
| Child | 17247635 | US | |
| Parent | 17247637 | Dec 2020 | US |
| Child | 17247636 | US |
