Patent Application
20210386345
The present disclosure relates, in various embodiments, to devices and processing systems configured to enable extended monitoring and analysis of subject neurological factors via blepharometric data collection, for example, in the context of analysis of neurological conditions using blepharometric data (data that records eyelid movement parameters as a function of time). For example, some embodiments provide methods and associated technology that enable detection of changes in neurological conditions in a human subject (for example, to assist in management/identification of conditions that may be associated with seizures, degenerative diseases, and the like). While some embodiments will be described herein with particular reference to that application, it will be appreciated that the disclosure is not limited to such a field of use, and is applicable in broader contexts.
Any discussion of the background art throughout the specification should in no way be considered as an admission that such art is widely known or forms part of common general knowledge in the field.
It is known to analyze neurological conditions from analysis of eye and/or eyelid movements. For example, U.S. Pat. No. 7,791,491 teaches a method and apparatus for measuring drowsiness based on the amplitude to velocity ratio for eyelids closing and opening during blinking as well as measuring duration of opening and closing. This enables an objective measurement of drowsiness.
The present inventors, through their research into relationships between eye and eyelid movement parameters and neurological conditions, have identified opportunities for probabilistic prediction and/or detection of additional neurological conditions via analysis of eyelid movement parameters.
It is an object of the present disclosure to overcome or ameliorate at least one of the disadvantages of the prior art, or to provide a useful alternative.
One embodiment provides a system configured to facilitate collection of blepharometric data from one or more subjects on a periodic basis thereby to enable extended time period analysis of subject neurological conditions, the system including:
One embodiment provides a system wherein the sensor device is an image capture device.
One embodiment provides a system wherein the system includes an image processing system that is configured to: (i) detect presence of a human face; (ii) identify one or more eye regions in the human face; and (iii) based on identification of the one or more eye regions, generate blepharometric data representative of eyelid position against time.
One embodiment provides a system wherein the subject identification module, which is configured to identify a unique human subject from which a set of blepharometric data is collected by the sensor device, leverages a facial recognition process thereby to extract biometric facial information from one or more frames of image data collected via the image capture device.
One embodiment provides a system wherein the subject identification module is configured to identify a unique human subject from which a set of blepharometric data is collected by the sensor device via collection of biometric data.
One embodiment provides a system wherein the biometric data includes facial data.
One embodiment provides a system wherein the subject identification module is configured to identify a unique human subject from which a set of blepharometric data is collected by the sensor device via user input of identifying credentials.
One embodiment provides a system wherein the subject identification module is configured to identify a unique human subject from which a set of blepharometric data is collected by the sensor device via communication with a user mobile device, which includes a token representative of identifying credentials.
One embodiment provides a system wherein defining current blepharometric data for the human subject includes processing blepharometric data for a period or sub-period of continuous blepharometric data collection via the sensor device, thereby to extract a set of blepharometric data artefacts.
One embodiment provides a system wherein the blepharometric data artefacts include any one or more of the following:
One embodiment provides a system wherein the memory module that is configured to maintain a record of historical blepharometric data for the identified human subject includes statistical information derived from processing of blepharometric data collected across a plurality of previous periods.
One embodiment provides a system wherein the blepharometric data collected across a plurality of previous periods is collected via a plurality of physically distinct collection systems.
One embodiment provides a system wherein the blepharometric data variation processing module is configured to identify a relationship between the current blepharometric data for the human subject and the record of historical blepharometric data for the identified human subject by processing methods including one or more of the following:
One embodiment provides a system wherein identifying a relationship between the current blepharometric data for the human subject and the record of historical blepharometric data for the identified human subject thereby to identify a long-term trend in blepharometric data includes determining whether, in response to a current set of blepharometric data, there is an identified threshold trend in one or more of the user's observed blepharometric artefacts that satisfies a predefined profile that is representative of prediction of a neurological condition.
One embodiment provides a system wherein identifying a relationship between the current blepharometric data for the human subject and the record of historical blepharometric data for the identified human subject thereby to identify a threshold current point-in-time deviation from historical statistical data includes determining whether, the current set of blepharometric data alone or in combination with one or more recent sets of blepharometric data, display a threshold deviation in one or more of the user's observed blepharometric artefacts compared to historical averages, wherein that deviation is representative of prediction of a neurological condition.
One embodiment provides a system wherein the output module is configured to cause delivery of an output signal via in in-vehicle display.
One embodiment provides a system wherein the output module is configured to cause delivery of an output signal via an electronic message sent over a network.
One embodiment provides a system wherein the vehicle is an automobile, and wherein the sensor device is mounted on or adjacent a dashboard or windscreen region.
One embodiment provides a system including multiple sensor devices, each mounted in the vehicle positioned to enable monitoring eyelid movement by a respective passenger or operator of the vehicle.
One embodiment provides a system including the blepharometric data monitoring system.
One embodiment provides a device configured to facilitate collection of blepharometric data from one or more subjects on a periodic basis thereby to enable extended time period analysis of subject neurological conditions, the device including:
One embodiment provides a device wherein the sensor device is an image capture device.
One embodiment provides a device wherein the device includes an image processing system that is configured to: (i) detect presence of a human face; (ii) identify one or more eye regions in the human face; and (iii) based on identification of the one or more eye regions, generate blepharometric data representative of eyelid position against time.
One embodiment provides a device wherein the subject identification module, which is configured to identify a unique human subject from which a set of blepharometric data is collected by the sensor device, leverages a facial recognition process thereby to extract biometric facial information from one or more frames of image data collected via the image capture device.
One embodiment provides a device wherein the subject identification module is configured to identify a unique human subject from which a set of blepharometric data is collected by the sensor device via collection of biometric data.
One embodiment provides a device wherein the biometric data includes facial data.
One embodiment provides a device wherein the subject identification module is configured to identify a unique human subject from which a set of blepharometric data is collected by the sensor device via user input of identifying credentials.
One embodiment provides a device wherein the subject identification module is configured to identify a unique human subject from which a set of blepharometric data is collected by the sensor device via communication with a user mobile device, which includes a token representative of identifying credentials.
One embodiment provides a device wherein defining current blepharometric data for the human subject includes processing blepharometric data for a period or sub-period of continuous blepharometric data collection via the sensor device, thereby to extract a set of blepharometric data artefacts.
One embodiment provides a device the blepharometric data artefacts include any one or more of the following:
One embodiment provides a device wherein the memory module that is configured to maintain a record of historical blepharometric data for the identified human subject includes statistical information derived from processing of blepharometric data collected across a plurality of previous periods.
One embodiment provides a device wherein the blepharometric data collected across a plurality of previous periods is collected via a plurality of physically distinct collection systems.
One embodiment provides a device wherein
One embodiment provides a device wherein identifying a threshold trend includes identifying a threshold trend in one or more of the user's observed blepharometric artefacts that satisfies a predefined profile that is representative of prediction of a neurological condition.
One embodiment provides a device wherein identifying point-in-time statistical deviation includes determining whether, the current set of blepharometric data alone or in combination with one or more recent sets of blepharometric data, display a threshold deviation in one or more of the user's observed blepharometric artefacts compared to historical averages, wherein that deviation is representative of prediction of a neurological condition.
One embodiment provides a device wherein the output module is configured to cause delivery of an output signal via in in-vehicle display.
One embodiment provides a device wherein the output module is configured to cause delivery of an output signal via an electronic message sent over a network.
One embodiment provides a device wherein the vehicle is an automobile, and wherein the sensor device is mounted on or adjacent a dashboard or windscreen region.
One embodiment provides a device including multiple sensor devices each mounted in the vehicle positioned to enable monitoring eyelid movement by a respective passenger or operator of the vehicle.
One embodiment provides a device including the blepharometric data monitoring system.
One embodiment provides a system configured to facilitate analysis of subject neurological conditions, the system including:
One embodiment provides a system wherein, for at least one of the sensor systems, the sensor system includes the sensor device including an image capture device that is configured to monitor blepharometric data.
One embodiment provides a system wherein the system includes an image processing system that is configured to: (i) detect presence of a human face; (ii) identify one or more eye regions in the human face; and (iii) based on identification of the one or more eye regions, generate blepharometric data representative of eyelid position against time.
One embodiment provides a system wherein the subject identification module, which is configured to identify a unique human subject from which a set of blepharometric data is collected by the sensor device, leverages a facial recognition process thereby to extract biometric facial information from one or more frames of image data collected via the image capture device.
One embodiment provides a system wherein the subject identification module is configured to identify a unique human subject from which a set of blepharometric data is collected by the sensor device via collection of biometric data.
One embodiment provides a system wherein the biometric data includes facial data.
One embodiment provides a system wherein the subject identification module is configured to identify a unique human subject from which a set of blepharometric data is collected by the sensor device via user input of identifying credentials.
One embodiment provides a system wherein the subject identification module is configured to identify a unique human subject from which a set of blepharometric data is collected by the sensor device via communication with a user mobile device, which includes a token representative of identifying credentials.
One embodiment provides a system wherein defining current blepharometric data for the human subject includes processing blepharometric data for a period or sub-period of continuous blepharometric data collection via the sensor device, thereby to extract a set of blepharometric data artefacts.
One embodiment provides a system wherein the blepharometric data artefacts include any one or more of the following:
One embodiment provides a system wherein the memory module that is configured to maintain a record of historical blepharometric data for the identified human subject includes statistical information derived from processing of blepharometric data collected across a plurality of previous periods.
One embodiment provides a system wherein the blepharometric data collected across a plurality of previous periods is collected via a plurality of physically distinct collection systems.
One embodiment provides a system including a module configured to determine point-in-time statistical variations between the current blepharometric data for the human subject and the record of historical blepharometric data for the identified human subject.
One embodiment provides a system wherein identifying a threshold trend includes identifying a threshold trend in one or more of the user's observed blepharometric artefacts that satisfies a predefined profile that is representative of prediction of a neurological condition.
One embodiment provides a system wherein identifying point-in-time statistical deviation includes determining whether, the current set of blepharometric data alone or in combination with one or more recent sets of blepharometric data, display a threshold deviation in one or more of the user's observed blepharometric artefacts compared to historical averages, wherein that deviation is representative of prediction of a neurological condition.
One embodiment provides a system wherein the plurality of sensor systems include a selection of the following:
One embodiment provides a system wherein the plurality of sensor systems includes a plurality of in-vehicle blepharometric data monitoring systems.
One embodiment provides a system wherein, for at least a subset of the in-vehicle blepharometric data monitoring systems, the vehicle is an automobile, and wherein the sensor device is mounted on or adjacent a dashboard or windscreen region.
One embodiment provides a system including multiple sensor devices each mounted in the vehicle positioned to enable monitoring eyelid movement by a respective passenger or operator of the vehicle.
One embodiment provides a system wherein the system includes a cloud-based processing facility.
One embodiment provides a system configured to facilitate monitoring of subject neurological conditions, the system including:
One embodiment provides a portable electronic device including:
One embodiment provides a device wherein the first software application is a messaging application.
One embodiment provides a device wherein the first software application is a social media application.
One embodiment provides computer-executable code that, when executed, causes delivery via a computing device of a messaging software application, wherein the computer-executable code is additionally configured to collect data from a front-facing camera thereby to facilitate analysis of blepharometric data.
One embodiment provides computer-executable code that, when executed, causes delivery via a computing device of a social media software application, wherein the computer-executable code is additionally configured to collect data from a front-facing camera thereby to facilitate analysis of blepharometric data.
One embodiment provides computer-executable code that when executed causes delivery via a computing device of a software application with which a user interacts for a purpose other than blepharometric data-based data collection, wherein the computer-executable code is additionally configured to collect data from a front-facing camera thereby to facilitate analysis of blepharometric data.
Reference throughout this specification to “one embodiment,” “some embodiments” or “an embodiment” means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Thus, appearances of the phrases “in one embodiment,” “in some embodiments” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment, but may. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure, in one or more embodiments.
As used herein, unless otherwise specified the use of the ordinal adjectives “first,” “second,” “third,” etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.
In the claims below and the description herein, any one of the terms “comprising,” “comprised of” or “which comprises” is an open term that means including at least the elements/features that follow, but not excluding others. Thus, the term “comprising,” when used in the claims, should not be interpreted as being limitative to the means or elements or steps listed thereafter. For example, the scope of the expression a device comprising A and B should not be limited to devices consisting only of elements A and B. Any one of the terms “including” or “which includes” or “that includes” as used herein is also an open term that also means including at least the elements/features that follow the term, but not excluding others. Thus, “including” is synonymous with and means “comprising.”
As used herein, the term “exemplary” is used in the sense of providing examples, as opposed to indicating quality. That is, an “exemplary embodiment” is an embodiment provided as an example, as opposed to necessarily being an embodiment of exemplary quality.
Embodiments of the disclosure will now be described, by way of example only, with reference to the accompanying drawings in which:
The present disclosure relates, in various embodiments, to extended monitoring and analysis of subject neurological factors via blepharometric data collection, for example, including devices and processing systems configured to enable such extended monitoring. This may include hardware and software components deployed at subject locations (for example, in-vehicle monitoring systems, portable device monitoring systems, and so on), and cloud-based hardware and software (for example, cloud-based blepharometric data processing systems.
Overview and Context
A human subject's involuntary blinks and eyelid movements are influenced by a range of factors, including the subject's behavioral state and brain function. For example, this has been used in the past for detection of drowsiness. More broadly, analysis of data derived from eye and eyelid movements can be performed thereby to identify data artefacts, patterns and the like, and these are reflective of the subject's behavioral state, brain function and the like.
The technology described herein is focused on collection and analysis of “blepharometric data,” with the term “blepharon” describing a human eyelid. The term “blepharometric data” is used to define data that describes eyelid movement as a function of time. For example, eyelid position may be recorded as an amplitude. Eyelid movements are commonly categorized as “blinks” or “partial blinks.” The term “blepharometric data” is used to distinguish technology described herein from other technologies that detect the presence of blinks for various purposes. The technology herein is focused on analyzing eyelid movement as a function of time, typically measured as an amplitude. This data may be used to infer the presence of what would traditionally be termed “blinks,” however, it is attributes of “events” and other parameters identifiable in eyelid movements that are of primary interest to technologies described herein. Events and other parameters that are identified from the processing of blepharometric data are referred to as “blepharometric artefacts.” These are referred to as “blepharometric artefacts,” with such artefacts being identifiable by application of various processing algorithms to a data set that described eyelid position as a function of time (i.e., blepharometric data). For example, the artefacts may include:
blink total duration (BTD), which is preferably measured as a time between commencement of closure movement that exceeds a defined threshold and completion of subsequent opening movement.
The determination of blepharometric artefacts may include any one or more of:
Analysis of “events,” including relative timing of events, with an “event” being defined as any positive or negative deflection that is greater than a given velocity threshold for a given duration. In this regard, a “blink” is in some embodiments defined as the pairing of positive and negative events that are within relative amplitude limits and relative position limits. There may be multiple events within a given blink, when an eyelid is outside of an “inter-blink” eyelid amplitude range.
Known eyelid movement monitoring systems (also referred to herein as blepharometric data monitoring systems) focus on point-in-time subject analysis. For example, commonly such technology is used as a means for assessing subject alertness/drowsiness at a specific moment, potentially benchmarked against known data for a demographically relevant population. There is a problem in that, for many neurological conditions, point-in-time assessment is inadequate. For example, many neurological conditions are degenerative and/or progressive, and for those and others point-in-time blepharometric data without historical baselines may be of limited usefulness. Currently, however, there is no practical way in which to collect blepharometric data for people, outside of requiring people to subject themselves to specialist testing (which is expensive and for a bulk of the population likely unfeasible).
A solution proposed herein is to deploy blepharometric data collection systems in a range of human environments, being environments in which humans are commonly positioned suitably for blepharometric data collection. Examples considered herein are vehicles (for example, cars, airplanes, trains, and the like), computing devices (for example, smartphones, tablets, and PCs), and other locations. This allows long term blepharometric data collection on an individualized basis, allowing for better management of neurological health (and other factors such as safety). For instance, specific use cases might include providing warnings in advance of seizures, informing a person of a risk of a degenerative brain illness, detection of brain injuries from accidents and/or sporting activities, and personalized detection of unusual levels of drowsiness.
In terms of behavioral state, there are many factors that have an effect on involuntary eyelid movements, with examples including: a subject's state of physical activity; a subject's posture; other aspects of a subject's positional state; subject movement; subject activity; how well slept the subject happens to be; levels of intoxication and/or impairment; and others. In terms of brain function, factors that have effects on involuntary eyelid movements include degenerative brain injuries (e.g., Parkinson's disease) and traumatic brain injuries.
Example Methodology
Block 301 represents a process including collecting data representative of eyelid movement (i.e., blepharometric data). For the majority of embodiments described below, this is achieved via a camera system having an image capture component that is positioned into a capture zone in which a subject's face is predicted to be positioned. For example, this may include:
Vehicles, including passenger vehicles or operator-only vehicles, wherein the image capture component is positioned to capture a region in which an operator's face is predicted to be contained during normal operation. For example, in the case of an automobile, the image capture component may include a camera mounting in or adjacent a dashboard or windscreen.
Vehicles, in the form of passenger vehicles, wherein the image component is positioned to capture a region in which a passenger's face is predicted to be contained during normal operation. For example, in the case of an automobile, the image capture component may include a camera mounting in or adjacent a dashboard or windscreen, the rear of a seat (including a seat headrest), and so on.
Mass transport vehicles, including passenger trains and/or aircraft, wherein the image component is positioned to capture a region in which a passenger's face is predicted to be contained during normal operation. For example, the image capture component may be mounted in the rear of a seat (including a seat headrest), optionally in a unit that contains other electronic equipment such as a display monitor.
Seating arrangements, such as theatres, cinemas, auditoriums, lecture theatres, and the like. Again, mounting image capture components in the rear of seats is an approach adopted in some embodiments.
The data that is captured is not limited to data captured for the purposes of extended monitoring and analysis of subject neurological factors via blepharometric data collection. For example, in some embodiments, that is one purpose, and there is an alternate purpose, which is optionally point-in-time based. For example, point-in-time drowsiness detection is relevant in many of the above scenarios. Furthermore, while embodiments below focus on individualized blepharometric data collection and/or monitoring, collected blepharometric data is optionally additionally collected for the purposes of group monitoring/analysis (including where blepharometric data is anonymized such that it is not attributable to a specific individual). For example, this may be used in the context of seating arrangements to assess overall student/viewer attention/drowsiness, or in the context of airplanes and other mass transport to perform analysis of passenger health factors.
Block 302 represents a process including identifying a subject from whom the blepharometric data collected at block 301 originates. This optionally includes:
Credential-based identification, for example, via a login. This may include pairing of a personal device (such as a smartphone) to blepharometric data monitoring system (e.g., pairing a phone to an in-vehicle system), inputting login credentials via an input device, or other means.
Biometric identification. For example, in some embodiments described herein, a camera-based blepharometric data monitoring system utilizes image data to additionally perform facial recognition functions, thereby to uniquely identify human subjects.
Other Forms of Identification.
Identification of the subject is relevant for the purposes of comparing current blepharometric data with historical blepharometric data for the same subject. For example, in some embodiments, an analysis system has access to a database of historical blepharometric data for one subject (for example, where the system is installed in a vehicle and monitors only a primary vehicle owner/driver) or multiple subjects (for example, a vehicle configured to monitor multiple subjects, or a cloud-hosted system that received blepharometric data from a plurality of networked systems, as described further below).
Block 303 represents a process including determination of blepharometric artefacts for a current time period. For example, the artefacts may include:
The “current period” may be either a current period defined by a current user interaction with a blepharometric data monitoring system, or a subset of that period. For instance, in the context of a vehicle, the “current period” is in one example defined as a total period of time for which a user operates the vehicle and has blepharometric data monitored, and in another embodiment is a subset of that time. In some embodiments, multiple “current periods” are defined, for example, using time block samples of between two and fifteen minutes (which are optionally overlapping), thereby to compare blepharometric data activity during periods of varying lengths (which may be relevant for differing neurological conditions, which, in some cases, present themselves based on changes in blepharometric data over a given period of time).
The current blepharometric data may be used for point-in-time neurological conditional analysis, for example, analysis of subject alertness/drowsiness, prediction of seizures, detection of seizures, and other such forms of analysis. Specific approaches for analyzing blepharometric data thereby to detect/predict particular neurological conditions fall beyond the scope of the present disclosure.
Block 304 represents a process including identification of relationships between current blepharometric artefacts and historical blepharometric artefacts. This allows for artefacts extracted in the current blepharometric data to be given context relative to baselines/trends already observed for that subject. The concept of “identification of relationships” should be afforded a broad interpretation to include at least the following:
Identification of long-term trends. For example, blepharometric data collected over a period of weeks, months or years may be processed thereby to identify any particular blepharometric data artefacts that are evolving/trending over time. In some embodiments, algorithms are configured to monitor such trends, and these are defined with a set threshold for variation, which may be triggered in response to a particular set of current blepharometric data.
Identification of current point-in-time deviations from baselines derived from historical blepharometric data. For example, current data may show anomalous spiking in particular artefacts, or other differences from baselines derived from the subject's historical blepharometric data, which may give rise for concern. By way of example, this form of analysis may be used to determine/predict the presence of: (i) onset of a neurological illness or degenerative condition; (ii) presence of a brain injury, including a traumatic brain injury; (iii) impairment by alcohol, drugs, or other physical condition; (iv) abnormal levels of drowsiness; (v) neurotoxicity; or (vi) other factors.
In relation to onset of a neurological illness or degenerative condition, this may include either or both of short term onsets (e.g., onset of neurological diseases and neurological condition such as strokes and/or seizures and long term onsets (for example, long-term detection rather than the short term is more appropriate for examples such as Alzheimer's, Parkinson's, Multiple Sclerosis, and Muscular Dystrophy).
Block 305 represents a process including identification of presence of one or more blepharometric variation indicators, for example, based on the identification of relationships at block 304. These indicators may be used to allow data-based determination/prediction of the presence of: (i) onset of a neurological illness or degenerative condition; (ii) presence of a brain injury, including a traumatic brain injury; (iii) impairment by alcohol, drugs, or other physical condition; (iv) abnormal levels of drowsiness; (v) neurotoxicity or (vi) other factors. By way of example, rules are defined that associate a data relationship (for example, deviation from baseline values, a trend identification, or the like) with a prediction on neurological condition. These may be defined, for example, using logical structures, such as:
It should be appreciated that these are examples only, and that the present disclosure is directed to hardware and software that enables the implementation of such analysis/alert processes, as opposed to those processes themselves.
Bock 306 represents a process including providing output to the human subject based on identified blepharometric variation indicators. This may include an instruction/suggestion to avoid a particular activity (such as driving), an instruction/suggestion to undertake a particular activity (such as medication, resting, walking around, or the like), or a suggestion to consult a medical expert about a potential neurological condition. The manner by which the output is delivered varies depending on both the nature of the alert/condition, and the hardware environment in place. Examples range from the sending of emails or other messages or the display of information on a local device (for example, an in-vehicle display).
Various hardware/software embodiments configured to enable the above methodology are described below.
Example In-Vehicle Blepharometric Data Monitoring System
The system of
Image capture device 120 is positioned to capture a facial region of a subject. Image capture device 120 is in one embodiment installed in a region of a vehicle in the form of an automobile, for example, on or adjacent the dashboard, windscreen, or visor, such that it is configured to capture a facial region of a driver. In another embodiment, image capture device 120 is positioned on or adjacent the dashboard, windscreen, or visor, such that it is configured to capture a facial region of a front seat passenger. In another embodiment, image capture device 120 is positioned in a region such as the rear of a seat such that it is configured to capture a facial region of a back-seat passenger. In some embodiments, a combination of these are provided, thereby to enable blepharometric data monitoring for both a driver and one or more passengers.
Although the system of
An in-vehicle image processing system 110 is configured to receive image data from image capture device 120 (or multiple image capture devices 120), and process that data thereby to generate blepharometric data. A control module 111 is configured to control image capture device 120, operation of image data processing, and management of generated data. This includes controlling operation of image data processing algorithms, which are configured to:
Algorithms 112 optionally operate to extract additional artefacts from blepharometric data, for example, amplitude-velocity ratios, blink total durations, inter-event durations, and the like. It will be appreciated, however, that extraction of such artefacts may occur in downstream processing.
A blepharometric data management module 113 is configured to coordinate storage of blepharometric data generated by algorithms 112 in user blepharometric data 152. This includes determining a user record against which blepharometric data is to be recorded (in some cases there is only a single user record, for example, where blepharometric data s collected only from a primary driver of an automobile). In some embodiments, the function of blepharometric data management module 113 includes determining whether a set of generated blepharometric data meets threshold data quality requirements for storage, for example, based on factors including a threshold unbroken time period for which eyelid tracking is achieved and blepharometric data is generated.
Memory system 150 includes user identification data 151 for one or more users. As noted, in some embodiments, system 101 is configured to collect and analyze blepharometric data for only a single user (for instance, the primary driver of a vehicle) and includes identification data to enable identification of only that user. In other embodiments, system 101 includes functionality to collect and analyze blepharometric data for multiple users, and includes identification data to enable identification of any of those users (and optionally, as noted above, defining of a new record for a previously unknown user). The identification data may include login credentials (for example, a user ID and/or password) that are inputted via an input device. Alternately, the identification data may be biometric, for example, using facial recognition as discussed above or an alternate biometric input (such as a fingerprint scanner). In some embodiments, this leverages an existing biometric identification system of the vehicle.
User blepharometric data 152 includes data associated with identified users, the data basing time coded thereby to enable identification of a date/time at which data was collected. The blepharometric data stored in user blepharometric data 152 optionally includes blepharometric data generated by algorithms 112 and further blepharometric data derived from further processing of that data, for example, data representing average periodic IEDs and/or BTDs, and other relevant statistics that may be determined over time. In some embodiments, data processing algorithms are updated over time, for example, to allow analysis of additional biomarkers determined to be representative of neurological conditions that require extraction of particular artefacts from blepharometric data.
Analysis modules 130 are configured to perform analysis of user blepharometric data 152. This includes executing a process including identification of relationships between current blepharometric artefacts (e.g., data recently received from in-vehicle image processing system 110) and historical blepharometric artefacts (e.g., older data pre-existing in memory system 150). This allows for artefacts extracted in the current blepharometric data to be given context relative to baselines/trends already observed for that subject. The concept of “identification of relationships” should be afforded a broad interpretation to include at least the following:
Identification of long-term trends. For example, blepharometric data collected over a period of weeks, months or years may be processed thereby to identify any particular blepharometric data artefacts that are evolving/trending over time. In some embodiments, algorithms are configured to monitor such trends, and these are defined with a set threshold for variation, which may be triggered in response to a particular set of current blepharometric data.
Identification of current point-in-time deviations from baselines derived from historical blepharometric data. For example, current data may show anomalous spiking in particular artefacts, or other differences from baselines derived from the subject's historical blepharometric data, which may give rise for concern. By way of example, this form of analysis may be used to determine/predict the presence of: (i) onset of a neurological illness or degenerative condition; (ii) presence of a brain injury, including a traumatic brain injury; (iii) impairment by alcohol, drugs, or other physical condition; (iv) abnormal levels of drowsiness; or (v) neurotoxicity or (vi) other factors.
Analysis modules are optionally updated over time (for example, via firmware updates or the like) thereby to allow for analysis of additional blepharometric artefacts and hence identification of neurological conditions. For example, when a new method for processing blepharometric data thereby to predict a neurological condition based on a change trend in one or more blepharometric artefacts, an analysis algorithm for that method is preferably deployed across a plurality of systems such as system 101 via a firmware update or the like.
System 101 additionally includes a communication system 160, which is configured to communicate information from system 101 to human users. This may include internal communication modules 161 that provide output data via components installed in the vehicle, for example, an in-car display, warning lights, and so on. External communication modules 162 are also optionally present, for example, to enable communication of data from system 101 to user devices (for example, via Bluetooth, WiFi, or other network interfaces), optionally by email or other messaging protocols. In this regard, communication system 160 is configured to communicate results of analysis by analysis modules 130.
A control system 141 included logic modules 140, which control overall operation of system 141. This includes execution of logical rules thereby to determine communications to be provide din response to outputs from analysis modules 130. For example, this may include:
It will be appreciated that these are examples only, and logic modules 140 are able to provide a wide range of functionalities thereby to cause system 101 to act based on determinations by analysis modules 130.
It should be appreciated that the system illustrated in
User-personalized drowsiness detection, based on detection of drowsiness-related blepharometric artefacts that are beyond a threshold deviation from average values for a particular user;
Prediction of neurological conditions, based on sudden changes and/or long term trends in change for one or more blepharometric artefacts that are known to be indicative of particular neurological conditions;
Personalized prediction of future neurological conditions, for example, prediction of future drowsiness based on known drowsiness development patters extracted for the individual from historical data, and prediction of likelihood of a seizure based on individually-verified seizure prediction biomarkers identifiable in blepharometric data.
Identification of point-in-time relevant neurological conditions based on sudden deviations from historical averages, which may be representative of sudden neurological changes, for example, traumatic brain injuries (e.g., concussion) and/or impairment based on other factors (such as neurotoxicity, medications, drugs, alcohol, illness, and so on).
Example In-Vehicle Blepharometric Data Monitoring Systems, With Cloud-Based Analysis
Cloud system 180 includes a control system 182 and logic modules 181 that are provided by computer-executable code executing across one or more computing devices thereby to control and deliver functionalities of cloud system 180.
Cloud system 180 additionally includes a memory system 183, which includes user identification data 184 and user blepharometric data 185. The interplay between memory system 183 and memory system 150 varies between embodiments, with examples discussed below:
In some embodiments, memory system 150 operates in parallel with memory system 183, such that certain records are synchronized between the systems based on a defined protocol. For example, this optionally includes a given memory system 150 maintaining user blepharometric data and user identification data for a set of subjects that have presented at that in-vehicle system, and that data is periodically synchronized with the cloud system. For example, upon an unrecognized user presenting at a given in-vehicle system, the system optionally performs a cloud (or other external) query thereby to obtain identification data for that user, and then downloads from the cloud system historical user blepharometric data for that user. Locally collected blepharometric data us uploaded to the server. This, and other similar approaches, provides for transportability of user blepharometric data between vehicles.
In some embodiments, memory system 150 is used primarily for minimal storage, with system 101 providing a main store for user blepharometric data. For example, in one example, memory system 150 includes data representative of historical blepharometric data baseline values (for instance, defined as statistical ranges), whereas detailed recordings of blepharometric data is maintained in the cloud system. In such embodiments, analysis modules 186 of cloud-based blepharometric data analysis system 180 performed more complex analysis of user blepharometric data thereby to extract the historical blepharometric data baseline values, which are provided to memory system 150 where a given user is present or known thereby to facilitate local analysis of relationships from baselines.
In some embodiments, local memory system 150 is omitted, with all persistent blepharometric data storage occurring in cloud memory system 183.
Cloud system 180 additionally includes analysis modules 186, which optionally perform a similar role to analysis modules 130 in
There are various advantages of incorporating a cloud-based system to operate with a plurality of in-vehicle systems, in particular an ability to maintain cloud storage of user identification data and user blepharometric data for a large number of users, and hence allow that data to “follow” the users between various vehicles over time. For example, a user may have a personal car with a system 101, and subsequently obtain a rental car while travelling with its own system 101, and as a result of cloud system 180 the rental car system: has access to the user's historical blepharometric data; is able to perform relationship analysis of the current data collected therein against historical data obtained from the cloud system; and feed into the cloud system the new blepharometric data collected to further enhance the user's historical data store.
Using a smartphone device as an intermediary between system 101 and cloud system 180 is in some embodiments implemented in a matter that provides additional technical benefits. For example:
In some embodiments, smartphone 170 provides to system 101 data that enabled identification of a unique user, avoiding a need for facial detection and/or other means. For instance, upon coupling a smartphone to a in-car system (which may include system 101 and one or more other in-car systems, such as an entertainment system) via Bluetooth, system 101 receives user identification data from smartphone 170.
In some embodiments, a most-recent version of a given user's historical blepharometric data (for example, defined as historical baseline values) is stored on smartphone 170, and downloaded to system 101 upon coupling.
In some embodiments, one or more functionalities of analysis modules 130 are alternately performed via smartphone 170, in which case, system 101 optionally is configured to, in effect, be a blepharometric data collection and communication system without substantive blepharometric data analysis functions (which are instead performed by smartphone 170, and optionally tailored via updating of smartphone app parameters by cloud system 180 for personalized analysis.
The use of smartphone 170 is also in some cases useful in terms of allowing users to retain individual control over their blepharometric data, with blepharometric data not being stored by an in-vehicle system in preference to being stored on the user's smartphone.
Additional Mass-Transit Functions
A system such as that of
In this example, each image capture device is provided in conjunction with a display screen that is configured to deliver audio-visual entertainment (for instance, as is common in airplanes), monitoring of subject blepharometric data may be used to provide an enhanced experience with respect to audio-visual data. This may include, for example:
It will be appreciated that provision of a system that enables collection and analysis of blepharometric data from multiple passengers in a mass-transit vehicle may have additional far-reaching advantages in terms of optimizing passenger health and/or comfort during transportation.
In mass-transport embodiments, there is preferably a clear distinction between personalizing health data, which is maintained with privacy on behalf of the user, and non-personalizing statistical data, which may be shared with other systems/people. For instance, an individual's neurological conditions are not made available to airline personnel, however data representative of drowsiness/alertness statistics in a cabin are made available to airline personnel.
Example Cloud-Based Extended Blepharometric Data Monitoring Framework
The local systems illustrated in
Vehicle operator configurations 201. These are in-vehicle systems, such as that of
Desktop/laptop computer configurations 202. In these configurations, a webcam or other image capture device is used to monitor user blepharometric data, with image-based eyelid movement detection as discussed herein. This may occur subject to: (i) a foreground application (for example, an application that instructs a user to perform a defined task during which blepharometric data is collected); and/or (ii) a background application that collects blepharometric data while a user engages in other activities on the computer (for example, word processing and/or interne browsing).
Mass-transport passenger configurations 203, for example, airlines as illustrated in
Vehicle passenger configurations 204. These are in-vehicle systems, such as that of
Smartphone/tablet configurations 205. In these configurations, a front facing camera is used to monitor user blepharometric data, with image-based eyelid movement detection as discussed herein. This may occur subject to: (i) a foreground application (for example, an application that instructs a user to perform a defined task during which blepharometric data is collected); and/or (ii) a background application that collects blepharometric data while a user engages in other activities on the computer (for example, messaging and/or social media application usage).
Medical facility configurations 206. These may make use of image processing-based blepharometric data monitoring, and/or other means of data collection (such as infrared reflectance oculography spectacles). These provide a highly valuable component in the overall framework: due to centralized collection of blepharometric data over time for a given subject from multiple locations over an extended period of time, a hospital is able to perform point-in-time blepharometric data collection and immediately reference that against historical data thereby to enable identification of irregularities in neurological conditions.
Beyond advantages of providing an ability to carry user blepharometric data baselines and data collection between physical collection systems, and added benefit of a system such as that of
In embodiments where infrared reflectance oculography techniques are used, the blepharometric data is optionally defined by a reading made by an infrared reflectance sensor, and as such is a proxy for eyelid position. That is, rather than monitoring the actual position of an eyelid, infrared reflectance oculography techniques use reflectance properties and in so doing are representative of the extent to which an eye is open (as the presence of an eyelid obstructing the eye affects reflectivity). In some embodiments, additional information beyond eyelid position may be inferred from infrared reflectance oculography, for example, whether a subject is undergoing tonic eye movement. For the present purposes, “blepharometric data” in some embodiments includes infrared reflectance oculography measurements, and hence may additionally be representative of tonic eye movement.
Example Blepharometric Data Relationship Analysis System
One or more new sets of blepharometric data 501, which may be defined via any collection system, for instance, as shown in
A statistical value determination module 510 applies an expandable set of processing algorithms to data in store 505 thereby to extract a range of statistical values (for example, averages for blepharometric artefacts, optionally categorized based on collection conditions and other factors). These statistical values are stored in data store 505 thereby to maintain richer detail regarding baseline blepharometric data values for the user, preferably in a way that is tied to defined relationship analysis algorithms. That is, if an algorithm X to determine a condition Y relies on analysis of a blepharometric artefact Z, then statistical value determination module 510 is preferably configured to apply an algorithm configured to extract artefact Z from user blepharometric data.
A new data relationship processing module 504 is configured to identify relationships between new data 501 and historical data 505. Data rules to facilitate the identification of particular relationships that are known to be representative (or predictively representative) of neurological conditions are defined in condition identification rules 506. Condition identification rules 506 are periodically updated based on new knowledge regarding blepharometric/neurological condition research. For example, a given rule defines a category of relationship between one or more blepharometric data artefacts in new data 501 and one or more baseline values extracted from historical data in data store 505 based on operation of statistical value determination module 510.
In the case that a defined category of relationship is identified by new data relationship processing module 504, representative data is passed to an output rules module that contains logical rules that define how a user is to be notified (e.g., in-vehicle alert, message to smartphone app, or email), and in response a given output module 509 is invoked to provide the designated output.
A trend analysis module 507 is configured to continuously, periodically or in an event-driven manner (for example, in response to receipt of new blepharometric data), identify trends/changes in user blepharometric data. Again, data rules to facilitate the identification of particular trends that are known to be representative (or predictively representative) of neurological conditions are defined in condition identification rules 506. Condition identification rules 506 are periodically updated based on new knowledge regarding blepharometric/neurological condition research. For example, a given rule defines a threshold deviation in one or more artefacts over a threshold time as being predictively representative of a neurological condition.
Again, the case that a defined category of relationship is identified by trend analysis module 507, representative data is passed to an output rules module 508, which contains logical rules that define how a user is to be notified (e.g., in-vehicle alert, message to smartphone app, or email), and in response, a given output module 509 is invoked to provide the designated output.
It will be appreciated that, in this manner, the system of
It will be appreciated that this form of data collection and analysis is of significant use in the context of predicting and understanding neurological conditions, for example, in terms of: (i) identifying potential degenerative conditions and rates of onset; (ii) identifying point-in-time events that led to sudden changes in neurological conditions; (iii) monitoring long-term effects of contact sports (e.g., concussive brain injuries) for participants, (iv) personalizing blepharometric data analysis for individual users.
Referring to
One embodiment provides computer-executable code that, when executed, causes delivery via a computing device of a software application with which a user interacts for a purpose other than blepharon-based data collection, wherein the computer-executable code is additionally configured to collect data from a front-facing camera thereby to facilitate analysis of blepharon data. The purpose may be, for example, messaging or social media.
Embodiments such as that of
Conclusions and Interpretation
It will be appreciated that the above disclosure provides analytic methods and associated technology that enables improved analysis of human neurological conditions.
It should be appreciated that in the above description of exemplary embodiments of the disclosure, various features of the disclosure are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.
Furthermore, while some embodiments described herein include some but not other features included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the invention, and form different embodiments, as would be understood by those skilled in the art. For example, in the following claims, any of the claimed embodiments can be used in any combination.
Furthermore, some of the embodiments are described herein as a method or combination of elements of a method that can be implemented by a processor of a computer system or by other means of carrying out the function. Thus, a processor with the necessary instructions for carrying out such a method or element of a method forms a means for carrying out the method or element of a method. Furthermore, an element described herein of an apparatus embodiment is an example of a means for carrying out the function performed by the element for the purpose of carrying out the disclosure.
In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the disclosure may be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it is to be noticed that the term coupled, when used in the claims, should not be interpreted as being limited to direct connections only. The terms “coupled” and “connected,” along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other. Thus, the scope of the expression a device A coupled to a device B should not be limited to devices or systems wherein an output of device A is directly connected to an input of device B. It means that there exists a path between an output of A and an input of B, which may be a path including other devices or means. “Coupled” may mean that two or more elements are either in direct physical or electrical contact, or that two or more elements are not in direct contact with each other but yet still co-operate or interact with each other.
Thus, while there has been described what are believed to be the preferred embodiments of the disclosure, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of the invention, and it is intended to claim all such changes and modifications as falling within the scope of the invention. For example, any formulas given above are merely representative of procedures that may be used. Functionality may be added or deleted from the block diagrams and operations may be interchanged among functional blocks. Steps may be added or deleted to methods described within the scope of the present invention.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2018904026 | Oct 2018 | AU | national |
| 2018904027 | Oct 2018 | AU | national |
| 2018904028 | Oct 2018 | AU | national |
| 2018904076 | Oct 2018 | AU | national |
| 2018904312 | Nov 2018 | AU | national |
| 2019900229 | Jan 2019 | AU | national |
This application is a national phase entry under 35 U.S.C. § 371 of International Patent Application PCT/AU2019/051157, filed Oct. 23, 2019, designating the United States of America and published as International Patent Publication WO 2020/082124 A1 on Apr. 30, 2020, which claims the benefit under Article 8 of the Patent Cooperation Treaty to Australian Patent Application Serial Nos. 2018904026, 2018904027, 2018904028, all filed Oct. 23, 2018; Australian Patent Application Serial No. 2018904076 filed Oct. 27, 2018; Australian Patent Application Serial No. 2018904312 filed Nov. 13, 2018; and Australian Patent Application Serial No. 2019900229 filed Jan. 25, 2019.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/AU2019/051157 | 10/23/2019 | WO | 00 |
