Patent Application
20240260872
The present disclosure relates to quantifying mental states. The methods and systems are usable in health monitoring devices, in particular vehicle-based health monitoring devices.
Determining a mental state of an individual, such as a cognitive load of a driver of a vehicle, is a task relevant in fields including vehicle safety. Therefore, there is an interest for a reliable quantification of a mental state.
The following documents relate to the determination of cognitive load and applications thereof:
Disclosed and claimed herein are systems and methods for quantifying a mental state.
A first aspect of the present disclosure relates to a computer-implemented method for quantifying a mental state. The method comprises the following steps:
collecting, as a first training data subset, at least one pair of biosignals, wherein each biosignal is related to an intensity of a mental state of one or more persons;
A mental state refers to a state of mind of one or more persons and may comprise a mood. A mental state may vary gradually in intensity, and the two biosignals indicate two different intensities of the mental state. For example, a mental state may comprise a cognitive load, and the first biosignal may be measured for a person experiencing high cognitive load such as when executing a cognitively demanding task, and the second biosignal may be measured for the same person at a low cognitive load, e.g. when relaxed. However, also other mental states may be chosen, such as stress. Also, mental states are not necessarily limited to one person. Rather, a mental state observed in two persons may be quantified. In that exemplary case, the first biosignal relates one person and the second biosignal relates to the second person. Biosignals may comprise different types of measurable physiological values, such as a heart interbeat interval, or eye openness. Biosignals may be treated immediately after measurement, or pre-recorded and used as an input later.
Annotations comprise binary values, such as Boolean values, that indicate which biosignal is related to a higher intensity of the mental state. For example, the annotation may comprise an indication that the first biosignal is related to a higher intensity of the mental state when the first biosignal is recorded for a person solving a cognitively demanding task, and the second biosignal is recorded for a person not solving a cognitively demanding task.
In a training phase, biosignals and annotations are collected and the artificial neural network is trained to predict values of the annotations. Training may thus comprise predicting, by the artificial neural network, a value for the intensity of the mental state for each biosignal, and adjusting the weights of the artificial neural network such that the predicted intensity for one of the biosignals is higher if the annotation indicates that the biosignal is related to a higher intensity of the mental state. The value for each mental state may preferably be a numerical value. Thereby, by receiving only binary inputs, but training on a dataset comprising a plurality of pairs of biosignals, a qualitative determination of the intensity of a mental state by the artificial neural network is possible.
In an inference phase, a single production biosignal may be treated the intensity of a mental state. Thereby, an artificial neural network trained to predict a value of an intensity of, e.g., cognitive load, may be used to predict cognitive load from new input data.
An advantage of the method is thus that the artificial neural network can be trained on a reliable dataset, since a comparison between two different levels of cognitive load can be made quite reliably. For example, a person can state much more easily, and at a higher level of confidence, which mental state out of two mental states is more intense, as opposed to, e.g., rating a mental state on a scale from one to ten. By training the artificial neural network using such a training dataset, mental states can be predicted reliably.
In an embodiment, the method further comprises pre-processing one or more of the biosignals before supplying the biosignals to the artificial neural network for training and/or processing. Pre-processing comprises removing noise and/or extracting a feature according to one or more predefined criteria.
By pre-processing, steps may be undertaken to reduce the entropy in the data. For example, if a heart interbeat interval is to be determined, camera images of a face of the person may be taken, colour changes over time in one or more positions on the face may be determined, and the signal may be analysed to identify heart interbeat interval by appropriate calculations as known in the art, such as Fourier transform, fitting a model, or using a machine learning based approach to extract the heart interbeat interval. In other embodiments, other types of pre-processing may be used as appropriate for the biosignals.
In a further embodiment, the first training data subset comprises at least two pairs of biosignals comprising the same biosignal. This may imply a step of automatically curating the training dataset. In this embodiment, a first training biosignal is be comprised in two or more different pairs of biosignals in the training dataset, along with annotations indicating the relative intensities. Thereby, the first training biosignal may be compared to two or more different second training signals. This facilitates the training process and reduces the amount of raw data needed for training.
In a further embodiment, training the artificial neural network comprises:
The input layer is thus configured to receive two biosignals and process the biosignals. The two output signals quantify intensities of mental states, and comprise preferably numerical values. They are compared to generate a comparator value, e.g. a Boolean value, indicating which value is higher. Training then comprises adjusting the weights such that the comparator value matches the annotation.
In a further embodiment, the artificial neural network comprises two parts. In this embodiment, each part is configured to receive one input biosignal of the pair of biosignals and to generate one output value indicative of the intensity of the mental state related to the input biosignal. Processing the production dataset comprises supplying the production input dataset to at least one of the parts in this embodiment. A part of the artificial neural network may comprise a plurality of nodes to process one of the input signals. The parts may therefore comprise neural networks themselves.
In a further embodiment, the parts are not in communication to each other. During a training phase, the weights are determined separately for each part.
In a further embodiment, the parts comprise identical node structures. Thereby, the artificial neural network is symmetric in structure with respect to the inputs.
In a further embodiment, training comprises determining a common set of weights for the parts. Thereby the artificial neural network is entirely symmetric with respect to swapping the inputs, such that the result is invariant as to whether a production biosignal is sent to the first part or second part. Training may thus lead to faster convergence as the common set of weights is trained for both parts of the artificial neural network.
In a further embodiment, the mental state comprises one or more of stress, readiness, attention, drowsiness, and/or cognitive load. These mental states can all be quantified based on comparisons.
In a further embodiment, the first biosignal and the second biosignal relate to mental states of the same person at different time intervals.
Thereby, an annotation may be generated by one and the same person, who can compare mental states as experienced. For example, a trend may be identified, i.e. the person may state being more stressed at a first time, when the first biosignal is recorded, than at a second time, when the second biosignal is recorded, or vice versa. The annotation may be set accordingly.
In a further embodiment, the first biosignal and the second biosignal relate to mental states of two different persons. Thereby, differences is physiological reaction to a mental state can be determined.
In a further embodiment, training the artificial neural network comprises supervised learning. For example, the weights may be set by backpropagation such that the comparator value predicts the annotation.
In a further embodiment, training the artificial neural network comprises minimising a mean squared error of the comparator value with respect to the second training data subset.
In a further embodiment, the steps of
In particular, the mobile device may be comprised in a vehicle. Upon inference, the mobile device may thus determine the cognitive load of a driver of a vehicle.
A second aspect of the disclosure relates to a system for quantifying an intensity of a mental state. The system comprises:
All properties and embodiments that apply to the first aspect also apply to the second aspect.
The features, objects, and advantages of the present disclosure will become more apparent from the detailed description set forth below when taken in conjunction with the drawings in which like reference numerals refer to similar elements.
The method 100 begins by collecting two biosignals 102. The biosignals carry information on a mental state. For example, a biosignal may comprise a heartbeat signal, which carries some information of a level of stress experienced by an individual. Other examples of biosignals may include an eye blink rate, an eye openness, electrocardiographic data, or other biosignals. Biosignals may be recorded directly from an individual or be pre-recorded and stored in a memory. At least a second biosignal of the mental state is collected. Here, the signals being of the same mental state refers to the signals being of the same category of a mental state, e.g. both biosignals relate to a level of stress so that the biosignals are comparable. The biosignals can, for example, pertain to the same person and be recorded at different times when the person is experiencing the mental state at a different intensity. For example, the person may be a test user subjected to different levels of stress. In an alternative embodiment, the mental state may comprise a cognitive load, which indicates whether the person is solving a demanding cognitive task. In this example, a first biosignal can be recorded when the person is solving a cognitively demanding problem, such as driving a car, or solving a mathematical problem. A second signal may be taken when the person is not solving a problem but in a relaxed state. However, the present disclosure is not restricted to comparing signals from the same person. For example, biosignals of two different persons may be recorded. Then, a difference between the biosignals carries information on their differences in physiological response to a situation. The biosignals may be pre-processed, 104. Pre-processing steps may comprise any kind of pre-processing known in the art. For example, noise may be removed from the data. Features may be extracted by application of a mathematical model. The biosignals form a first training data subset to be supplied to an input layer of the neural network, as detailed below.
At 106, an annotation of a difference in the mental state related to the biosignals is received. The annotation need not comprise quantitative information. However, the annotation comprises a binary value indicating whether the first or the second biosignal relates to a higher intensity of the mental state. If, in an embodiment, the mental state comprises a cognitive load, then the annotation may be a Boolean value that is true if the person was solving a demanding cognitive task when the first biosignal was recorded. The annotation may also be determined by the person in a self-assessment of a current intensity of a mental state. Thereby, advantage is taken from the fact that humans can give more reliable information on whether one out of two mental states is more intense than by quantifying mental states, e.g. on a scale from zero to hundred. The annotations form a second training data subset.
At 108, the training dataset comprising the first training data subset and the second training data subset is supplied to the artificial neural network. The training dataset comprises at least two biosignals and an annotation. However, for training the neural network, preferably a plurality of data triples are used, wherein each triple comprises two biosignals and one annotation indicating which biosignal relates to a higher intensity of the mental state. Using this training dataset, the artificial neural network is trained, 110, to predict a numerical value quantifying the mental state. The output of the neural network therefore comprises at least one numerical value relating to a relative intensity of the mental state, as described with reference to
The output signals 322, 324 quantify the intensity of the associated mental state. They may be expressed as numerical value, or encoded differently. The output signals 322, 324 are supplied to the comparator 326 that yields an output 328 as to which of the output signals 322, 324 is higher. The comparator 326 therefore provides a value 328 that is to predict the annotations of the second training data subset. Training on a training dataset therefore leads to output signals 322 that cause the comparator to yield an accurate prediction of the annotations.
The server 402 and the client device 414 are in communication via a network 412. The network enables the client device 414 to exchange data with the server, comprising receiving weights of an updated version of an artificial neural network, and/or sending one or more biosignals to the server for analysis and/or one or more pre-processing steps.
In an embodiment, the client device 414 may be a mobile device. In particular, it may be comprised in a vehicle and configured to determine drowsiness of a driver. The determination of drowsiness may allow the vehicle electronics to react, e.g. by alerting the driver by an acoustic signal or by sending a warning to other vehicles in proximity. In this illustrative example, the sensor may comprise one or more cameras to observe the driver.
In alternative embodiments, the client device 414 may be a stationary device.
From the camera images, an eye openness value may be determined, from which, e.g., a blink rate may be determined as a pre-processing step. Alternatively, a heart rate may be determined by remote photoplethysmography. The biosignal may then either be analysed on the client device by execution of method 200 by the processing unit, or sent via the network to the server 402 for processing.
It should be noted that the present disclosure is not limited to the above embodiment. The methods described with reference to
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/RU2021/000224 | 5/28/2021 | WO |
