This application claims the benefit of and priority to French patent application number 19 07512 filed on Jul. 5, 2019, the entire disclosure of which is incorporated by reference herein.
The disclosure herein relates to monitoring a state of consciousness of an operator, in particular an aircraft pilot, in a cockpit of an aircraft.
Monitoring the state of consciousness of an operator in his cockpit, for example a pilot on board an aircraft, is a vital action in order to detect a loss of consciousness, which may lead to the pilot being unable to perform his activities. The definition of a loss of consciousness may be as follows: loss of the ability to maintain awareness of self and environment combined with markedly reduced responsiveness to environmental sensory stimuli.
At present, the state of consciousness of an operator is monitored by human beings, in particular a copilot or a flight crew member. In order to supplement monitoring by the copilot, it would be desirable to provide an automated solution for detecting loss of consciousness.
The disclosure herein aims to overcome these drawbacks by proposing a method and a system for automatically monitoring the state of consciousness of an operator.
To this end, the disclosure herein relates to a method for monitoring a state of consciousness of at least one operator in an aircraft cockpit.
According to the disclosure herein, the method comprises at least one set of steps comprising at least the following steps, implemented automatically and iteratively:
Thus, by virtue of the method, the state of consciousness of the operator or operators is monitored automatically in order to supplement the monitoring by the flight crew members.
The method furthermore comprises:
The method additionally comprises:
According to a first embodiment, the first movement or movements of the body of the operator or operators correspond to at least one blink of at least one eye of the operator or operators, the first movement feature corresponding to a number of blinks of at least one eye, the first movement feature meeting the predetermined first condition if the number of blinks of at least one eye is less than a predetermined number of blinks of at least one eye and the second movement or movements of the body of the operator or operators corresponding to at least one head direction movement of the operator or operators, the second movement feature corresponding to a head direction movement speed, the second movement feature meeting the predetermined second condition if the head direction movement speed is less than a predetermined head direction movement speed.
According to a second embodiment, the first movement or movements of the body of the operator or operators correspond to at least one head direction movement of the operator or operators, the first movement feature corresponding to a head direction movement speed, the first movement feature meeting the predetermined first condition if the head direction movement speed is less than a predetermined head direction movement speed and the second movement or movements of the body of the operator or operators corresponding to at least one blink of at least one eye of the operator or operators, the second movement feature corresponding to a number of blinks of at least one eye, the second movement feature meeting the predetermined second condition if the number of blinks of at least one eye is less than a predetermined number of blinks of at least one eye.
According to a third embodiment, the first movement feature corresponds to a sequence of movements of at least one body part of the operator or operators, the first movement feature meeting the predetermined first condition if the sequence of movements is similar to a predetermined sequence of movements, the first movement feature not meeting the predetermined first condition if the sequence of movements is not similar to the predetermined sequence of movements.
According to a first variant, the method comprises:
According to a second variant, the method comprises:
According to a third variant, the method comprises an initialization step, implemented by an initialization module, the initialization step comprising at least one first detection substep, implemented by a first detection submodule, consisting in or comprising detecting an action of the operator or operators on at least one control device on board the aircraft, the set of steps being implemented if no action has been detected during a predetermined first period.
According to a fourth variant, the method comprises:
According to a first embodiment, the third movement feature corresponds to a head direction movement speed, the third movement feature meeting the predetermined third condition if the head direction movement speed is greater than a predetermined head direction movement speed, the third movement feature not meeting the predetermined third condition if the head direction movement speed is less than or equal to the predetermined head direction movement speed.
According to a second embodiment, the third movement feature corresponds to a sequence of head direction movements, the third movement feature meeting the predetermined third condition if the sequence of head direction movements is similar to a predetermined sequence of head direction movements, the third movement feature not meeting the predetermined third condition if the sequence of head direction movements is not similar to the predetermined sequence of head direction movements.
According to a fifth variant, the initialization step comprises a second detection substep, implemented by a second detection submodule, consisting in or comprising detecting a pressure exerted by the mass of the operator or operators on a seat on which they are likely to sit, the set of steps being implemented:
In this fifth variant, the method furthermore comprises:
According to a sixth variant, the method comprises:
The disclosure herein also relates to a system for monitoring a state of consciousness of at least one operator in an aircraft cockpit.
According to the disclosure herein, the system comprises at least one set of modules comprising at least:
The system additionally comprises:
The system furthermore comprises:
The disclosure herein also relates to an aircraft, in particular a transport plane, comprising a system for monitoring a state of consciousness of at least one operator in an aircraft cockpit, as described above.
The disclosure herein, with its features and advantages, will become more clearly apparent upon reading the description provided with reference to the appended drawings, in which:
The disclosure herein relates to a system 1 for monitoring a state of consciousness of an operator in an aircraft AC cockpit 2 (
One embodiment of the monitoring system 1 is shown in
The monitoring system comprises an initialization module INIT (INIT for initialization module) 3 configured so as to initialize the monitoring system 1, The initialization module 3 makes it possible to implement the monitoring system 1 under given conditions. For example, the given conditions may be the presence of an operator in the cockpit 2. Some embodiments of the initialization module 3 are presented below in the description.
The monitoring system 1 furthermore comprises:
The determination module 6 is configured so as to be implemented if at least one first movement of the body of the operator or operators is detected by the detection module 5 during at least the first time proportion.
The detection by the detection module 5 may be unavailable at certain times during the time window and available at other times during the time window. For example, the detection may be unavailable at times when the operator turns and his face is no longer visible to the image acquisition module 4, at times when there is an obstacle (book or other objects) between the face of the operator and the image acquisition module 4, or else at times when the image acquisition module 4 has an intermittent technical problem. A time proportion (or percentage) of a time window then corresponds to the ratio between the sum of the times at which detection is available during the time window, on the one hand, and the time window, on the other hand. This definition of a time proportion is the same in the remainder of the description.
The movement or movements of the body of the operator or operators may be one or more movements of at least one body part of the operator or operators. The body part or parts may correspond to at least one organ of the operator or operators that is able to be identified and/or monitored by the image acquisition module 4 and that is likely to move in a known way.
The transmission module 7 is configured so as to be implemented if the first movement feature determined by the determination module 6 meets a predetermined first condition.
The image acquisition module may correspond to a camera or may comprise a camera.
The monitoring system may furthermore comprise a detection module DETECT2 9 configured so as to detect at least one second movement of the body of the operator or operators from the images.
According to a first embodiment of the detection module 9, the detection module 9 is configured so as to be implemented if no first movement of the body of the operator or operators is detected by the detection module 5 during the first time proportion. In
According to a second embodiment of the detection module 9, the detection module 9 is configured so as to be implemented if the first movement feature determined by the determination module 6 meets the predetermined first condition. In
The monitoring system 1 may also comprise a determination module DET2 10 configured so as to determine a second movement feature of the body of the operator or operators during at least one second time proportion of a second time window.
The determination module 10 is configured so as to be implemented if at least one second movement of the body of the operator is detected by the detection module 9 during at least the second time proportion.
The monitoring system 1 may comprise a transmission module TRANS2 11 configured so as to transmit a signal representative of a state of loss of consciousness of the operator to the user device. The transmission module 11 is implemented if the second movement feature meets a predetermined second condition.
Advantageously, the monitoring system 1 comprises a transmission module TRANS3 12 configured so as to transmit a signal representative of a state of possible malfunctioning of the movement detection to the user device 8. Thus, if the detection modules 5 and 9 do not detect any first movement or possibly any second movement of the body of the operator or operators, the signal is transmitted to the user device 8 in order to indicate possible malfunctioning of the detection modules 5 and/or 9.
The transmission module 12 is implemented if no second movement of the body of the operator or operators is detected during at least the second time proportion.
According to a first embodiment, the first movement or movements of the body of the operator or operators correspond to at least one blink of the eye or eyes of the operator or operators and the second movement of movements of the body of the operator or operators correspond to a head direction movement of the operator or operators.
The first movement feature corresponds to a number of blinks of at least one eye. The first movement feature meets the predetermined first condition if the number of blinks of at least one eye is less than a predetermined number of blinks of at least one eye. The first movement feature does not meet the predetermined first condition if the number of blinks of at least one eye is greater than or equal to the predetermined number of blinks of at least one eye.
The second movement feature corresponds to a head direction movement speed. The second movement feature meets the predetermined second condition if the head direction movement speed is less than a predetermined head direction movement speed. The second movement feature does not meet the predetermined second condition if the head direction movement speed is greater than or equal to the predetermined head direction movement speed.
According to a second embodiment, the first movement or movements of the body of the operator or operators correspond to at least one head direction movement of the operator or operators and the second movement of movements of the body of the operator or operators correspond to at least one blink of the eye or eyes of the operator or operators.
The first movement feature corresponds to a head direction movement speed. The first movement feature meets the predetermined first condition if the head direction movement speed is less than a predetermined head direction movement speed. The first movement feature does not meet the predetermined first condition if the head direction movement speed is greater than or equal to the predetermined head direction movement speed.
The second movement feature corresponds to a number of blinks of at least one eye. The second movement feature meets the predetermined second condition if the number of blinks of at least one eye is less than a predetermined number of blinks of at least one eye. The second movement feature does not meet the predetermined second condition if the number of blinks of at least one eye is greater than or equal to the predetermined number of blinks of at least one eye.
Without limitation for the first embodiment and the second embodiment, in the case of blinks of the eye or eyes, the predetermined number of movements is between 0.5 and 5 blinks per minute and the time proportion is between 20% and 80% of the time window. In the case of head direction movements, the predetermined head direction movement speed is between 0 m/s and 10 m/s and the time proportion is between 20% and 80% of the time window.
According to a third embodiment, the first movement feature corresponds to a sequence (or pattern) of movements of at least one body part of the operator or operators. The first movement feature meets the predetermined first condition if the sequence of movements is similar to a predetermined sequence of movements. The first movement feature does not meet the predetermined first condition if the sequence of movements is not similar to the predetermined sequence of movements.
A sequence of movements may correspond to a set of successive or simultaneous or both successive and simultaneous movements of one or more body parts of the operator or operators. The movements may be at least blinks of at least one eye, head movements or shoulder movements. According to one non-limiting example, a sequence of movements corresponds to head movements followed by one or more eye blinks.
Without limitation, the similarity between a detected sequence and a predetermined sequence may be determined through movement recognition by way of image processing. The detected sequence is similar to the predetermined sequence when the detected sequence is close to the predetermined sequence, within a confidence interval.
According to a first variant embodiment, the monitoring system 1 comprises:
The determination module 13 is configured so as to be implemented if the head direction speed determined by the determination module 10 is greater than or equal to the predetermined head direction movement speed.
The transmission module 14 is configured so as to be implemented if the head orientation determined by the determination module 13 is within a predetermined orientation range during at least one third time proportion of the second time window.
Without limitation, in the case of a head orientation in terms of pitch (with respect to the aircraft AC), the lower limit of the orientation range is between −60° and −10° and the upper limit is between 10° and 90°.
Without limitation, in the case of a head orientation in terms of roll (with respect to the aircraft AC), the lower limit is between −90° and −10° and the upper limit is between 10° and 90°.
According to a second variant, the monitoring system 1 comprises a detection module DETECT3 15 configured so as to detect a shoulder movement of the operator or operators from the images acquired by the image acquisition module 4.
According to a first embodiment of the detection module 15, the detection module 15 is configured so as to be implemented if the second movement feature determined by the determination module 10 does not meet the predetermined second condition.
According to a second embodiment of the detection module 15, the detection module 15 is configured so as to be implemented if the head orientation determined by the determination module 13 is greater than or equal to the predetermined orientation during at least the third time proportion.
In this second variant, the monitoring system 1 furthermore comprises:
The determination module 16 is implemented if at least one shoulder movement of the operator is detected by the detection module 15 during at least the third time proportion.
The transmission module 17 is configured so as to be implemented if the shoulder movement speed determined by the determination module 16 is less than a predetermined shoulder movement speed.
Without limitation, the predetermined shoulder movement speed is between 0 m/s and 10 m/s.
According to a third variant, the initialization module furthermore comprises a detection submodule SUB_DETECT1 31 configured so as to detect an action of the operator or operators on at least one control device on board the aircraft AC. The modules of the monitoring system 1 may be implemented if no action has been detected during a predetermined first period.
The on-board control device may correspond to a human-machine interface (HMI).
The modules of the monitoring system 1 may preferably be implemented if no action has been detected during a predetermined first period on at least two separate control devices. This makes it possible to avoid detecting involuntary actions of the operator or operators.
According to a fourth variant, the monitoring system 1 furthermore comprises:
The detection module 18 is configured so as to be implemented if no movement of the second body part of the operator or operators has been detected by the detection module 9 during at least the fourth time proportion. One example of not detecting movements of the second body part of the operator or operators may correspond to the case where the operator or operators turn(s) or lean(s) forward in order to speak to another operator or to access a particular area of the cockpit 2.
The determination module 19 is configured so as to be implemented if at least one head direction movement of the operator or operators is detected by the detection module 18 during at least the fourth time proportion.
The transmission module 20 is configured so as to be implemented if the third head direction movement feature meets a predetermined third condition.
According to a first embodiment, the third movement feature corresponds to a head direction movement speed. The third movement feature meets the predetermined third condition if the head direction movement speed is greater than a predetermined head direction movement speed. The third movement feature does not meet the predetermined third condition if the head direction movement speed is less than or equal to the predetermined head direction movement speed.
According to a second embodiment, the third movement feature corresponds to a sequence of head direction movements. The third movement feature meets the predetermined third condition if the sequence of head direction movements is similar to a predetermined sequence of head direction movements. The third movement feature does not meet the predetermined third condition if the sequence of head direction movements is not similar to the predetermined sequence of head direction movements.
The determination module 21 is configured so as to be implemented if no head direction movement of the operator or operators has been detected by the detection module 18.
The transmission module 22 is configured so as to be implemented if the direction angle determined by the determination module 21 is less than a predetermined direction angle.
The transmission module 23 is configured so as to be implemented if the direction angle determined by the determination module 21 is greater than or equal to the predetermined direction angle.
According to a fifth variant, the initialization module comprises a detection submodule SUB_DETECT2 32 configured so as to detect a pressure exerted by the mass of the operator or operators on at least one seat on which the operator or operators are likely to sit. The modules of the monitoring system 1 may be implemented if at least one operator is sitting on the seat.
The seat may be a pilot's seat.
The detection submodule 32 may correspond to a pressure mat integrated into the seat.
The monitoring system 1 may thus be initialized when at least one operator is seated.
According to a first embodiment of the detection submodule 32, the monitoring system 1 is initialized if the pressure has been detected by the detection submodule 32 during a predetermined second period.
According to a second embodiment of the detection submodule 32, the monitoring system 1 is initialized if an action of the operator has been detected by the detection submodule 31 during the predetermined first period and if the pressure has been detected by the detection submodule 32 during the predetermined second period.
In this fifth variant, the monitoring system 1 furthermore comprises:
According to a first embodiment of the determination module 24, the determination module 24 is configured so as to be implemented if no second movement of the body of the operator has been detected by the detection module 9. If the determination module 24 corresponds to the pressure mat, the determination module 24 may monitor a difference in mass distribution between the various parts of the body of the pressure mat.
According to a second embodiment of the determination module 24, the determination module 24 is configured so as to be implemented if the direction angle determined by the determination module 21 is greater than or equal to the predetermined direction angle.
The transmission module 25 is configured so as to be implemented if the distribution of the pressure exerted by the mass of the operator or operators on the seat has a difference proportion, in comparison with a reference pressure distribution, less than a predetermined proportion. For example, in the case of a pressure difference between two parts of the pressure mat that is greater than a predetermined threshold, it is considered that the operator or operators is or are leaning too far to one side due to a loss of consciousness.
Without limitation, the predetermined proportion is between 0% and 50%.
According to a sixth variant, the monitoring system 1 furthermore comprises a detection module DETECT5 26 configured so as to detect a heartbeat of the operator or operators.
According to a first embodiment of the detection module 26, the detection module 26 is implemented if no second movement of the body of the operator or operators has been detected by the detection module 9.
According to a second embodiment of the detection module 26, the detection module 26 is implemented if the distribution of the pressure exerted by the mass of the operator or operators on the seat determined by the determination module 24 has a difference proportion, in comparison with the reference pressure distribution, greater than or equal to the predetermined proportion.
In this sixth variant, the monitoring system 1 also comprises:
The determination module 27 is configured so as to be implemented if at least one heartbeat of the operator or operators is detected by the detection module 26.
The transmission module 28 is configured so as to be implemented if the heart rate determined by the determination module 27 is not within a predetermined heart rate range.
Without limitation, the lower limit of the heart rate range is between 0 beats per minute and 60 beats per minute, and the upper limit of the heart rate range is between 140 beats per minute and 220 beats per minute.
The transmission module 29 is configured so as to be implemented if the heart rate determined by the determination module 27 is within the predetermined heart rate range.
The variants of the method may be combined as shown in
The disclosure herein also relates to a method for monitoring a state of consciousness of at least one operator in an aircraft AC cockpit 2 (
As shown in
The method may also comprise:
The method may furthermore comprise a transmission step E9, implemented by the transmission module 12, consisting in or comprising transmitting the signal representative of a possible state of malfunctioning of the movement detection to the user device 8, the transmission step E9 being implemented if no second movement of the body of the operator or operators is detected during at least the second time proportion.
According to the first variant, the method may comprise:
According to the second variant, the method may comprise:
As shown in
According to the third variant shown in
According to the fourth variant shown in
According to the fifth variant shown in
In this case, the method furthermore comprises:
According to the sixth variant shown in
The monitoring method may correspond to a combination of the variants described above, as shown in
The subject matter disclosed herein can be implemented in software in combination with hardware and/or firmware. For example, the subject matter described herein can be implemented in software executed by a processor or processing unit. In one exemplary implementation, the subject matter described herein can be implemented using a computer readable medium having stored thereon computer executable instructions that when executed by a processor of a computer control the computer to perform steps. Exemplary computer readable mediums suitable for implementing the subject matter described herein include non-transitory devices, such as disk memory devices, chip memory devices, programmable logic devices, and application specific integrated circuits. In addition, a computer readable medium that implements the subject matter described herein can be located on a single device or computing platform or can be distributed across multiple devices or computing platforms.
While at least one example embodiment of the invention(s) is disclosed herein, it should be understood that modifications, substitutions and alternatives may be apparent to one of ordinary skill in the art and can be made without departing from the scope of this disclosure. This disclosure is intended to cover any adaptations or variations of the example embodiment(s). In addition, in this disclosure, the terms “comprise” or “comprising” do not exclude other elements or steps, the terms “a”, “an” or “one” do not exclude a plural number, and the term “or” means either or both. Furthermore, characteristics or steps which have been described may also be used in combination with other characteristics or steps and in any order unless the disclosure or context suggests otherwise. This disclosure hereby incorporates by reference the complete disclosure of any patent or application from which it claims benefit or priority.
Number | Date | Country | Kind |
---|---|---|---|
1907512 | Jul 2019 | FR | national |