Patent Application
20160370774
The present invention more particularly relates to the control of a mobile apparatus taking into account brain data.
A preferred application relates to the industry of wheelchairs for disabled persons.
TECHNOLOGICAL BACKGROUND
Most of the current mobile apparatus are controlled by partially mechanical controls. A user must thus use a joystick, a wheel, a bar, or any other manual control system to steer a mobile apparatus. Unfortunately, such apparatus cannot be used by some persons, specifically tetraplegic persons. New means for controlling a mobile apparatus may be interesting to even non tetraplegic persons. This could more particularly enable the user to use his/her limbs for other tasks, not related to the control of the mobile apparatus. A need thus exists for a technical solution for controlling a apparatus, without any mechanical control.
As regards this latter field, the solution disclosed in document CN103263324 is known. Such solution makes it possible to control a wheelchair using brain data of the SSVEP (Steady-State Visual-Evoked Potentials) type (or PEVRP (Potentiel Evoqué Visuel de Régime Permanent) in French). Such technique makes it possible to control said wheelchair using an analysis of the brain waves emitted by the user upon a stimulation of his/her eyes by a specific frequency (from 5 Hz). A continuous or harmonic frequency response is generated at the visual region of the cerebral cortex with the same frequency of occurrence of the stimulus. Once the match is established between a specific light frequency and the answer thereto by the brain, the image visualized by the user's eye can be determined, by analysing an electroencephalography (EEG) Sensors detecting the obstacles are provided on the wheelchair to enhance the user's safety.
Such solution has some disadvantages. As a matter of fact, controlling a mobile apparatus using visual controls is difficult and tiring. This results in a visual saccade, a sporadic focussing of the eye and a reduced reliability of the control. The motions of the apparatus are thus less precise, the user's safety is not optimum, and the user's exhaustion makes an extended use impossible.
A need therefore exists, which consists in providing a solution making it possible to eliminate, or at least reduce some of the above-mentioned drawbacks.
One aspect of the invention more particularly relates to a method for is controlling the motion of a mobile apparatus by a user, wherein the motion control is based on a motion directive, with said motion directive comprising a path directive and a speed directive, comprising a step of determining said motion instruction.
This method is advantageously so designed that the step of determination comprises the following steps, implemented by a computer using at least one microprocessor:
Such provision enables to control a mobile apparatus, taking into account visual, cerebral and physiological data. Said user is then assisted in controlling the apparatus, which significantly reduces the fatigue resulting from the use thereof. Besides, the user's safety is enhanced if the environment is taken into consideration. The user's motion instruction is thus first validated by his/her brain data, which makes it possible to avoid undesirable motions. Besides, such motion instruction is assimilated with control data originating from brain and physiological data. Such control data makes it possible to still sharpen the user's will or non-will while taking into account data such as stress or fatigue.
The general control of the apparatus thus becomes less tiring, safer and more precise.
The invention also relates to a mobile apparatus, the motion of which is controlled through the method.
Such apparatus advantageously comprises various types of sensors so configured as to detect at least one data according to a user's gaze point, brain data and physiological data, as well as space data.
Such solution makes it possible to collect the data required for a correct implementation of the method. Operating the mobile apparatus thus becomes easier and less tiring thanks to such sensors.
Other characteristics, aims and advantages of the present invention will appear upon reading the following detailed description and referring to the appended drawings given as non-limiting examples and wherein:
Prior to going into details relating to the preferred embodiments of the invention while referring more particularly to the drawings, other optional characteristics of the invention which may be implemented in any combination or alternately, are mentioned hereafter:
In order to enable a perfect understanding of the terms used in the present description, and unless otherwise mentioned, the following expressions will means:
PEVRP (Potentiel Evoqué de Régime Permanent) in French: brain activity in response to an eye stimulation. According to the light frequency detected by the eye, the brain emits a continuous or harmonic frequency response at the visual region of the cerebral cortex with the same frequency of occurrence of the stimulus.
A preferred embodiment of the invention relates to a method for controlling the motion of a wheelchair for a disabled person. The present invention is of course not limited to such type of mobile apparatus and an application to the control of a car, a plane or a drone for instance is possible. More generally, any mobile apparatus controlled by a user can use said method to move.
In order to generate said motion directive 500, the method comprises first the acquisition of user's data 100 and space data 400.
User's data 100 comprise at least one of the following data, and preferably all of these: a user's gaze point 103 or a sequence of the user's gaze points, at least one brain data of the P300 110 or SSVEP 120 type, and preferably according to two types of brain waves P300 101 and SSVEP 102 as well as physiological data. Physiological data preferably comprise at least one is among the following data: heart rhythm 104 and temperature 105 data. Other types of data may be added, of course. Brain waves are sensed by at least one EEG (Electroencephalography) sensor 106.
P300 brain data 110 are determined by the difference between the P300 brain waves 101 upon a change in the amplitude. The SSVEP brain data 120 are determined by the difference in the SSVEP brain waves 102 upon the change in the power spectral density thereof (PSDC). To determine the difference in the amplitude of a brain wave of the P300 type 101, the following steps will advantageously be executed:
In order to determine the difference in the change in PSDC of the SSVEP waves 102, the following steps are executed:
Such steps are illustrated in
The motion instruction 206 comprises the analysis and the interpretation of said at least one gaze point 103 or the sequence of a user's gaze points and of at least one first and preferably two first brain data of the P300 110 and/or SSVEP 120 types (
The user's gaze point 103 is acquired by an eye-tracking monitor 107 preferably positioned under the screen 202. In other embodiments, the sensor is positioned above the screen, or in any other position making it possible to optimally detect the user's gaze. The eye-tracking apparatus 107 advantageously makes it possible to detect the user's gaze point on the screen 202.
The light frequency of each one of the grid cells ranges from 10 Hz to 25 Hz. In a preferred embodiment of the invention, a representation of the environment facing the mobile apparatus is displayed behind the grid 201. In order to show the environment facing the apparatus, a first camera is provided on the front face of the mobile apparatus. A second camera is advantageously provided at the back of the mobile apparatus. Such second camera also makes it possible to display the environment behind the rear face of the mobile apparatus. No camera is shown in the figures. Information is displayed on the screen 202 when the mobile apparatus moves backward.
Determining the motion instruction 206 thus comprises the selection, by the user, of a direction 203 displayed in one cell of the grid 201. Selecting a cell in the grid 201 is a stimulation of the brain which will generate a brain wave of the P300 101 type. Besides, as the cell has a specific light frequency, a SSVEP brain wave 102 will also be generated. The P300 101 and SSVEP 102 brain waves will make it possible to determine a first P300 110 and SSVEP 120 brain data. The fuzzy logic system simultaneously takes into account the first two is brain data of the P300 110 and SSVEP 120 type for the motion instruction. An exemplary assimilation of the first P300 and SSVEP 204 data by fuzzy logic is given in
The interpretation of said first brain data of the P300 110 and SSSVEP 120 types by the fuzzy logic system will make it possible to execute one step of validation 205 of the selected direction 203.
Thus, if the difference between CT and CDT+300 ms is less than or equal to 100 ms and/or the difference between CF and CDF is greater than Y, then the direction is validated by the first SSVEP brain data 110 and/or the first SSVEP data. If the difference between CT and CDT+300 ms is greater than 100 ms and/or the difference between CF and CDF is greater than Y, then the direction is not validated by the first SSVEP brain data 120 and/or the first SSVEP data.
If the direction selected by the user is validated, a path instruction is generated with a validated status.
If the direction selected by the user is not validated, a path directive is generated with a pending status.
Said validated 210 or pending 220 path directive is then integrated into the motion instruction.
The method for validating 205 the selected direction 203 using at least a first brain data of the P300 110 and SSVEP 120 type is illustrated in
The control data 340 will advantageously make it possible, through the s step of selecting a control mode 350 of the mobile apparatus among the following ones: manual, semi-autonomous and autonomous. In other embodiments, control modes 350 may be added or suppressed. The selected control mode 350 will be integrated in the motion instruction.
Said control data 340 comprises at least one physiological and/or physiological data 331, at least one second brain data of the P300 110 or SSVEP 120 types and/or alpha and/or beta 130 frequency band, at least one pre-recorded data originating from a <<fatigue>> 310 data base and another data originating from an <<emotion>> 320 data base. The control data 340 preferably comprises two user's second brain data of the P300 110 and/or SSVEP 120 and/or alpha and/or beta frequency band 130 types and one physiological and/or psychological 331 data according to the heart rhythm (heart rhythm data 104) and temperature (temperature data 105). Such data, whether combined together or not, make it possible to determine the user's state of fatigue 312 or emotional state 322. The <<fatigue>> data base contains the alpha and beta brain waves as well as the maximum amplitudes and the time of occurrence of
P300 at the EEG sensors.
Advantageously, the first brain data of the P300 110 and/or SSVEP 120 type and the second brain data of the P300 110 and/or SSVEP 120 type originate from the same P300 101 and/or SSVEP 102 brain waves. The <<emotion>> data base preferably contains the standardized variations of the asymmetries of the alpha and beta frequency bands at the parietal, central and frontal regions of the cerebral cortex. Besides, it integrates the changes in the heart rhythm which are correlated with the various emotional states as well as the body temperature data.
The user's fatigue state 312 advantageously depends on a second P300 110 and/or SSVEP 120 brain data and at least one data pre-recorded in a <<fatigue>> data base 310 Said second P300 110 and/or SSVEP 120 brain data is preferably the same as the first brain data previously used by the motion instruction 206. The interpretation of the difference between CT and CDT+300 ms and/or between CF and CDF and is different. Besides, the simultaneous taking into account 311 of the P300 101 and/or SSVEP 102 brain waves by the control data 340 to determine the fatigue state 312 is executed using the Dempster-Shafer theory or theory of evidence. In such configuration, the difference between these values makes it possible to determine the user's state of fatigue 312. As a matter of fact, a correlation exists between the user's fatigue state and the difference with the maximum amplitude as well as the duration of occurrence of P300 and the noted difference of the CF maximum amplitude. Now, if the difference between the standardized amplitudes of P300 is less than 10% and/or the difference in the amplitudes of the prevailing frequency of SSVEP is less than 15%, a medium fatigue state is determined. In a preferred, but not restrictive, embodiment of the invention, four levels of fatigue can be determined (high, medium, low and no fatigue). In other embodiments of the invention, the number of fatigue levels may vary positively or negatively. Matching the differences in the P300 110 and/or SSVEP 120 brain data and the fatigue state 312 is possible when comparing such differences with the <<fatigue>> data base 310 and integrating plausibility rules. The method for determining the state of fatigue 312 is illustrated in
An emotional state 322 is advantageously generated too according to the physiological data and the at least one second P300 110 and/or SSVEP 120 brain data and/or one alpha and/or beta 130 data and at least one data pre-recorded in an <<emotion>> data base 320.>>. The evolution of the alpha and/or beta waves 112 over time is correlated with the expressed emotional state. The difference between the standardized amplitude of such waves is once again computed. If the latter increases by 15%, a stress is detected. The user's emotional state 322 can be determined when same are associated with the user's heart rhythm 104 and temperature 105 data. The logic system enabling to interpret and assimilate 321 such multiple data more particularly consists of neural networks. Other theories can of course also be used for interpreting such data.
Advantageously, the first brain data of the P300 110 and/or SSVEP 120 type and the second brain data of the P300 110 and/or SSVEP 120 type originate from the same P300 101 and/or SSVEP 102 brain waves. The emotional state 322 is advantageously determined among the following four states: stress, nervousness, relaxation, excitation. The number of present emotional states 322 is not restrictive, and adding or eliminating emotional states is possible. The simultaneous taking into account 321 of the brain data, the physiological data (temperature and heart rhythm) as well as at least one data originating from the <<emotion>> data base 322 is executed by a neural network. Such method for determining the user's emotional state 322 is illustrated in
Once the user's state of fatigue 312 and emotional state 322 are determined, combining such states 330, using a fuzzy logic system, makes it possible to determine the user's psychological state 331. The control mode 350 is will then be selected according to said user's psychological state 331. Once the control mode 350 is selected, said selection will be integrated in the motion directive.
Space data 400 advantageously depends on at least one environment sensor. In one preferred embodiment of the invention at least one environment sensor comprises at least one motion sensor 402 and at least one ultrasonic sensor 401, and preferably four motion sensors 402 and ten ultrasonic sensors 401 (
The ultrasonic data 403 and the motion data 404 make it possible to locate the apparatus, and to determine the distance thereof to the obstacles and is the number of obstacles. All these elements define the environment data 412. Said environment data 412 is then integrated into the motion directive.
Eventually, the data originating from the various types of ultrasonic 401 and motion 402 sensors are assimilated by a fuzzy logic system 411 disclosed in
The motion directive process and then analyses the validated 210 or non validated 220 path directive, the control mode 350 as well as the environment data 412. To process such information, it advantageously comprises a computer (preferably a microprocessor and/or a programmable logic circuit (or FGPA)) and an internal memory. In another embodiment of the invention, the computer is outset and the motion directive 500 only is sent to the mobile apparatus to be applied. Such data are processed using a fuzzy logic system so as to determine a motion speed 501 as well as a motion path 502 (
Speed can advantageously be determined by two methods: either using encoders mounted on the wheelchair or using the data originating from the motion sensors 402 of the ultrasonic sensors 401. The mobile apparatus thus moves at a default maximum speed, in a preferred embodiment of the invention. A speed reduction coefficient is then applied according to the received data. The sensor detecting the apparatus speed is preferably an odometer. Other s speed sensors may be added, of course. In another embodiment of the invention, no default speed is used. Speed is then determined by the position of the at least one gaze point or visual sequence 103 in the grid 201.
Taking into account all such data makes it possible to sharpen the motion directive 500. For example, if the user is tired, the mobile apparatus will determine an autonomous control mode 350, and enable the validation of the <<pending >> directions 210. Taking into account the state of fatigue 312 makes it possible to correct such data and makes all motions easier. Similarly, in an obstacle is present, the ultrasonic sensor 401 enables the automatic bypassing, or U-turning of the mobile apparatus. The user will thus not be stuck by the is obstacle.
When reading the above description, it clearly appears that the invention provides a particularly efficient solution to control a mobile apparatus in a precise, reliable and comfortable way, for the user. The mobile apparatus will then be possibly operated during much longer time intervals than in the other known solutions.
The invention is not limited to the embodiments described above but applies to any embodiment complying with the spirit of the claims.
