Patent Application
20250103135
The invention relates to a brain control interface system for controlling a controllable device. The invention further relates to a method of controlling a lighting device and to a computer program product for executing the method.
Brainwave-based device control is a rising new technology. A brain-computer-interface (BCI) is used to detect brain signals of a user, whereupon information from these brain signals is derived. This information may, for example, be indicative of a thought, a mental state (e.g. happy, relaxed, concentrated, neutral) or an action of the user. The thought may, for example, be indicative of a control command for a controllable device, such as a lighting device. An example of such as system is disclosed in U.S. Pat. No. 10,551,921B2. There are two main types of BCIs: non-invasive and invasive BCIs. The non-invasive versions are the most common, and comprise sensors (electrodes) placed on the human head. These measure brain activity and translate that data to a computer. Most BCIs utilize electroencephalography (EEG) systems, which typically feature electrodes are attached to the scalp, which measure the electrical current sent by the neurons inside the brain. Changes in this electrical current reflect brain activity, because when an individual performs an action or thinks about something, hundreds of thousands of neurons are fired. This generates the electrical current, which is large enough to be measured on the scalp. A computer system then tries to make sense of this data to derive the user's action or thought. Alternatives to EEG systems are electrooculography (EOG), electromyography (EMG), electrodermal activity (EDA) and photoplethysmography (PPG) systems. As alternative to utilizing electrodes on the surface of the scalp, implantable brain-computer interfaces may be used. Here, probes are inserted into the brain through an automated process performed by a surgical robot. Each probe comprises an area of wires that contains electrodes capable of locating electrical signals in the brain, and a sensory area where the wire interacts with an electronic system that allows amplification and acquisition of brain signals.
US 2019387998 A1 discloses a system and method may be provided for associating bio-signal data (e.g. EEG brain scan data) from at least one user with at least one music data item (e.g. song, or piece of music). By associating bio-signal data, or emotions determined therefrom, with music, the system may establish a data store of music associated with emotions. That database may then be leveraged upon determining that a user is feeling a particular emotion through an EEG scan. When a particular emotion is detected in EEG data of a user, the system may then respond based at least partly on the same or similar emotion being associated with one or more music data items in the system. For example, the system may recommend a particular song associated with the same emotion presently being experienced by the user.
The inventors have realized that external stimuli may affect the brain activity of a user, and that such an external stimulus may be interpreted by a BCI as a control command for a controllable device. For instance, when the lighting in an area changes, the user may recognize this change and render corresponding brain activity, which may be interpreted by a brain control interface as a control command to change the output of the light based on the control command. This may result in undesired control of the lighting in the area. It is therefore an object of the present invention to reduce the chance of false positives in brain control interfaces.
According to a first aspect, the object is achieved by a brain control interface system for controlling a controllable device located in an environment, the brain control interface system comprising:
The brain control interface may be configured to analyze the brain activity of the user and derive a control command from the brain activity. It may, however, occur that the brain control command is accidentally detected because the user has observed a change of the environmental characteristic (e.g. the environmental light level, the environmental temperature, an environmental sound, an airflow, the environmental humidity etc.). This would result in that the detected change of the environmental characteristic and the detected brain activity of the user would be (substantially) simultaneously. The processor may thus determine if a temporal correlation between a detected change of the environmental characteristic and the detected brain activity of the user is present. If there is a temporal correlation between the detected brain control command (as determined by the brain control interface) and a detected external stimulus (i.e. change of the environmental characteristic), the brain control command is disregarded. If no temporal correlation is present, the controllable device is controlled according to the control command. By determining the presence of the temporal correlation, and controlling/refrain from controlling the controllable device accordingly, the chance of detecting false brain control commands (false positives) is reduced and the accuracy of the brain control interface system is improved.
The processor may be configured to determine if the detected change of the environmental characteristic exceeds a threshold, and to refrain from controlling the controllable device according to the control command only if the environmental characteristic exceeds the threshold. The processor may be further configured to, if the detected change of the environmental characteristic does not exceed the threshold, control the controllable device according to the control command. If the change of the environmental characteristic is not substantive, the control command may be determined as a genuine control command, and the controllable device may be controlled accordingly. This reduces the chance of detecting false negatives caused by at least the features of the brain control interface system, thereby further improving the accuracy of the brain control interface system.
The processor may be configured to determine the presence of the temporal correlation between the detected change of the environmental characteristic and the detected brain activity of the user by determining if the detected change of the environmental characteristic and the detected brain activity of the user occur within a predetermined time window. The predetermined time window may be less than 1 second. Depending on the application of the brain control interface system, the predetermined time window may for example be less than 500 ms, or even less than 200 ms. Optionally, the processor may be l configured to determine the rate of change of the environmental parameter. If the rate of change of the environmental parameter (e.g. light) is above a certain rate threshold (such as when abruptly switching on the light within 100 ms), the controllable device may be controlled according to the control command. If the rate of change of the environmental parameter (e.g. light) is below a certain rate threshold (gently fading in the light within 3 s from off the 100% light output), the controllable device may not be controlled according to the control command.
The processor may be further configured to determine a presence of a second correlation between the type of environmental characteristic and the detected brain activity of the user, and,
The processor may be further configured to: when the processor has refrained from controlling the controllable device according to the control command, request a user operating the brain control interface system to control the controllable device based on the control command, and when the user has approved the request via a user interface, control the controllable device according to the control command, and when the user has disapproved the request via the user interface, again refrain from controlling the controllable device according to the control command. The processor may thus request/prompt the user to confirm or disconfirm that the controllable device is to be controlled or not. The user response may be used to teach the processor when to control or refrain from controlling the controllable device. The user response may, for instance, be used to train the brain control interface system, for instance to determine the above-mentioned threshold for the detected change of the environmental characteristic (which when exceeded results in refraining from controlling the controllable device according to the control command).
The processor may be further configured to control a further device when the temporal correlation is present, wherein the control of the further device is based on the brain activity as a response to the detected change of the environmental characteristic. The further device may be the device that caused the change of the environmental characteristic. The processor may be configured to determine which further device (of a plurality of further devices) has caused the change of the environmental characteristic, for instance based on sensor data from the sensor or based on a further signal received from the further device. For instance, when an environmental light level changes, the processor may determine that a light source in the environment caused this change, and control the light output of the light source based on the brain activity of the user (the manner in which the user responded to the change of the environmental characteristic, e.g. the user wanting to override the automated change of light with a brainwave command).
The sensor may be a light sensor, and the environmental characteristic may be the environmental light level. The sensor may be a temperature sensor, and the environmental characteristic may be the environmental temperature. The sensor may be an audio sensor, and the environmental characteristic may be environmental audio. The sensor may be a humidity sensor, and the environmental characteristic may be the environmental humidity.
The sensor may comprise multiple sensors each configured to detect a change of a respective different environmental characteristic, and the processor may be configured to determine if a plurality of changes of a plurality of environmental characteristics have occurred (substantially) simultaneously, and determine the presence of the temporal correlation between the plurality of changes and the detected brain activity of the user.
The brain control interface and the sensor may be comprised in a brain control interface device. This is beneficial because the brain control interface and the sensor are co-located, and the sensor is located at the location of the user operating (wearing) the brain control interface.
Alternatively, the sensor may be comprised in a separate device, and the processor may be configured to obtain information about the distance between the sensor and the brain control interface (e.g. from a positioning system, based on signal characteristics communicated between the sensor and the brain control interface, etc.), and determine whether to control or refrain from controlling the controllable device further based on the distance between the sensor and the brain control interface. If the distance exceeds a threshold, the processor may determine to control the controllable device according to the control command, even if the temporal correlation is present. If the distance does not exceed the threshold and the temporal correlation is present, the processor may determine to refrain from controlling the controllable device according to the control command.
The processor may be further configured to: obtain first data indicative of the location of the change of the environmental characteristic, obtain second data indicative of the location of the user, determine, based on the location of the change of the environmental characteristic and the location of the user, if the change of the environmental characteristic occurred within a predefined proximity of the user, and, if the change of the environmental characteristic has occurred outside the predefined proximity and if the temporal correlation is present, control the controllable device according to the control command. This is beneficial because if the change of the environmental characteristic has occurred outside the predefined proximity of the user and the temporal correlation is present, the control command is executed, which reduces the chance of false negatives.
Additionally, the second data may be further indicative of an orientation of the user, and the processor may be further configured to: determine, based on the location of the change of the environmental characteristic and the orientation of the user, if the change of the environmental characteristic occurred within a field of view of the user, and, if the change of the environmental characteristic has occurred outside the field of view of the user and if the temporal correlation is present, control the controllable device according to the control command. This is beneficial because by taking the orientation of the user into account the chance of false negatives is further reduced.
The controllable device may be a lighting device. Alternatively, the controllable device may, for example, be a connected (home) appliance or connected (office) equipment.
According to a second aspect, the object is achieved by a method of controlling a controllable device located in an environment, the method comprising:
According to a third aspect, the object is achieved by a computer program product for a computing device, the computer program product comprising computer program code to perform the method when the computer program product is run on a processing unit of the computing device, the computing device comprising a brain control interface, a sensor and a processor.
It should be understood that the computer program product and the method may have similar and/or identical embodiments and advantages as the above-mentioned system.
The above, as well as additional objects, features and advantages of the disclosed systems, devices and methods will be better understood through the following illustrative and non-limiting detailed description of embodiments of devices and methods, with reference to the appended drawings, in which:
All the figures are schematic, not necessarily to scale, and generally only show parts which are necessary in order to elucidate the invention, wherein other parts may be omitted or merely suggested.
The processor 106 is configured to control the controllable device 130. The processor 106 may be configured to control the controllable device 130 according to the control command by communicating control signals to the controllable device 130 (e.g. via Zigbee, BLE, Ethernet, etc.). The processor 106 may be comprised in the controllable device 130, and control the controllable device 130 directly. Alternatively, the processor 106 may be comprised in a remote device 170, or for example in the BCI 120, and the processor 106 may control the controllable device via a communication unit 104 configured to communicate the control signals to the controllable device 130.
The processor 106 may be comprised in a single device or distributed across multiple devices, which may depend on the system architecture of the BCI system 100. For instance, in the example of
The system 100 comprises a brain control interface configured to detect brain activity of a user indicative of a control command for controlling the controllable device 130, and to derive the control command from the brain activity. The controllable device 130 may be a device configured to adjust an environmental characteristic that corresponds to the environmental characteristic detected by the sensor. The controllable device 130 may, for example, be a connected (home) appliance or connected (office) equipment. The controllable device 130 may comprise a receiver configured to receive control signals indicative of the control command, for instance via a wireless network. The controllable device 130 may, for example, be a connected speaker, a lighting device comprising one or more LED light sources, a thermostat, a tv, a (tablet) pc, a smartphone, a game console, etc.
The sensor 102 is configured to detect changes of an environmental characteristic in the environment. The sensor 102 may for example be a light sensor, and the environmental characteristic may be the environmental light level. The sensor 102 may for example be a temperature sensor, and the environmental characteristic may be the environmental temperature. The sensor 102 may for example be an audio sensor, and the environmental characteristic may be environmental audio. The sensor 102 may for example be a humidity sensor, and the environmental characteristic may be the environmental humidity. The sensor 102 may comprise multiple sensors each configured to detect a change of a respective different environmental characteristic.
The processor 106 is configured to determine if there is a temporal correlation between a detected change of the environmental characteristic (as detected by the sensor 102) and the detected brain activity of the user 160. The processor 106 may be configured to determine the presence of the temporal correlation between the detected change of the environmental characteristic and the detected brain activity of the user by determining if the detected change of the environmental characteristic and the detected brain activity of the user occur at substantially the same time or at least within a predetermined time window. The predetermined time window may be less than 1 second. Depending on the application of the brain control interface system, the predetermined time window may for example be less than 500 ms, or even less than 200 ms. For instance, the sensor 102 may be a light sensor and the controllable device 130 may be a lighting device. The lighting device may be configured to receive control signals from the processor 106 and comprise driver configured to adjust the light output of one or more (LED) light sources accordingly. The sensor 102 may detect a change in light level (e.g. due to switching on of a device such display, due to the sun appearing from behind the clouds, due to blinds opening, etc.) or a change in color (e.g. due to a display changing from red to blue, due to an automated change in color temperature of the ceiling lights). If the brain control interface 120 detects brain activity indicative of a control command for the controllable device 130 (e.g. a control command to switch the light on) at substantially the same time, the processor 106 may determine that there is a temporal correlation between the detected change of the environmental characteristic (the light level) and the detected brain activity of the user 160. If the temporal correlation is present, the processor 106 refrains from controlling the controllable device 130 according to the control command. If the temporal correlation is not present, the processor 106 controls the controllable device 130 according to the control command (e.g. the control command to switch the lighting device on).
The processor 106 may be configured to determine if the detected change of the environmental characteristic exceeds a threshold, and to refrain from controlling the controllable device 130 according to the control command only if the environmental characteristic exceeds the threshold. The processor 106 may be further configured to, if the detected change of the environmental characteristic does not exceed the threshold, control the controllable device 130 according to the control command.
The processor 106 may be further configured to determine a presence of a second correlation between the type of environmental characteristic and the (type of) detected brain activity of the user. The processor 106 may thus determine if the type of the environmental characteristic detected by the sensor 102 corresponds to an environmental characteristic that is to be changed by the brain activity. The controllable device 130 may be a device configured to adjust an environmental characteristic that corresponds to the environmental characteristic detected by the sensor 102. The processor 106 may be further configured to control the controllable device according to the control command if the temporal correlation is not present and the second correlation is present. The processor 106 may be further configured to control the controllable device according to the control command if the second correlation is not present and the temporal correlation is present. The processor 106 may be further configured to control the controllable device according to the control command if the temporal correlation and the second correlation are not present. The processor 106 may be further configured to refrain from controlling the controllable device according to the control command if the temporal correlation and the second correlation are present. For instance, if the environmental characteristic is of a first type (e.g. a light level in the environment) and the brain activity (and therewith the control command) is of a correlated type (e.g. a lighting control command), the processor 106 may determine that the second correlation is present. For instance, if the environmental characteristic is of a first type (e.g. a light level in the environment) and the brain activity (and therewith the control command) is of a non-correlated type (e.g. an audio control command), the processor 106 may determine that the second correlation is not present. For instance, if the environmental characteristic is of a first type (e.g. a temperature in the environment) and the brain activity (and therewith the control command) is of a correlated type (e.g. a color temperature of the light control command), the processor 106 may determine that the second correlation is present. The processor 106 may be configured to access a (local or remote) memory configured to store correlations between types of environmental characteristics and respective (types of) brain activities (and therewith (types of) control commands) of the user, and to determine the presence of the second correlation between the type of environmental characteristic and the (type of) detected brain activity of the user based on the stored correlations.
The processor 106 may be further configured to request the user 160 operating the brain control interface system 100 to control the controllable device 130 based on the control command when the processor 106 has refrained from controlling the controllable device according to the control command. The user 160 may then approve or disapprove the request via a user interface (e.g. a voice assistant, a touch screen, one or more buttons on a switch, etc.). The processor 106 may be communicatively coupled (e.g. wirelessly, directly) to the user interface. When the user 160 has approved the request via the user interface, the processor 106 may control the controllable device 130 according to the control command, and when the user has disapproved the request via the user interface, the processor 106 may again refrain from controlling the controllable device 130 according to the control command.
The processor 106 may be further configured to control a further device when the temporal correlation is present, wherein the control of the further device is based on the brain activity as a response to the detected change of the environmental characteristic. The processor 106 may be configured to communicate with the further device in a similar manner as with the controllable device 130. The further device may be the device that caused the change of the environmental characteristic. The processor 106 may be configured to determine which further device (of a plurality of further devices) has caused the change of the environmental characteristic, for instance based on sensor data from the sensor or based on a signal received from the further device. For instance, when an environmental light level changes, the processor 106 may determine that a light source in the environment caused this change, and control the light output of the light source based on the brain activity of the user (the manner in which the user responded to the change of the environmental characteristic).
The processor 106 may be further configured to determine whether to control the controllable device 130 further based on the location of the change of the environmental characteristic. The processor 106 may be configured to obtain first data indicative of the location of the change of the environmental characteristic. The processor 106 may, for example, obtain information about the location and/or orientation of the sensor 102 relative to the environment 150, and determine the location of the change of the environmental characteristic based thereon. This has been illustrated in
The processor 106 may be further configured to obtain second data indicative of the location of the user 160 relative to the environment 150 or relative to the sensor 102. The second data may be obtained from an (indoor) positioning system. Examples of such a positioning system include a radio frequency (RF) beacon system, a coded light positioning system, etc. Alternatively, the location of the user 160 may have been defined by the user 160 via a user interface. It should be understood that techniques for determining a location of a user relative to an environment 150 are known in the art and will therefore not be discussed in detail. The processor 106 may be further configured to determine, based on the location of the change of the environmental characteristic and the location of the user, if the change of the environmental characteristic occurred within a predefined proximity of the user. If the change of the environmental characteristic has occurred outside the predefined proximity and if the temporal correlation is present, control the controllable device according to the control command. The predefined proximity may, for example, be defined as a distance, or be defined as an area (e.g. a (part of) a room in the environment 150).
The second data may be further indicative of an orientation of the user 160. The orientation may be obtained from an orientation sensor (e.g. from a magnetometer comprised in a user-worn or held device, from a remote camera, etc.) or the orientation of the user 160 may have been defined by the user via a user interface. The processor 106 may be further configured to determine whether to control the controllable device 130 further based on the orientation of the user relative to the change of the environmental characteristic. The processor 106 may be further configured to determine, based on the location of the change of the environmental characteristic and the orientation of the user, if the change of the environmental characteristic occurred within a field of view (FoV) of the user 160. If the change of the environmental characteristic has occurred outside the field of view of the user 160 and if the temporal correlation is present, control the controllable device 130 according to the control command.
The method 400 may be executed by computer program code of a computer program product when the computer program product is run on a computing system, such as the system 100.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims.
In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. Use of the verb “comprise” and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. The article “a” or “an” preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer or processing unit. In the device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.
Aspects of the invention may be implemented in a computer program product, which may be a collection of computer program instructions stored on a computer readable storage device which may be executed by a computer. The instructions of the present invention may be in any interpretable or executable code mechanism, including but not limited to scripts, interpretable programs, dynamic link libraries (DLLs) or Java classes. The instructions can be provided as complete executable programs, partial executable programs, as modifications to existing programs (e.g. updates) or extensions for existing programs (e.g. plugins). Moreover, parts of the processing of the present invention may be distributed over multiple computers or processors or even the ‘cloud’.
Storage media suitable for storing computer program instructions include all forms of nonvolatile memory, including but not limited to EPROM, EEPROM and flash memory devices, magnetic disks such as the internal and external hard disk drives, removable disks and CD-ROM disks. The computer program product may be distributed on such a storage medium, or may be offered for download through HTTP, FTP, email or through a server connected to a network such as the Internet.
| Number | Date | Country | Kind |
|---|---|---|---|
| 22153808.5 | Jan 2022 | EP | regional |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2023/050775 | 1/13/2023 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 63300765 | Jan 2022 | US |
