Information processing system and non-transitory computer readable medium

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on and claims priority under 35 USC 119 from Japanese Patent Application No. 2019-219153 filed Dec. 3, 2019.

BACKGROUND
(i) Technical Field

The present disclosure relates to an information processing system and a non-transitory computer readable medium.

(ii) Related Art

It is desired to use brain waves as a user interface of a next generation (e.g., refer to U.S. Patent Application Publication No. 2013/0096453).

SUMMARY

In order to use brain waves as a user interface, a user needs to be able to generate certain types of brain waves intentionally, but it requires special training to achieve this with a certain level of reproducibility. Although measurement of brain waves at positions other than measurement points specified in a medical field has been studied, the accuracy of association with a user's intention is low. It is therefore expected that, at an initial stage of development of such a user interface, a mechanism for improving the accuracy of operation by complementing measured brain waves will be required.

Aspects of non-limiting embodiments of the present disclosure relate to improve the accuracy of operation of a device compared to when only brain waves are used.

Aspects of certain non-limiting embodiments of the present disclosure overcome the above disadvantages and/or other disadvantages not described above. However, aspects of the non-limiting embodiments are not required to overcome the disadvantages described above, and aspects of the non-limiting embodiments of the present disclosure may not overcome any of the disadvantages described above.

According to an aspect of the present disclosure, there is provided an information processing system including a processor configured to determine an operation based on a combination of first features of first biological information detected from a user and second features of second, different biological information detected from the user, the user being monitored for both the first biological information and the second biological information simultaneously, the first biological information being brain waves, and instruct a device to perform the operation.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments of the present disclosure will be described in detail based on the following figures, wherein:

FIG. 1 is a diagram illustrating a schematic configuration of a brain wave operation system used in a first exemplary embodiment;

FIG. 2 is a diagram illustrating an example of the appearance of earphones used in the first exemplary embodiment;

FIG. 3 is a diagram illustrating an example of the internal configuration of the earphones used in the first exemplary embodiment;

FIG. 4 is a diagram illustrating an example of the internal configuration of an information terminal used in the first exemplary embodiment;

FIG. 5 is a diagram illustrating an example of an intentional movement of muscles by a user while the user is desiring a certain operation in his/her mind;

FIGS. 6A and 6B are diagrams illustrating an example of correspondence tables used in the first exemplary embodiment: FIG. 6A illustrates a correspondence table for features of brain wave information and FIG. 6B illustrates a correspondence table for features of biological information other than brain waves;

FIG. 7 is a flowchart illustrating an example of a process performed by the information terminal after the information terminal receives a digital signal including brain wave information;

FIG. 8 is a flowchart illustrating another example of the process performed by the information terminal after the information terminal receives a digital signal including brain wave information;

FIG. 9 is a diagram illustrating a measurement point of a headset equipped with a brain wave sensor capable of measuring brain waves with the earphones worn by the user;

FIG. 10 is a diagram illustrating measurement points for brain waves described in a thesis;

FIG. 11 is a diagram illustrating evaluation of output α waves;

FIGS. 12A and 12B are diagrams illustrating results of measurement performed with MindWave: FIG. 12A illustrates a result of measurement at a time when an open eye state and a closed eye state are alternated twice without blinking and FIG. 12B illustrates a result of measurement at a time when the open eye state and the closed eye state are alternated twice with blinking;

FIGS. 13A and 13B are diagrams illustrating results of measurement performed with the earphones used in the first exemplary embodiment: FIG. 13A illustrates a result of measurement at a time when the open eye state and the closed eye state are alternated twice without blinking and FIG. 13B illustrates a result of measurement at a time when the open eye state and the closed eye state are alternated with blinking and movement of the jaw;

FIGS. 14A to 14C are diagrams illustrating other results of the measurement performed with MindWave: FIG. 14A illustrates changes in the percentage of spectral intensity in each frequency band at a time when subjects have entered the closed eye state from the open eye state with blinking, FIG. 14B illustrates changes in the percentage of spectral intensity in each frequency band at a time when the subjects have entered the closed eye state from the open eye state without blinking, and FIG. 14C illustrates a case where a waves do not increase;

FIGS. 15A to 15C are diagrams illustrating other results of the measurement performed with the earphones used in the first exemplary embodiment: FIG. 15A illustrates changes in the percentage of spectral intensity in each frequency band at a time when the subjects have entered the closed eye state from the open eye state with blinking, FIG. 15B illustrates changes in the percentage of spectral intensity in each frequency band at a time when the subjects have entered the closed eye state from the open eye state without blinking, and FIG. 15C illustrates a case where α waves do not increase;

FIGS. 16A and 16B are diagrams illustrating an example of presentation of parts in which spectral intensity has increased: FIG. 16A illustrates a result of the measurement performed with MindWave and FIG. 16B illustrates a result of the measurement performed with the earphones used in the first exemplary embodiment;

FIG. 17 is a diagram illustrating a schematic configuration of a brain wave operation system used in a second exemplary embodiment;

FIG. 18 is a diagram illustrating an example of the internal configuration of an information terminal used in the second exemplary embodiment;

FIG. 19 is a diagram illustrating a schematic configuration of a brain wave operation system used in a third exemplary embodiment;

FIG. 20 is a diagram illustrating an example of the internal configuration of an information terminal used in the third exemplary embodiment;

FIG. 21 is a diagram illustrating a schematic configuration of a brain wave operation system used in a fourth exemplary embodiment;

FIG. 22 is a diagram illustrating an example of the internal configuration of an information terminal used in the fourth exemplary embodiment;

FIGS. 23A and 23B are diagrams illustrating an example of tables storing a history of operations and information indicating whether the operations are correct: FIG. 23A illustrates a table storing a history of operations estimated from features of brain wave information and information indicating whether operations are correct and FIG. 23B illustrates a table storing a history of operations estimated from features of biological information other than brain waves and information indicating whether the operations are correct;

FIG. 24 is a flowchart illustrating an example of a process performed by the information terminal after the information terminal receives a digital signal including brain wave information;

FIG. 25 is a diagram illustrating a schematic configuration of a brain wave operation system used in a fifth exemplary embodiment;

FIG. 26 is a diagram illustrating an example of the internal configuration of an information terminal used in the fifth exemplary embodiment;

FIG. 27 is a diagram illustrating an example of a table storing a history of operations based on brain waves and information indicating whether the operations are correct;

FIG. 28 is a flowchart illustrating an example of a process performed by the information terminal after the information terminal receives a digital signal including brain wave information;

FIG. 29 is a diagram illustrating an example of the appearance of an earphone inserted into one of the ears;

FIG. 30 is a diagram illustrating an example of an earring for which electrodes for measuring brain waves are provided;

FIG. 31 is a diagram illustrating an example of spectacles for which the electrodes for measuring brain waves are provided;

FIG. 32 is a diagram illustrating an example in which the electrodes for measuring brain waves are provided for a headset having a function of displaying an image assimilating to a surrounding environment of the user;

FIGS. 33A to 33D are diagrams illustrating an example in which a function of tracking a line of sight is executed to use the user's line of sight as a feature of biological information: FIG. 33A illustrates a case where the user's line of sight is leftward, FIG. 33B illustrates a case where the user's line of sight is rightward, FIG. 33C illustrates a case where the user's line of sight is upward, and FIG. 33D illustrates a case where the user's line of sight is downward;

FIG. 34 is a diagram illustrating an example of a device that measures changes in the amount of blood flow caused by brain activity using near-infrared light;

FIG. 35 is a diagram illustrating an example of a magnetoencephalography (MEG) machine; and

FIG. 36 is a diagram illustrating an example of a brain wave operation system that uses the user's facial expressions as features of biological information other than brain waves.

DETAILED DESCRIPTION

Exemplary embodiments of the present disclosure will be described hereinafter with reference to the drawings.

First Exemplary Embodiment

System Configuration

FIG. 1 is a diagram illustrating a schematic configuration of a brain wave operation system 1 used in a first exemplary embodiment.

The brain wave operation system 1 illustrated in FIG. 1 includes earphones 10 inserted into the external auditory meatuses, an information terminal 20 wirelessly connected to the earphones 10, and a device (hereinafter referred to as an “operation target device”) 30 to be operated by a user.

The earphones 10 and the information terminal 20 according to the present exemplary embodiment constitute an example of an information processing system.

The earphones 10 according to the present exemplary embodiment include a circuit that plays back sound received from the information terminal 20 and a circuit that measures electrical signals (hereinafter referred to as “brain waves”) caused by brain activity.

The earphones 10 used in the present exemplary embodiment are a wireless device. The earphones 10, therefore, are connected to the information terminal 20 through wireless communication.

In the present exemplary embodiment, Bluetooth (registered trademark) is used for the wireless connection between the earphones 10 and the information terminal 20. Alternatively, another communication standard such as Wi-Fi (registered trademark) may be used for the wireless connection, although the earphones 10 and the information terminal 20 may be connected to each other by cable.

The information terminal 20 estimates an operation desired by the user by processing a digital signal received from the earphones 10 and transmits a signal (hereinafter referred to as an “operation signal”) indicating the estimated operation to the operation target device 30. In the example illustrated in FIG. 1, a smartphone is used as the information terminal 20. Alternatively, the information terminal 20 may be a tablet terminal, a laptop computer, a wearable computer, or the like.

In FIG. 1, an operation signal corresponding to “turn on” desired by the user in his/her mind is transmitted to the operation target device 30. Infrared radiation or another existing communication scheme such as a low-power wide-area (LPWA) network may be used for the transmission of the operation signal to the operation target device 30. The LPWA network is an example of a communication scheme used for Internet of Things (IoT) communication.

The operation target device 30 according to the present exemplary embodiment may be any device having a function of receiving signals from the outside and controlling operations inside thereof. The operation target device 30 is, for example, a device including an infrared reception unit or a device that achieves IoT communication.

In FIG. 1, the operation target device 30 receives the operation signal corresponding to “turn on” desired by the user in his/her mind from the information terminal 20.

First, a reason why the earphones 10 are used to measure brain waves will be described.

When interfaces employing brain waves will begin to spread in the future, users might not like wearing devices visibly designed to measure brain waves. Helmet-shaped devices, for example, might not gain popularity in terms of both design and a burden on the body.

For this reason, the earphones 10 are used in the present exemplary embodiment as a device for measuring brain waves. Since the earphones 10 are already widely used as an audio device, users will not be reluctant to wear the earphones 10.

In addition, since the external auditory meatuses, into which the earphones 10 are inserted, are close to the brain, brain waves can be easily measured. How the earphones 10 can measure brain waves will be described later in a “Results of Experiments” section. The external auditory meatuses are examples of the ears. In the present exemplary embodiment, the ears include the auricles and the external auditory meatuses.

In addition, because the earphones 10 each include a speaker, which is not illustrated, information can be easily transmitted to the user.

In addition, because the ears into which the earphones 10 are inserted are close to the user's mouth, voices uttered by the user can be easily detected.

Configuration of Earphones 10

FIG. 2 is a diagram illustrating an example of the appearance of the earphones 10 used in the first exemplary embodiment.

The earphones 10 include earphone chips 11R and 11L inserted into the external auditory meatuses, earphone bodies 12R and 12L on which the earphone chips 11R and 11L are mounted, respectively, ear hooks 13R and 13L attached between the auricles and the temples, a cable 14 connecting the earphone bodies 12R and 12L to each other, and a controller 15 on which a power button and volume buttons are provided.

“R” in FIG. 2 denotes that a corresponding part is located on a side of the user's right ear, and “L” denotes a corresponding part is located on a side of the user's left ear.

The earphone chip 11R according to the present exemplary embodiment is inserted in the external auditory meatus and includes a dome-shaped electrode 11R1 that comes into contact with an inner wall of the external auditory meatus and a ring-shaped electrode 11R2 that comes into contact with the concha cavity.

The electrodes 11R1 and 11R2 according to the present exemplary embodiment are composed of conductive rubber in order to measure electrical signals observed on the skin. The electrodes 11R1 and 11R2 are electrically insulated from each other by an insulator.

In the present exemplary embodiment, the electrode 11R1 is a terminal used to measure potential variations including brain waves (electroencephalogram (EEG)) and biological information other than the brain waves (hereinafter referred to as an “EEG measuring terminal”). In the present exemplary embodiment, the biological information other than brain waves is components of electrical signals caused by movement of muscles relating to facial expression, such as ones relating to the eyes, nose, mouth, and eyebrows (hereinafter referred to as “mimic muscles”), muscles for masticatory movement (hereinafter referred to as “masseter muscles”), and muscles for swallowing (hereinafter referred to as “hyoid muscles”), and the like.

The electrode 11R2 is a ground electrode (hereinafter referred to as a “GND terminal”).

The earphone chip 11L, on the other hand, includes a dome-shaped electrode 11L1 that is inserted into the external auditory meatus and that comes into contact with an inner wall of the external auditory meatus. In the present exemplary embodiment, the electrode 11L1 is a terminal used to measure a reference (REF) potential (hereinafter referred to as a “REF terminal”). In the present exemplary embodiment, the electrode 11R2 and the electrode 11L1 are electrically short-circuited to each other.

As described later, in the present exemplary embodiment, potential variations including brain waves and biological information other than brain waves are measured as differential signals between electrical signals measured by the electrodes 11R1 and 11L1.

Potential variations including brain waves and biological information other than brain waves will be generically referred to as “biological information such as brain waves” hereinafter.

In the field of brain science, all potential variations derived from biological information other than brain waves are called “artifacts”. It is considered that electrical signals obtained by measuring brain waves invariably include artifacts.

Components included in artifacts are classified into those derived from a living body, those derived from a measurement system such as electrodes, and those derived from external devices and environment. Among these three types of component, the components other than those derived from a living body can be measured as noise detected by the earphones 10. The noise can be measured as electrical signals at a time when the electrodes 11R1 and 11L1 are electrically short-circuited to each other.

The earphone body 12R according to the present exemplary embodiment includes a circuit that generates digital signals corresponding to biological signals such as brain waves, a circuit that generates audio data from electrical signals output from a microphone, which is not illustrated, and a circuit that performs a process for decoding audio data received from the information terminal (refer to FIG. 1) and outputting the decoded audio data to the speaker, which is not illustrated.

The earphone body 12L, on the other hand, includes a battery.

FIG. 3 is a diagram illustrating an example of the internal configuration of the earphones 10 used in the first exemplary embodiment.

FIG. 3 illustrates the internal configuration of the earphone bodies 12R and 12L of the earphones 10.

In the present exemplary embodiment, the earphone body 12R includes a digital electroencephalograph (EEG) 121, a microphone 122, a speaker 123, a six-axis sensor 124, a Bluetooth module 125, a semiconductor memory 126, and a microprocessor unit (MPU) 127.

The digital EEG 121 includes a differential amplifier that differentially amplifies potential variations detected by the electrodes 11R1 and 11L1, a sampling circuit that samples an output of the differential amplifier, and an analog-to-digital (A/D) conversion circuit that converts an analog potential after the sampling into a digital value. In the present exemplary embodiment, a sampling rate is 600 Hz. The resolution of the A/D conversion circuit is 16 bits.

The microphone 122 includes a diaphragm that vibrates in accordance with sounds uttered by the user, a voice coil that converts the vibration of the diaphragm into electrical signals, and an amplifier that amplifies the electrical signals. An A/D conversion circuit that converts the analog potential of the electrical signals output from the amplifier into digital values is separately prepared.

The speaker 123 includes a diaphragm and a voice coil that vibrates the diaphragm using a current according to audio data. Audio data input from the MPU 127 is converted by a digital-to-analog (D/A) conversion circuit into an analog signal.

The six-axis sensor 124 includes a three-axis acceleration sensor and a three-axis gyro sensor. The six-axis sensor 124 detects an attitude of the user.

The Bluetooth module 125 communicates data with the information terminal 20 (refer to FIG. 1). In the present exemplary embodiment, the Bluetooth module 125 transmits digital signals output from the digital EEG 121 and audio data obtained by the microphone 122 and receives audio data from the information terminal 20.

The semiconductor memory 126 includes, for example, a read-only memory (ROM) storing basic input-output system (BIOS), a random-access memory (RAM) used as a working area, and a rewritable nonvolatile memory (hereinafter referred to as a “flash memory”).

In the present exemplary embodiment, the flash memory stores digital signals, which are outputs of the digital EEG 121, audio data obtained by the microphone 122, audio data received from the information terminal 20, and the like.

The MPU 127 controls communication of digital signals with the information terminal 20, processes digital signal to be transmitted to the information terminal 20, and processes digital signals received from the information terminal 20. In the present exemplary embodiment, the MPU 127 performs a process, such as a Fourier transform, on digital signals output from the digital EEG 121. The MPU 127 and the semiconductor memory 126 operate as a computer.

The earphone body 12L, on the other hand, includes a lithium battery 128.

Configuration of Information Terminal 20

FIG. 4 is a diagram illustrating an example of the internal configuration of the information terminal 20 used in the first exemplary embodiment.

FIG. 4 illustrates only devices of the information terminal 20 relating to generation of an operation signal for operating the operation target device 30 (refer to FIG. 1) on the basis of biological information such as brain waves.

The information terminal 20 illustrated in FIG. 4 includes a Bluetooth module 201, an MPU 202, a semiconductor memory 203, and a wireless IoT module 204.

The Bluetooth module 201 is used to communicate with the Bluetooth module 125 provided for the earphones 10.

The MPU 202 executes a function of obtaining, from a digital signal received from the earphones 10 (refer to FIG. 1), information regarding brain waves (hereinafter referred to as “brain wave information”) and biological information other than the brain waves and estimating an operation to be performed on the operation target device 30 (refer to FIG. 1). The function is achieved by executing an application program.

The MPU 202 illustrated in FIG. 4 functions as a brain wave information obtaining section 221 that obtains features of brain wave information from a digital signal received from the earphones 10, a biological information obtaining section 222 that obtains features of biological information other than brain waves from a digital signal received from the earphones 10, and an operation estimation section 223 that estimates an operation to be performed on the operation target device 30 in accordance with a combination of features of brain wave information and features of biological information other than brain waves.

The brain wave information obtaining section 221 separates a waveform component unique to brain waves observed in a digital signal received from the earphones 10 and obtains features of brain wave information included in the separated wave form component. An independent component analysis (ICA) or another known technique is used to obtain features of brain wave information. The features of brain wave information include, for example, a waveform component unique to brain waves observed in a digital signal, the spectral intensity and distribution of each frequency component included in the waveform component, the spectral intensity of a certain frequency component included in the waveform component, and the percentage of increase in α waves.

The biological information obtaining section 222 obtains, as features of biological information other than brain waves, potential variations caused by an intentional movement of head muscles by the user while the user is desiring a certain operation in his/her mind. The features of biological information other than brain waves include, for example, a waveform component unique to biological information other than brain waves observed in a digital signal, the spectral intensity and distribution of each frequency component included in the waveform component, and the spectral intensity of a certain frequency component included in the waveform component.

A reason why potential variations caused by an intentional movement of head muscles by the user while the user is desiring a certain operation in his/her mind is as follows.

As described above, biological information other than brain waves invariably mix with an electrical signal caused by brain waves. Biological information other than brain waves, therefore, has been considered as information that prevents measurement of brain wave information.

In addition, although it is easy to desire a certain operation in one's mind, it is not necessarily easy to intentionally generate brain waves corresponding to a certain operation. It is needless to say that some practice is required to output such brain waves with a high level of reproducibility. Furthermore, it has been pointed out that it is difficult for different users to reproduce similar brain waves.

It is much easier, however, for a user to intentionally move certain muscles than to output certain brain waves. That is, it is not difficult for a user to cause certain features of biological information.

In the present exemplary embodiment, therefore, an electrical signal caused by movement of head muscles, which can be intentionally moved by the user, is obtained as features of biological information, and features of brain wave information are complemented by the obtained features.

FIG. 5 is a diagram illustrating an example of the intentional movement of muscles by the user while the user is desiring a certain operation in his/her mind. FIG. 5 illustrates an example of movement of the jaw accompanied by swallowing of saliva. As described above, although it needs some practice and experience to output certain brain waves, most users can intentionally swallow saliva. In addition, as described later, an electrical signal caused by movement of muscles has a larger amplitude than a waveform of brain waves and can be easily distinguished from brain wave information. Furthermore, an electrical signal caused by movement of muscles is intermittent and does not prevent measurement of brain wave information while the muscles are not moving.

For this reason, features of biological information other than brain waves can be obtained while obtaining features of brain wave information, which are observed while the user is desiring a certain operation in his/her mind.

FIG. 4 will be referred to again.

The operation estimation section 223 according to the present exemplary embodiment estimates an operation corresponding to each set of features by referring to correspondence tables prepared for features of brain wave information and features of biological information other than brain waves. The operation estimation section 223 then outputs the estimated operation on the basis of predetermined rules to the wireless IoT module 204 as an operation signal.

There are some methods for identifying an operation on the basis of the predetermined rules.

In one method, for example, if an operation can be estimated from features of brain wave information, the estimated operation takes priority over an operation estimated from features of biological information other than brain waves.

There is another method in which the user is asked whether an operation estimated from features of brain wave information is correct before an operation signal is output to the operation target device 30 (refer to FIG. 1).

The semiconductor 203 according to the present exemplary embodiment stores correspondence tables 231 and 232 used by the operation estimation section 223 to determine an operation.

FIGS. 6A and 6B are diagrams illustrating an example of the correspondence tables 231 and 232 used in the first exemplary embodiment. FIG. 6A illustrates the correspondence table 231 for features of brain wave information, and FIG. 6B illustrates the correspondence table 232 for features of biological information other than brain waves.

The correspondence table 231 includes management numbers, features of brain wave information, and corresponding operations.

In FIG. 6A, “turn on” is associated with features AA, and “turn off” is associated with features AB. The operations also include information for identifying an operation target device 30.

The correspondence table 232 includes management numbers, features of biological information other than brain waves, and corresponding operations.

In FIG. 6B, “turn on” is associated with features #102, and “turn off” is associated with features #103. For example, features observed when the user swallows saliva once are associated with “turn on”, and features observed when the user swallows saliva twice are associated with “turn off”. The operations also include information for identifying an operation target device 30.

Although correspondences stored in the correspondence table 232 are determined in advance in the present exemplary embodiment, the user may register new correspondences.

The semiconductor 203 also includes a ROM storing BIOS, a RAM used as a working area, and a flash memory, as well as the correspondence tables 231 and 232.

FIG. 4 will be referred to again.

In the present exemplary embodiment, the wireless IoT module 204 transmits an operation signal on the basis of a communication standard such as LPWA.

Processes Performed by Information Terminal 20

An example of processes performed by the information terminal 20 (refer to FIG. 1) by executing programs using the MPU 202 (refer to FIG. 4) will be described hereinafter.

First Process

In the first process, the user is not asked whether an operation is correct before the operation is performed on the basis of features of brain wave information.

FIG. 7 is a flowchart illustrating an example of a process performed by the information terminal 20 after the information terminal 20 receives a digital signal including brain wave information. “S” in FIG. 7 denotes a step.

In the present exemplary embodiment, digital information including brain wave information is transmitted to the information terminal 20 from the earphones 10 (refer to FIG. 1). The user moves his/her jaw in a certain way corresponding to a certain operation to be performed on the operation target device 30 (refer to FIG. 1) while desiring the operation in his/her mind.

Upon receiving the digital signal including the brain wave information from the earphones 10 (refer to FIG. 1), the MPU 202 obtains features of the brain wave information and features of biological information other than brain waves from the received digital signal (S1).

Next, the MPU 202 determines whether the correspondence table 231 (refer to FIG. 6A) includes the features of the brain wave information (S2).

If a result of S2 is positive, the MPU 202 transmits an operation corresponding to the features of the brain wave information (S3).

If the result of S2 is negative, on the other hand, the MPU 202 determines whether the correspondence table 232 (refer to FIG. 6B) includes the features of the biological information other than brain waves.

If a result of S4 is positive, the MPU 202 transmits an operation corresponding to the features of the biological information (S5).

If the result of S4 is negative, on the other hand, the MPU 202 ends the process. In this case, the user notices that the operation target device 30 is not performing any operation and his/her operation has failed.

Alternatively, if the result of S4 is negative, the user may be notified that no operation has been identified. For example, a message may be displayed on a display screen of the information terminal 20 (refer to FIG. 1) or output from the speaker 123 (refer to FIG. 3) provided for the earphones 10.

In the first process, the user can operate operation target device 30 on the basis of features caused by movement of muscles in his/her jaw, which can be intentionally moved by the user, before the user gets accustomed to performing operations through brain waves.

After getting accustomed to performing operations through brain waves and becoming able to intentionally reproduce certain brain waves, the user can operate the operation target device 30 without intentionally moving the muscles in his/her jaw or the like.

With the brain wave operation system 1 according to the present exemplary embodiment, operations based on brain waves can be performed while making the user feel less reluctant to wear a device for operating the operation target device 30 through brain waves. In addition, before the user gets accustomed to performing operations through brain waves, the user can complement operations based on brain waves by intentionally moving the muscles in his/her jaw or the like. After getting accustomed to performing operations through brain waves, the user can operate the operation target device 30 as intended by desiring certain operations in his/her mind without intentionally moving the muscles in his/her jaw or the like.

Second Process

It is assumed in the first process that, if features of brain wave information obtained by the brain wave information obtaining section 221 (refer to FIG. 4) are found in the correspondence table 231 (refer to FIG. 6), the found features and an operation desired in the user's mind match.

In the case of a user who is not accustomed to performing operations through brain waves, however, an operation desired in the user's mind and features of obtained brain waves might not match. The operation target device 30, therefore, can perform an unintended operation.

In a second process, therefore, a step of making the user check an operation estimated from features of brain wave information is added before the operation is transmitted to the operation target device 30.

FIG. 8 is a flowchart illustrating another example of the process performed by the information terminal 20 (refer to FIG. 1) after the information terminal 20 receives a digital signal including brain wave information. In FIG. 8, the same steps as in FIG. 7 are given the same reference numerals.

In this process, too, the MPU 202 that has received a digital signal including brain wave information from the earphones 10 obtains features of the brain wave information and features of biological information other than brain waves from the received digital signal (S1).

Next, the MPU 202 determines whether the correspondence table 231 includes the features of the brain wave information (S2).

If the result of S2 is positive, the MPU 202 asks the user whether an operation corresponding to the features of the brain wave information is correct (S11). Here, a message may be displayed on the display screen, which is not illustrated, of the information terminal 20 or output from the speaker 123 (refer to FIG. 3) provided for the earphones 10. The message may be, for example, “Turn on?”

After S11, the MPU 202 determines whether the user's response indicates that the operation is correct (S12). In the present exemplary embodiment, the user moves the muscles in his/her jaw or the like to respond. The user swallows saliva once to indicate “correct” and twice to indicate “incorrect”. The number of times of swallowing is counted in a predetermined period of time.

The user may respond, however, through sound using a smart speaker, instead. In this case, the audio response from the user is obtained by the microphone 122 (refer to FIG. 3) provided for the earphones 10 and transmitted to the information terminal 20 as audio data. The MPU 202 analyzes the received audio response and determines whether to use the operation corresponding to the features of the brain wave information.

If a result of S12 is positive, the MPU 202 transmits the operation corresponding to the features of the brain wave information (S3).

If the result of S12 is negative, or if the result of S2 is negative, the MPU 202 determines whether the correspondence table 232 includes the features of the biological information other than brain waves (S4). The features of the biological information have been obtained in S1.

If the result of S4 is positive, the MPU 202 transmits the operation corresponding to the features of the biological information (S5).

If the result of S4 is negative, on the other hand, the MPU 202 ends the process.

Even when corresponding features are found in the correspondence table 231, an incorrect operation can be avoided by making the user check an operation as in the second process before an operation signal is transmitted to the operation target device 30.

With the second process, operation accuracy is expected to improve even in a period when the user's reproducibility of features of brain wave information is low.

Although the correspondence table 231 is used in the present exemplary embodiment, a learning model storing relationships between input features and output operations may be used, instead, as described in a later exemplary embodiment.

Results of Experiments

A fact that the earphones 10 (refer to FIG. 2) can obtain the user's brain wave information will be described hereinafter on the basis of results of an experiment conducted by a third party and results of an experiment conducted by the present applicant.

Reliability of MindWave (NeuroSky) in Comparison with Earphones 10

FIG. 9 is a diagram illustrating a measurement point of a headset 40 equipped with a brain wave sensor capable of measuring brain waves with the earphones 10 worn by the user.

In this experiment, MindWave (registered trademark), which is manufactured by NeuroSky, Inc. and commercially available, is used as the headset 40 equipped with a brain wave sensor.

Whereas the earphones 10 use the external auditory meatuses as measurement points as described above, MindWave manufactured by NeuroSky Inc. uses a forehead 40A as a measurement point for brain waves.

The forehead 40A illustrated in FIG. 9 corresponds to Fp1, which is one of 21 sites specified in the 10-20 system recommended as an international standard of arrangement of electrodes for measuring brain waves.

Elena Ratti, et al., “Comparison of Medical and Consumer Wireless EEG Systems for Use in Clinical Trials” (https://www.frontiersin.org/articles/10.3389/fnhum.2017.003 98/full) has verified that brain waves measured by MindWave are equivalent to ones measured by medically approved EEG systems.

This thesis has been reviewed by Dimiter Dimitrov, PhD, a senior scientist at Duke University, and Marta Parazzini, PhD, at Polytechnic University of Milan and the National Research Council (CNR) in Italy.

FIG. 10 is a diagram illustrating measurement points for brain waves used in the thesis.

“B-Alert” (registered trademark) and “Enobio” in FIG. 10 are names of medically approved EEG systems in Europe and the U.S. “Muse” (registered trademark) and “MindWave” are names of consumer EEG systems.

In FIG. 10, sites indicated by hollow circles are measurement points used only by the medically approved EEG systems. Sites AF7, Ap1, AF8, A1, and A2, on the other hand, are measurement points used only by Muse, which is a consumer EEG system. Fp1 is a measurement point used by all the four EEG systems. That is, Fp1 is a measurement point of MindWave. The measurement points A1 and A2 are located between the auricles and the temples, not in the external auditory meatuses.

Details of the thesis will not be described here, but brain waves of five healthy subjects at rest are measured on two separate days. In this experiment, Fp1 on the forehead is used as a common measurement point, and brain wave patterns and power spectrum densities in an open eye state and a closed eye state are compared with each other. Evaluation in the thesis corresponds to evaluation of output α waves in the closed eye state.

In addition, in a “Conclusion” section of the thesis, it is described that power spectra measured by MindWave at Fp1 are substantially the same as with B-Alert and Enobio, which are the medically approved EEG systems, including results of reproduction tests, and peaks of α waves have been detected. It is also described that brain waves measured by MindWave include, as noise, blinking and movement in the open eye state. As a reason why the reliability of Muse is low, the thesis pointed out the possibility of an effect of artifacts.

Comparison between Results of Measurement with Earphones 10 and Results of Measurement with MindWave

Results of an experiment in which brain waves have been measured with the subjects wearing the earphones 10 (refer to FIG. 9) or MindWave will be described hereinafter.

As illustrated in FIG. 9, the earphones 10 uses the exterior auditory meatuses as measurement points, and MindWave uses the forehead 40A as a measurement point.

In the experiment conducted by the present applicant, there are 58 subjects. Three attention enhancement tests and three meditation enhancement tests on the same day are designed for each subject, and α waves observed in the closed eye state are measured.

Although the actual number of subjects is 83, an effect of artifacts is excessive in results of measurement performed on 25 subjects, and these results are excluded.

In each attention enhancement test, the subjects are instructed to keep looking at a tip of a pen 150 mm away for 30 seconds with their eyes open. This test creates a concentrated state to suppress appearance of α waves and increase beta waves.

In each meditation enhancement test, the subjects are instructed to meditate for 30 seconds with their eyes closed. This test corresponds to evaluation of output α waves in the closed eye state. In other words, this test aims to detect the percentage of increase in α waves in a relaxed state.

In the experiment, the meditation enhancement test is conducted after the attention enhancement test, and output α waves are evaluated.

When output α waves are evaluated, a subject is usually instructed to keep his/her eyes open for 30 seconds and then keep his/her eyes closed for 30 seconds, and this process is repeated twice, and an increase in α waves in the closed eye state is detected.

In the present experiment, however, the number of sets is increased in order to collect a large amount of data.

First, a reason why the meditation enhancement tests are conducted and a method used in the evaluation of output α waves in the closed eye state will be described.

FIG. 11 is a diagram illustrating the evaluation of output α waves. As illustrated in FIG. 11, raw data regarding brain waves can be roughly classified into delta waves, theta waves, α waves, beta waves, and gamma waves.

It is considered that the reproducibility of brain waves based on human motion is low and evaluation of the reproducibility of acquisition performance based on clinical data is difficult. α waves, however, tend to remain constant regardless of whether a person's eyes are open or closed.

Every type of brain wave tend to be observed in the open eye state, but every type of brain wave other than α waves tend to attenuate in the closed eye state. That is, α waves can be relatively easily observed without being affected even in the open eye state.

On the basis of this characteristic, raw data regarding brain waves in the experiment is subjected to a Fourier transform, and a spectral intensity Sn in a frequency band corresponding to each type of brain wave is determined as a feature value.

In the experiment, an α wave intensity ratio Tα is defined as a ratio (=Sα/ΣSn) of a spectral intensity Sα in an α band to the sum of spectral intensities in all the frequency bands (i.e., ΣSn), and whether the α wave intensity ratio Tα has increased in the closed eye state is determined.

If an increase in the α wave intensity ratio Tα is observed, it is proved that brain waves have been measured.

The comparison between the results of the measurement performed with the earphones 10 and the results of the measurement performed with MindWave will be described with reference to FIGS. 12A to 13B.

FIGS. 12A and 12B are diagrams illustrating the results of the measurement performed with MindWave. FIG. 12A illustrates a result of measurement at a time when the open eye state and the closed eye state are alternated twice without blinking, and FIG. 12B illustrates a result of measurement at a time when the open eye state and the closed eye state are alternated twice with blinking.

FIGS. 13A and 13B are diagrams illustrating the results of the measurement performed with the earphones 10 (refer to FIG. 2) used in the present exemplary embodiment. FIG. 13A illustrates a result of measurement at a time when the open eye state and the closed eye state are alternated twice without blinking, and FIG. 13B illustrates a result of measurement at a time when the open eye state and the closed eye state are alternated with blinking and movement of the jaw.

Without blinking, the result of the measurement performed with the earphones 10 and the result of the measurement performed with MindWave are closely similar to each other.

With blinking, however, artifacts caused by the blinking are evident in the results of the measurement performed with MindWave. This is probably because the forehead, which is used for measurement by MindWave, is close to the eyes and blinking in the open eye state tends to be detected as major artifacts. This has also been pointed out in the above-described thesis by Elena Ratti, et al.

Most of the artifacts due to blinking are observed in the delta band. When there are major artifacts as in FIG. 12, however, an increase in a waves might be erroneously detected. This is because as a result of an increase in the sum of the spectral intensities in all the frequency bands in the open eye state, the α wave intensity ratio Tα in the open eye state decreases, and the α wave intensity ratio Tα in the closed eye state looks relatively large. This is why the number of subjects has been reduced.

The artifacts caused by the blinking include not only potential variations derived from a living body due to movement of the eyelids but also potential variations derived from brain waves caused when the subjects try to move their eyelids.

In the result of the measurement performed with the earphones 10 (refer to FIG. 2) according to the present exemplary embodiment, on the other hand, artifacts due to blinking have not been described in a period of 0 to 30 seconds.

It has been confirmed, however, that artifacts due to a movement of the jaw for swallowing saliva are detected regardless of whether the subjects' eyes are open or closed. Most of the artifacts due to a movement of the jaw for swallowing saliva have been observed in the theta band.

The spectral intensity of the artifacts caused by swallowing of saliva, on the other hand, is much lower than that of the artifacts corresponding to blinking detected by MindWave. The spectral intensity of the artifacts caused by swallowing of saliva, therefore, has not affected an increase in α waves unlike in the case of MindWave.

The artifacts caused by swallowing of saliva, too, include not only potential variations derived from a living body in accordance with movement of the muscles in the jaw but also potential variations derived from brain waves caused when the subjects try to move the muscles in their jaws.

A reason why the movement of the jaw for swallowing saliva is taken as an example of the intentional movement of muscles while a user is desiring a certain operation in his/her mind in the above description is the occurrence of the artifacts illustrated in FIG. 13.

Next, an increase in α waves observed in the result of the measurement performed with the earphones 10 and an increase in α waves observed in the result of measurement performed with MindWave will be described with reference to FIG. 14A to FIG. 15C.

FIGS. 14A to 14C are diagrams illustrating other results of the measurement performed with MindWave. FIG. 14A illustrates changes in the percentage of spectral intensity in each frequency band at a time when the subjects have entered the closed eye state from the open eye state with blinking. FIG. 14B illustrates changes in the percentage of spectral intensity in each frequency band at a time when the subjects have entered the closed eye state from the open eye state without blinking. FIG. 14C illustrates a case where α waves do not increase.

FIGS. 15A to 15C are diagrams illustrating the results of the measurement performed with the earphones 10 (refer to FIG. 2) used in the present exemplary embodiment. FIG. 15A illustrates changes in the percentage of spectral intensity in each frequency band at a time when the subjects have entered the closed eye state from the open eye state with blinking. FIG. 15B illustrates changes in the percentage of spectral intensity in each frequency band at a time when the subjects have entered the closed eye state from the open eye state without blinking. FIG. 15C illustrates a case where α waves do not increase.

Vertical axes in FIGS. 14A to 15C represent the percentage of spectral intensity, and horizontal axes represent the frequency bands. Subjects corresponding to FIG. 14A are the same as those corresponding to FIG. 15A. Similarly, subjects corresponding to FIG. 14B are the same as those corresponding to FIG. 15B, and subjects corresponding to FIG. 14C are the same as those corresponding to FIG. 15C.

The distribution of the spectral intensity of MindWave (refer to FIGS. 14A to 14C) and the distribution of the spectral intensity of the earphones 10 (refer to FIGS. 15A to 15C) are different from each other in low frequency bands of delta waves to theta waves, but the same in the α band and higher.

In the experiment, the number of subjects with whom an increase in α waves has been observed with both MindWave and the earphones 10 is 46, which is slightly less than 80% of the total number of subjects, namely 58.

The number of subjects with whom an increase in a waves has been observed only with the earphones 10 is seven. In other words, an increase in a waves has been observed with 53 subjects in the case of the earphones 10. That is, in the case of the earphones 10, an increase in a waves has been observed with slightly more than 90% of the total number of subjects.

The number of subjects with whom an increase in a waves has been observed with neither MindWave nor the earphones 10 is five. Waveforms illustrated in FIGS. 14C and 15C indicate results of measurement performed on the five subjects.

FIGS. 16A and 16B are diagrams illustrating an example of presentation of parts in which spectral intensity has increased. FIG. 16A illustrates a result of the measurement performed with MindWave, and FIG. 16B illustrates a result of the measurement performed with the earphones 10 (refer to FIG. 2) used in the present exemplary embodiment. Vertical axes represent the percentage of spectral intensity, and horizontal axes represent frequency.

In FIGS. 16A and 16B, unlike in FIGS. 14A to 15C, actual frequency is used for the horizontal axes. In the above-described thesis by Elena Ratti, et al., horizontal axes represent actual frequency to describe an increase in α waves. The parts in which spectral frequency has increased are indicated by hollow circles in FIGS. 16A and 16B.

As illustrated in FIGS. 16A and 16B, in either measurement method, the percentage of spectral intensity decreases as the frequency increases. This holds true in the thesis by Elena Ratti, et al.

It has thus been confirmed that the earphones 10 used in the present exemplary embodiment, which measure brain waves at the exterior auditory meatuses, has measurement capability equivalent to that of MindWave.

Second Exemplary Embodiment

In the first exemplary embodiment, correspondence between features and operations stored in the correspondence tables 231 and 232 (refer to FIGS. 6A and 6B) are defined in advance.

For this reason, it is difficult to operate the operation target device 30 (refer to FIG. 1) as intended unless the user wearing the earphones 10 correctly generates features of brain wave information. In addition, standardized correspondences might not be applicable to every user.

In a second exemplary embodiment, therefore, an example of a mechanism for updating correspondences through machine learning will be described.

FIG. 17 is a diagram illustrating a schematic configuration of a brain wave operation system 1A used in the second exemplary embodiment. In FIG. 17, the same elements as in FIG. 1 are given the same reference numerals.

The brain wave operation system 1A illustrated in FIG. 17 includes the earphones 10 attached to the ears, an information terminal 20A wirelessly connected to the earphones 10, an operation target device 30A to be operated by the user, and a machine learning apparatus 50.

The operation target device 30A and the machine learning apparatus 50 are connected to each other over an IoT network. The machine learning apparatus 50 need not exist in the same space as the information terminal 20A and the operation target device 30A. The machine learning apparatus 50 may exist, for example, on the Internet.

The information terminal 20A according to the present exemplary embodiment has a function of transmitting features of brain wave information used to determine an operation signal to the machine learning apparatus 50. The information terminal 20A may also have a function of transmitting features of biological information other than brain waves to the machine learning apparatus 50.

The operation target device 30A according to the present exemplary embodiment has a function of transmitting a log (hereinafter referred to as a “reception log”) of a received operation signal to the machine learning apparatus 50. The operation target device 30A may transmit a reception log each time the operation target device 30A receives an operation signal or only if the machine learning apparatus 50 requests a reception log.

The machine learning apparatus 50 according to the present exemplary embodiment compares features of brain wave information and reception logs with each other and mechanically learns relationships between the brain wave information and operations. The machine learning apparatus 50 extracts reception logs relating to times at which the information terminal 20A has transmitted operation signals and learns relationships between features of brain wave information and operations intended by users.

A method in which correct operations are used, for example, may be performed for the learning.

First, a correct operation needs to be identified. If the operation target device 30A keeps performing a received operation for a certain period of time or more immediately after an operation signal is transmitted, for example, the operation performed by the operation target device 30A is regarded as an operation intended by the user. If the information terminal 20A transmits an operation signal for turning off the operation target device 30A while the operation target device 30A is off, however, this operation is probably incorrect. In this case, the operation target device 30A should receive an operation signal for turning on the operation target device 30A after receiving the operation signal for turning off the operation target device 30A. If the operation target device 30A receives an opposite operation signal immediately after a time at which an operation signal has been transmitted, therefore, an operation corresponding to the opposite operation signal is regarded as an operation intended by the user.

Although operations intended by the user can be identified in accordance with predetermined rules like this, operations intended by the user when the user has operated the operation target device 30A through brain waves may be determined using a model obtained through machine learning based on reception logs, instead. Alternatively, the user may transmit correct operations to the machine learning apparatus 50 through the information terminal 20A.

In any case, the machine learning apparatus 50 performs so-called supervised learning using pairs of an identified operation and a corresponding set of features of brain wave information. A method of deep learning is used for the learning. Long short-term memory (LSTM) blocks may be introduced to hidden layers. Convolutional layers or pooling layers may also be added as hidden layers.

The machine learning apparatus 50 learns correspondences between features of brain wave information and operations through machine learning and transmits a result of the machine learning to the information terminal 20A as a correspondence table or a learning model.

FIG. 18 is a diagram illustrating an example of the internal configuration of the information terminal 20A used in the second exemplary embodiment. In FIG. 18, the same components as in FIG. 4 are given the same reference numerals.

FIG. 18, too, illustrates only devices of the information terminal 20A relating to a function of generating an operation signal for operating the operation target device 30A (refer to FIG. 17) from biological information such as brain waves.

In the case of the information terminal 20A illustrated in FIG. 18, a function of transmitting features of brain wave information used to estimate an operation signal to the machine learning apparatus 50 (refer to FIG. 17) is added to an operation estimation section 223A.

The information terminal 20A is also provided with a Wi-Fi (registered trademark) module 205 for communication with the machine learning apparatus 50. The Wi-Fi module 205 is used to transmit features of brain wave information to the machine learning apparatus 50 and receive a correspondence table or a learning model from the machine learning apparatus 50. The learning model stores correspondences between input features of brain wave information and output operations.

In FIG. 18, the semiconductor 203 stores a correspondence table 231A for features of brain wave information received from the machine learning apparatus 50. It is needless to say that the semiconductor 203 may store a learning model instead of the correspondence table 231A.

In the case of the brain wave operation system 1A according to the present exemplary embodiment, relationships between features of brain wave information caused in relation to operations performed by the user wearing the earphones 10 (refer to FIG. 17) through brain waves and operations are learned through machine learning, and the accuracy of correspondences between features of brain wave information and operations stored in the correspondence table 231A improves.

As a result, when machine learning is performed without identifying users, the accuracy of operations performed, through brain waves, by unspecified users wearing the earphones 10 capable of measuring brain waves improves.

When machine learning is performed with certain users, on the other hand, the accuracy of operations based on brain waves improves thanks to a correspondence table 231 or a learning model specific to the certain users.

Third Exemplary Embodiment

Although the operation target device 30A (refer to FIG. 17) is connected to the machine learning apparatus 50 (refer to FIG. 17) over an IoT network in the second exemplary embodiment, a case where the operation target device 30A is not connected to the machine learning apparatus 50 (refer to FIG. 17) over an IoT network or the like will be described in a third exemplary embodiment.

FIG. 19 is a diagram illustrating a schematic configuration of a brain wave operation system 1B used in the third exemplary embodiment. In FIG. 19, the same components as in FIG. 17 are given the same reference numerals.

The brain wave operation system 1B illustrated in FIG. 19 includes the earphones 10 attached to the ears, an information terminal 20B wirelessly connected to the earphones 10, the operation target device 30 to be operated by the user, and a machine learning apparatus 50A.

In the present exemplary embodiment, the machine learning apparatus 50A obtains information regarding correct operations through sounds uttered by the user. The information terminal 20B therefore has a function of analyzing a sound obtained by the microphone 122 (refer to FIG. 3) and transmitting a result of the analysis to the machine learning apparatus 50A.

The machine learning apparatus 50A may analyze a sound, instead. A sound recognition service provided through a server on the Internet or the like may be used to analyze a sound.

In FIG. 19, the user utters a sound corresponding to “turn on”, which is also being desired by the user in his/her mind. Although the whole operation looks the same as one achieved using a smart speaker because the operation target device 30 is operated in accordance with an operation orally requested by the user, a sound uttered by the user in the present exemplary embodiment is given as a correct operation estimated from with features of brain wave information.

FIG. 20 is a diagram illustrating an example of the internal configuration of the information terminal 20B used in the third exemplary embodiment. In FIG. 20, the same components as in FIG. 18 are given the same reference numerals.

FIG. 20, too, illustrates only devices of the information terminal 20B relating to a function of generating an operation signal for operating the operation target device 30 from biological information such as brain waves.

A sound information obtaining section 224 that obtains an operation from audio data received through the Bluetooth module 201 is added to the information terminal 20B illustrated in FIG. 20.

The sound information obtaining section 224 used in the present exemplary embodiment performs a process for analyzing a sound and outputting an operation.

When the machine learning apparatus 50A or an external sound recognition service is used, the sound information obtaining section 224 transmits obtained audio data to the machine learning apparatus 50A or the external sound recognition service.

In the brain wave operation system 1B according to the present exemplary embodiment, an oral instruction given by the user can be used as a correct operation based on brain waves, and a processing load is smaller than when a correct operation is estimated using a reception log.

In addition, since a correct operation is given as a sound, the accuracy of training data improves, which accordingly improves the accuracy of machine learning in the present exemplary embodiment.

Fourth Exemplary Embodiment

FIG. 21 is a diagram illustrating a schematic configuration of a brain wave operation system 1C used in a fourth exemplary embodiment. In FIG. 21, the same components as in FIG. 19 are given the same reference numerals.

In the present exemplary embodiment, machine learning is performed inside an information terminal 20C.

FIG. 22 is a diagram illustrating an example of the internal configuration of the information terminal 20C according to the fourth exemplary embodiment. In FIG. 22, the same components as in FIG. 20 are given the same reference numerals.

A machine learning section 225 is added to the information terminal 20C illustrated in FIG. 22 as a function of an MPU 202C. The machine learning section 225 is provided by executing a program using the MPU 202C.

In addition, in the present exemplary embodiment, tables 233 and 234 storing a history of operations based on brain waves and information indicating whether the operations are correct are added to the semiconductor 203. The table 233 stores features of brain wave information, and the table 234 stores biological information other than brain waves.

The machine learning section 225 according to the present exemplary embodiment performs machine learning using the information stored in the table 233 as training data to update the correspondence table 231A for features of brain wave information. Similarly, the machine learning section 225 performs machine learning using the information stored in the table 234 as training data to update a correspondence table 232A for features of biological information other than brain waves.

The sound information obtaining section 224 according to the present exemplary embodiment, too, performs a process for analyzing a sound and outputting an operation. When an external sound recognition service is used, the sound information obtaining section 224 transmits obtained audio data to the external sound recognition service and obtains an operation as a result of an analysis conducted by the external sound recognition service. In either case, the operation is stored in the table 233 of the semiconductor 203.

If the accuracy of an operation estimated from features of brain wave information or features of biological information other than brain waves is low, an operation estimation section 223C according to the present exemplary embodiment transmits an orally requested operation to the operation target device 30.

FIGS. 23A and 23B are diagrams illustrating an example of the tables 233 and 234 storing a history of operations and information indicating whether the operations are correct. FIG. 23A illustrates the table 233 storing a history of operations estimated from features of brain wave information and information indicating whether the operations are correct, and FIG. 23B illustrates the table 234 storing a history of operations estimated from features of biological information other than brain waves and information indicating whether the operations are correct.

The table 233 stores times at which operations based on brain waves have been performed, features of brain wave information, estimated operations, operations indicated by sounds uttered by the user as the user's intention, and information indicating whether operations associated with the features are correct.

In FIG. 23A, it is indicated that an operation estimated from features of brain wave information at 12:45:52 on 10/15/20XX is the same as an operation indicated by a sound uttered by the user. The estimated operation is therefore “correct”.

An operation estimated from features of brain wave information at 12:46:10 on 10/15/20XX, on the other hand, is different from an operation indicated by a sound uttered by the user. The estimated operation is therefore “incorrect”.

The table 234 stores times at which operations based on brain waves have been performed, features of biological information other than brain waves, estimated operations, operations indicated by sounds uttered by the user as the user's intention, and information indicating whether operations associated with the features are correct.

Operations estimated from features of biological information other than brain waves at 12:45:52 and 12:46:10 on 10/15/20XX are the same as operations indicated by sounds uttered by the user. The estimated operations are therefore “correct”.

The machine learning section 225 (refer to FIG. 22) according to the present exemplary embodiment keeps performing machine learning using the history stored in the table 233 as training data to update the correspondence table 231A for features of brain wave information. The machine learning section 225 also keeps performing machine learning using the history stored in the table 234 as training data to update the correspondence table 232A for features of biological information other than brain waves. When a learning model is stored instead of the correspondence table 231A, the learning model stored in the semiconductor 203 is updated using a learning model updated through machine learning.

Next, an example of a process performed by the information terminal 20C (refer to FIG. 21) by executing a program using the MPU 202C (refer to FIG. 22) will be described.

FIG. 24 is a flowchart illustrating an example of a process performed by the information terminal 20C after the information terminal 20C receives a digital signal including brain wave information. “S” in FIG. 24 denotes a step.

Upon receiving a digital signal including brain wave information from the earphones 10 (refer to FIG. 21), the MPU 202C obtains, from the received digital signal, features of brain wave information, features of biological information other than brain waves, and an operation indicated by a sound (S21).

Next, the MPU 202C refers to the table 233 (refer to FIG. 23A) and calculates a matching level between operations estimated from features of brain wave information and intended operations (S22). The matching level may be calculated only on the basis of features similar to the features of brain wave information obtained in S21 or all samples in the past.

The matching level may be calculated while specifying a certain period of time. This is because, even when a matching level of latest samples is high, for example, an operation based on brain waves might not be achieved if a large number of past samples whose matching level is low are included. The matching level may be calculated, therefore, while limiting a period of time in which samples used for the calculation have appeared.

Next, the MPU 202C compares the calculated matching level with threshold 1 (S23). Threshold 1 is, say, 90%. This percentage, however, is just an example. In addition, the user may change threshold 1. Threshold 1 is an example of a first threshold.

When the number of samples is small, the accuracy of the estimation of an operation based on the current features of the brain wave information is not assured even if the matching level is high. When the number of samples is smaller than a predetermined value, therefore, a negative result may be automatically output.

If a result of S23 is positive, the MPU 202C transmits the operation corresponding to the features of the brain wave information (S24). This is a state in which the operation target device 30 (refer to FIG. 21) can be accurately operated through brain waves.

If the result of S23 is negative, on the other hand, the MPU 202C according to the present exemplary embodiment calculates a matching level between operations estimated from features of biological information other than brain waves and intended operations (S25).

Although an operation estimated from features of biological information other than brain waves is assumed to be correct in the first exemplary embodiment, an operation estimated from obtained features of biological information can be incorrect in the present exemplary embodiment. A case where the movement of the jaw for swallowing saliva is not large enough and a case where a movement of muscles from which features can be hardly obtained is used for performing an operation, for example, are also assumed.

This matching level, too, may be calculated only on the basis of features similar to the features of the biological information other than brain waves obtained in S21 or all the samples in the past.

A period of time to which samples belong, which is used to calculate the matching level, may be determined as a period of time used to calculate the matching level as in the case of the features of the brain wave information.

Next, the MPU 202C compares the calculated matching level with threshold 2 (S26). Threshold 2 is, say, 95%. Because the features of the biological information other than brain waves are based on a movement of muscles that can be intentionally moved by the user, threshold 2 may be higher than threshold 1 used in S23. The percentage, however, is just an example, and the user may be change threshold 2. Threshold 2 is an example of a second threshold.

If a result of S26 is positive, the MPU 202C transmits an operation corresponding to the features of the biological information other than brain waves (S27).

If the result of S26 is negative, on the other hand, the MPU 202C transmits the operation indicated by the sound (S28).

Because the brain wave operation system 1C according to the present exemplary embodiment, too, can use an oral instruction given by the user as a correct operation based on brain waves, the accuracy of training data improves, which accordingly improves the accuracy of machine learning.

In addition, because correspondences between features of brain wave information and operations can be learned while associating the correspondences with an account of the user wearing the earphones 10 (refer to FIG. 21) in the present exemplary embodiment, the correspondence tables 231A and 232A can be adjusted to the user. Consequently, the accuracy of the user's operations based on brain waves improves.

Fifth Exemplary Embodiment

FIG. 25 is a diagram illustrating a schematic configuration of a brain wave operation system 1D used in a fifth exemplary embodiment. In FIG. 25, the same components as in FIG. 21 are given the same reference numerals.

The present exemplary embodiment is the same as the fourth exemplary embodiment in that the process of machine learning is performed inside an information terminal 20D.

The present exemplary embodiment, however, is different from the fourth exemplary embodiment in terms of a process performed to estimate an operation.

FIG. 26 is a diagram illustrating an example of the internal configuration of the information terminal 20D used in the fifth exemplary embodiment. In FIG. 26, the same components as in FIG. 22 are given the same reference numerals.

The information terminal 20D illustrated in FIG. 26 is different from the information terminal 20C in that the semiconductor 203 stores a table 235. A process performed by an operation estimation section 223D is also different from that performed by the operation estimation section 223C.

FIG. 27 is a diagram illustrating an example of the table 235 storing a history of operations based on brain waves and information indicating whether the operations are correct.

The fifth exemplary embodiment is different from the fourth exemplary embodiment in that combinations of features of brain wave information and features of biological information other than brain waves are each associated with a single time point in the table 235.

The table 235 stores times at which operations based on brain waves have been performed, operations indicated by sounds uttered by the user as the user's intention, features of brain wave information, features of biological information other than brain waves, operations estimated from the features of the brain wave information, information indicating whether the operations estimated from the features of the brain wave information are correct, operations estimated from the features of the biological information other than brain waves, information indicating whether the operations estimated from the features of the biological information other than brain waves are correct, and operations indicated by operation signals transmitted to the operation target device 30.

FIG. 27 stores records at 12:45:52 and 12:46:10 on 10/15/20XX.

The operation estimation section 223D according to the present exemplary embodiment uses the table 235 to determine an operation to be transmitted to the operation target device 30.

Next, an example of a process performed by the information terminal 20D (refer to FIG. 25) by executing a program using an MPU 202D (refer to FIG. 26) will be described.

FIG. 28 is a flowchart illustrating an example of a process performed by the information terminal 20D after the information terminal 20D receives a digital signal including brain wave information. “S” in FIG. 28 denotes a step.

In the present exemplary embodiment, the MPU 202D calculates a probability that features of brain wave information and features of biological information other than brain waves cooccur (S31). In the present exemplary embodiment, a probability that certain features of brain waves information occur at the same time that certain features of biological information other than brain waves occur.

The probability may be calculated while specifying on the basis of newly obtained certain features of brain wave information or all samples in the past.

The probability may be calculated using a certain period of time. This is because, even when a probability calculated from latest samples is high, for example, an operation based on brain waves might not be achieved if a large number of past samples from a period in which the user has not gotten accustomed to performing operations through brain waves are included. The probability may be calculated, therefore, while specifying a certain period of time in which samples used for the calculation have appeared.

Next, the MPU 202D determines whether the calculated probability is higher than threshold 3 (S32). Threshold 3 is, say, 95%. The percentage, however, is just an example. In addition, the user may change threshold 3. Threshold 3 is an example of a third threshold.

If a result of S32 is negative, the MPU 202D transmits an operation indicated by a sound to the operation target device 30 (S33). The result of S32 becomes negative when meaningful cooccurrence has not been detected.

If the result of S32 is positive, on the other hand, the MPU 202D calculates a matching level between operations estimated from features of biological information other than brain waves and intended operations (S34).

The result of S32 becomes positive when features of brain wave information and features of biological information other than brain waves have substantially one-to-one correspondences. In other words, a state has been established in which if the user moves certain muscles while desiring a certain operations in his/her mind, brain wave information having the same features is obtained.

Detection accuracy of the movement for swallowing saliva, for example, is high as described with reference to FIGS. 13A and 13B. When the user combines the movement of the jaw for swallowing saliva with an act of desiring an operation in his/her mind, therefore, it can be expected after the result of S32 becomes positive that an operation based only on brain waves can be quite accurately achieved.

In the present exemplary embodiment, however, an operation based on a movement of other muscles as an intentional movement of muscles is also assumed, and the following process is performed.

First, the MPU 202D calculates a matching level between operations estimated from features of biological information other than brain waves and intended operations (S34). This is because, even when there is cooccurrence between features of brain wave information and features of biological information other than brain waves, an operation based on the features of the biological information becomes incorrect if the features of the biological information other than brain waves are not ones intended by the user.

This matching level, too, may be calculated only on the basis of newly obtained features of brain wave information or all samples in the past.

The matching level may be calculated while specifying a certain period of time. This is because, even when a matching level of latest samples is high, for example, an operation based on brain waves might not be achieved if a large number of past samples from a period in which the user has not gotten accustomed to performing operations through brain waves are included. The matching level may be calculated, therefore, while limiting a period of time in which samples used for the calculation have appeared.

Next, the MPU 202D determines whether the calculated matching level is higher than threshold 4 (S35). Threshold 4 is, say, 95%. The percentage, however, is just an example. In addition, the user may change threshold 4. Threshold 4 is an example of a fourth threshold.

If a result of S35 is negative, the MPU 202D transmits an operation indicated by a sound to the operation target device 30 (S33).

If the result of S35 is positive, on the other hand, the MPU 202D transmits an operation associated with the features of the brain wave information to the operation target device 30 (refer to FIG. 25) (S36). The result of S35 becomes positive when the features of the brain wave information are likely to correctly reflect the user's intention.

In the present exemplary embodiment, the MPU 202D switches, after S36, to an operation mode in which only features of brain wave information are used (S37). As a result, operations based only on brain waves will be performed next time and later.

In the present exemplary embodiment, a state in which features that reflect the user's intention stably appear is thus identified using cooccurrence with features of biological information, whose reproducibility is higher than that of brain waves. As a result, switching from operations based on sounds to operations based on brain waves can be achieved.

Furthermore, in the present exemplary embodiment, the accuracy of correspondences between features of brain wave information and operations is improved using the machine learning section 225 (refer to FIG. 26).

The accuracy of the user's operations based on brain waves thus improves with the brain wave operation system 1D according to the present exemplary embodiment.

Other Exemplary Embodiments

Although some exemplary embodiments of the present disclosure have been described, the technical scope of the present disclosure is not limited to that of the above exemplary embodiments. It is obvious from the claims that the technical scope of the present disclosure also includes modes obtained by modifying or improving the above exemplary embodiments in various ways.

Although the movement for swallowing saliva has been taken as an example of the user's intentional movement for causing features of biological information other than brain waves in the above exemplary embodiments, the mimic muscles or the masseter muscles may be used, instead, as described above. The movement for swallowing saliva and a movement of the masseter muscles are examples of a movement of the jaw. Movements of the mimic muscles include a movement of muscles for moving the eyeballs.

In addition, although brain waves have been taken as an example of potential variations that can be measured by the earphones 10 (refer to FIG. 1), myoelectric potentials, heartbeats, heart potentials, pulsation, or pulse waves may be used, instead.

Although brain waves are measured with the earphones 10 inserted into the exterior auditory meatuses in the above exemplary embodiments, an earphone 10 may be inserted into one of the exterior auditory meatus.

FIG. 29 is a diagram illustrating an example of the appearance of an earphone 10A inserted into one of the ears. In FIG. 29, the same components as in FIG. 2 are given the same reference numerals. In the earphone 10A illustrated in FIG. 29, the earphone chip 11R is electrically separated by an insulation ring into a tip side and a body side. The electrode 11R1 is provided on the tip side, and the electrode 11L1 is provided on the body side. The electrode 11R2, which is the GND terminal, is electrically separated from the electrode 11L1 by an insulator, which is not illustrated.

With this configuration, the lithium battery 128 (refer to FIG. 3) is stored in the earphone body 12R.

Although only the mechanism for sensing potential variations is provided inside the earphones 10 (refer to FIG. 1) and the mechanism for estimating an operation from features of brain wave information or the like is provided for the information terminal 20 (refer to FIG. 1) or the like in the above exemplary embodiments, the earphones 10 may have a function of estimating an operation from features of brain wave information or the like, instead. In this case, the earphones 10 singlehandedly serve as an example of the information processing system.

In addition, although the information terminal 20 (refer to FIG. 1) or the like has a function of estimating an operation from features of brain wave information or the like in the above exemplary embodiments, a server on the Internet may execute part or the entirety of the function of estimating an operation from features of brain wave information or the like, instead. In this case, the server is an example of the information processing system.

Although the electrodes for measuring potential variations caused by brain waves and the like are provided for the earphones 10 in the above exemplary embodiments, the electrodes may be provided for another article, instead. Some specific examples will be described hereinafter.

For example, the electrodes for measuring potential variations caused by brain waves and the like may be provided for headphones that cover the auricles. In the case of headphones, the electrodes are provided at parts of earpads that come into contact with the head. At this time, the electrodes are arranged at positions where there is little hair and the electrodes can come into direct contact with the skin.

The article that comes into contact with an auricle may be a fashion accessory such as an earring or a spectacle-shaped device. These are examples of a wearable device.

FIG. 30 is a diagram illustrating an example of an earring 60 for which the electrodes for measuring brain waves are provided. The earring 60 illustrated in FIG. 30 includes the electrode 11R1 that comes into contact with an earlobe on a front side of the ear, the electrode 11L1 that comes into contact with the earlobe on a back side of the ear, and the electrode 11R2 that comes into contact with the earlobe at some position of a U-shaped body thereof. These electrodes are electrically separated from one another by an insulator, which is not illustrated. A battery for supplying power necessary for operation and a communication module such as Bluetooth are incorporated into an ornament, the U-shaped body, a screw for moving a plate-like member on which the electrode 11L1 is arranged, or the like.

FIG. 31 is a diagram illustrating an example of spectacles 70 for which the electrodes for measuring brain waves are provided. In the spectacles 70 illustrated in FIG. 31, the electrodes 11R1 and 11L1 are provided on a tip of a right temple (hereinafter referred to as a “temple tip”), and the electrode 11R2 is provided on a tip of a left temple. These electrodes are electrically separated from one another by an insulator, which is not illustrated. A battery for supplying power necessary for operation and a communication module such as Bluetooth are incorporated into a temple or a temple tip.

The electrodes for measuring brain waves may be incorporated into smart glasses or a headset called “head-mounted display” that displays information, instead. The electrodes may be mounted on a headset having a function of detecting a surrounding environment of the user and displaying an image assimilating to the surrounding environment.

FIG. 32 is a diagram illustrating an example in which the electrodes for measuring brain waves are provided for a headset 80 having a function of displaying an image assimilating to the surrounding environment of the user. The headset 80 illustrated in FIG. 32 has a configuration in which the electrodes for measuring brain waves are provided for hololens (registered trademark) manufactured by Microsoft (registered trademark) Corporation. A virtual environment experienced by the user wearing the headset 80 is called “augmented reality” or “mixed reality”.

In the headset 80 illustrated in FIG. 32, the electrodes 11R1, 11R2, and 11L1 are arranged in parts of a ring-shaped member that come into contact with the ears, the ring-shaped member being attached to the head. In the case of the headset 80 illustrated in FIG. 32, the electrodes 11R1 and 11R2 are arranged on a side of the right ear, and the electrode 11L1 is arranged on a side of the left ear.

A function of tracking a line of sight provided for the headset 80 may be executed to use the user's line of sight as a feature of biological information other than brain waves.

FIGS. 33A to 33D are diagrams illustrating an example in which the function of tracking a line of sight is executed to use the user's line of sight as a feature of biological information. FIG. 33A illustrates a case where the user's light of line is leftward, FIG. 33B illustrates a case where the user's line of sight is rightward, FIG. 33C illustrates a case where the user's line of sight is upward, and FIG. 33D illustrates a case where the user's line of sight is downward. Different operations are assigned to these lines of sight.

Although a case where biological information including brain waves is obtained using electrodes in contact with the user's ears in the above exemplary embodiments, a position at which the biological information including brain waves is obtained is not limited to the ears. For example, the electrodes may be provided at the forehead or another part on the head, instead.

In the case of the headset 80 (refer to FIG. 32), for example, the electrodes may be provided at some positions on the ring-shaped member attached to the head.

Although biological information including brain waves is obtained using the electrodes in contact with the user's head including the ears in the above exemplary embodiments, brain activity may be measured on the basis of changes in the amount of blood flow.

FIG. 34 is a diagram illustrating an example of a headset 90 that measures changes in the amount of blood flow caused by brain activity using near-infrared light. The headset 90 includes a ring-shaped body attached to the head. Inside the body, one or plurality of probes 91 that radiate near-infrared light onto the scalp and one or plurality of detection probes 92 that receive reflected light are provided. An MPU 93 controls the radiation of near-infrared light by the probes 91 and detects features of the user's brain waves by processing signals output from the detection probes 92.

Alternatively, an MEG may be used to obtain biological information including brain waves. A tunnel magnetoresistance (TMR) sensor, for example, is used to measure magnetic fields caused by electrical activity of nerve cells of the brain.

FIG. 35 is a diagram illustrating an example of an MEG machine 100. The MEG machine 100 illustrated in FIG. 35 has a structure in which TMR sensors 102 are arranged on a cap 101 attached to the head. Outputs of the TMR sensors 102 are input to an MPU, which is not illustrated, and a magnetoencephalogram is generated. In this case, the distribution of magnetic fields in the magnetoencephalogram is used as features of the user's brain waves.

Although artifacts measured along with brain waves and a line of sight are used as biological information other than brain waves in the above exemplary embodiments, features obtained from the user's facial expressions may be used, instead. For example, a smile, a face with the eyes closed, a face with the tongue stuck out, and the like may be obtained as features.

FIG. 36 is a diagram illustrating an example of a brain wave operation system 1E that uses the user's facial expressions as features of biological information other than brain waves. In FIG. 36, the same components as in FIG. 1 are given the same reference numerals.

The brain wave operation system 1E illustrated in FIG. 36 is provided with a camera 111 that captures images of the user's face. The images captured by the camera 111 are transmitted to an information terminal 20E. The information terminal 20E in this system configuration obtains features of facial expressions from the user's images using the biological information obtaining section 222 (refer to FIG. 4).

In the embodiments above, the term “MPU” refers to hardware in a broad sense. Examples of the MPU include general processors (e.g., CPU: Central Processing Unit) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application-Specific Integrated Circuit, FPGA: Field Programmable Gate Array, and programmable logic device).

In the embodiments above, the term “processor” is broad enough to encompass one processor or plural processors in collaboration which are located physically apart from each other but may work cooperatively. The order of operations of the processor is not limited to one described in the embodiments above, and may be changed.

The foregoing description of the exemplary embodiments of the present disclosure has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Obviously, many modifications and variations will be apparent to practitioners skilled in the art. The embodiments were chosen and described in order to best explain the principles of the disclosure and its practical applications, thereby enabling others skilled in the art to understand the disclosure for various embodiments and with the various modifications as are suited to the particular use contemplated. It is intended that the scope of the disclosure be defined by the following claims and their equivalents.

Number	Name	Date	Kind
20130096453	Chung et al.	Apr 2013	A1
20140108309	Frank	Apr 2014	A1
20170042439	Yeow	Feb 2017	A1
20180333575	Bouton	Nov 2018	A1
20190247662	Poltroak	Aug 2019	A1

Information processing system and non-transitory computer readable medium

Information

Patent Number

Date Filed

Date Issued

Inventors

Original Assignees

Examiners

Agents

CPC

Field of Search

CPC

International Classifications

Abstract

Description

Claims

Priority Claims (1)

US Referenced Citations (5)

Non-Patent Literature Citations (1)

Related Publications (1)