Patent Application
20240217533
Pursuant to 35 U.S.C. § 119(a), this application claims the benefit of earlier filing dates and right of priority to Korean Application No. 10-2023-0001311, filed on Jan. 4, 2023 and Korean Application No. 10-2023-0090408, filed on Jul. 12, 2023, the contents of which are hereby incorporated by reference herein in their entirety.
The present disclosure relates to a system and a method for assisting a driver using an electroencephalogram that grant objectivity to determination on driver's carelessness via an electroencephalogram signal of the driver and provide an alarm to the driver when a careless state of the driver is determined.
In general, in modern society, the number of casualties resulted from various traffic accidents is increasing along with an explosive increase in vehicles caused by an increase in income and a pursuit of convenience of consumers. To minimize such casualties in the traffic accidents, many countries around the world are making great efforts to develop safety vehicles.
Such a safety vehicle refers to a vehicle that reduces fatalities in the event of the traffic accident, reduces fatigue of a driver, and is convenient to drive. A basic concept thereof is to prevent the accidents in advance to improve safety of the vehicle and protect a pedestrian.
Recently, to provide a more convenient interface to a user, a development of a human-friendly interface using a voice, a facial expression, a gesture, and a bio-signal such as an electroencephalogram, an electrooculogram, and an electromyogram has been attempted.
The electroencephalogram, the representative bio-signal, refers to a bio-signal that directly or indirectly reflects a person's conscious or unconscious state, and refers to a wave that is measured in all areas of a human scalp and mainly has a frequency equal to or lower than 50 Hz with a potential difference of several tens of microvolts.
Such an electroencephalogram may be classified into a delta wave, a theta wave, an alpha wave, a beta wave, a gamma wave, and the like based on the frequency of the wave.
The delta wave is an electroencephalogram with a frequency equal to or lower than 4 Hz and typically appears in a normal sleep state. The theta wave is an electroencephalogram with a frequency in a range from about 4 to 8 Hz, appears mainly when mentally unstable or distracted, and occurs when in a drowsy state or falling asleep. The alpha wave is an electroencephalogram with a frequency in a range from about 8 to 12 Hz and mostly appears clearly when mentally stable and in a comfortable psychological state with eyes closed. The beta wave refers to an electroencephalogram with a frequency in a range from about 12 to 30 Hz, appears stronger when under stress, such as a little anxiety or tension, and mainly appears when paying more than a certain amount of attention. In addition, the gamma wave refers to an electroencephalogram with a frequency in a range from about 30 to 50 Hz and appears in states of extreme arousal and excitement.
As described above, the driver is required to have considerable concentration while driving the vehicle, and the accident resulting in damage to the vehicle and a human life occurs because of momentary carelessness of the driver during the driving. In addition, the carelessness of the driver causes a large-scale accident that not only damages himself, but also damages others.
In addition, because a frequency band, a spectral density, or the like of an electroencephalogram signal measured from the driver may be different for each driver, there is a limit in consistently determining a careless state of the driver via the electroencephalogram signal.
This Summary is provided to introduce a selection of concepts in simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
The present disclosure provides a system and a method for assisting a driver using an electroencephalogram. More specifically, it is to provide a system and a method for assisting a driver that may grant objectivity to determination on carelessness of the driver by analyzing a frequency component in an electroencephalogram signal of the driver measured by an electroencephalogram measurer and quantifying a carelessness state of the driver using the analyzed frequency component.
In addition, it is to provide a system and a method for assisting a driver that may reduce carelessness of a driver by determining a careless state of the driver and outputting the same via an app on a mobile terminal, and controlling a vehicle to provide an alarm to the driver.
In addition, it is to provide a system and a method for assisting a driver that may grant accuracy to determination on carelessness of the driver by converting information that takes into account not only an electroencephalogram signal of the driver but also driving information of a mobility into big data.
In addition, it is to provide alarm information necessary for each situation by collecting/refining driving information acquired from a plurality of mobilities and an electroencephalogram signal for each driving information at a server end and providing a carelessness-related alarm.
In addition, it is to provide a wearable device whose performance is improved and whose weight is reduced by making up for shortcomings of the wearable device used for electroencephalogram measurement.
In addition, it is to provide a mobile terminal to which an app that considers electroencephalogram characteristics for each driver and processes an electroencephalogram signal based on an operation state is applied.
The problems to be solved by the present disclosure are not limited to the above, and other problems not mentioned will be clearly understood by those skilled in the art to which the present disclosure belongs from the description below.
A first aspect of the present disclosure provides a system for assisting a driver using an electroencephalogram including an electroencephalogram measurer that measures an electroencephalogram signal of the driver, a mobile terminal that receives the measured electroencephalogram signal and outputs a careless state of the driver determined based on the received electroencephalogram signal via an app, and a controller that controls a vehicle based on the determined careless state of the driver and provides an alarm to the driver, wherein the mobile terminal includes a processor that analyzes a frequency component in the received electroencephalogram signal and quantifies the careless state of the driver using the analyzed frequency component.
In one implementation of the first aspect, the processor may analyze the frequency component in the received electroencephalogram signal via a window size in a first time unit.
In one implementation of the first aspect, the processor may shift the window size into a second time unit on a time axis and analyze the frequency component based on the window size.
In one implementation of the first aspect, the analyzing of the frequency component by the processor may include classifying the frequency component by performing Fourier transform on the received electroencephalogram signal, and the classified frequency component may include at least one of theta, beta, or SMR frequencies.
In one implementation of the first aspect, the processor may use the classified frequency component to quantify the careless state of the driver into a numerical value.
In one implementation of the first aspect, the processor may determine the careless state of the driver by comparing the numerical value with a preset alertness value.
In one implementation of the first aspect, the mobile terminal may output at least one of an emoji, a text, a color, and the received electroencephalogram signal via the app.
In one implementation of the first aspect, the mobile terminal may output a navigation screen or the number of alarms provided to the driver via the app.
In one implementation of the first aspect, the controller may provide the alarm to the driver by controlling the vehicle, including at least one of a display, an interior light, an air conditioner, a seat, and a speaker of the vehicle.
A second aspect of the present disclosure provides a system for assisting a driver using an electroencephalogram including a wearable device that measures an electroencephalogram signal of the driver, a mobile terminal equipped with an app, wherein the app receives the electroencephalogram signal measured by the wearable device and a driving information signal acquired from a mobility, processes the received electroencephalogram signal and driving information signal and provides the processed electroencephalogram signal and driving information signal to a server, and receives carelessness alarm information converted into big data from the server, and a controller that controls the mobility based on the carelessness alarm information and provides a carelessness-related alarm to the driver.
In one implementation of the second aspect, the carelessness alarm information may include driving information acquired from a plurality of mobilities and information determined based on an electroencephalogram signal for each driving information.
In one implementation of the second aspect, the driving information signal may include at least one of a GPS signal, a moving path signal of the mobility, a time information signal, and a weather information signal.
In one implementation of the second aspect, the carelessness alarm information may include at least one of information on in which moving path or location carelessness occurs, information on in what time period the carelessness occurs, and information on under what weather the carelessness occurs.
In one implementation of the second aspect, the controller may provide the carelessness-related alarm to the driver based on at least one of a moving path, a time, and a weather corresponding to the carelessness alarm information.
In one implementation of the second aspect, the wearable device may include a measurer wearable on a left ear of the driver, and a functional member including a battery that supplies power to the measurer, and the functional member may be fixable to a body part other than the ear of the driver or a fixture.
In one implementation of the second aspect, the app may measure electroencephalogram characteristics of the driver using an electroencephalogram signal measured during a predetermined time period while the driver wears the wearable device and drives, and measure an electroencephalogram signal after the predetermined time period in consideration of the electroencephalogram characteristics of the driver.
A third aspect of the present disclosure provides a method for assisting a driver using an electroencephalogram including measuring, by an electroencephalogram measurer, an electroencephalogram signal of the driver, receiving, by a mobile terminal, the measured electroencephalogram signal and determining a careless state of the driver based on the received electroencephalogram signal, outputting, by an app installed on the mobile terminal, the determined careless state of the driver, and controlling, by a controller, a vehicle based on the determined careless state of the driver and providing an alarm to the driver, wherein the determining of the careless state of the driver includes analyzing a frequency component in the received electroencephalogram signal and quantifying the careless state of the driver using the analyzed frequency component.
In one implementation of the third aspect, the determining of the careless state of the driver may include analyzing the frequency component in the received electroencephalogram signal via a window size in a first time unit.
In one implementation of the third aspect, the window size may be shifted into a second time unit on a time axis and the frequency component may be analyzed based on the window size.
In one implementation of the third aspect, the analyzing of the frequency component may include classifying the frequency component by performing Fourier transform on the received electroencephalogram signal, and the classified frequency component may include at least one of theta, beta, and SMR frequencies.
The system and the method for assisting the driver using the electroencephalogram according to the present disclosure may grant the objectivity to the determination on the carelessness of the driver by analyzing the frequency component in the electroencephalogram signal of the driver measured by the electroencephalogram measurer and quantifying the carelessness state of the driver using the analyzed frequency component.
In addition, the carelessness of the driver may be reduced by determining the careless state of the driver and outputting the same via the app on the mobile terminal, and controlling the vehicle to provide the alarm to the driver.
In addition, the accuracy may be granted to the determination on the carelessness of the driver by converting the information that takes into account not only the electroencephalogram signal of the driver but also the driving information of the mobility into the big data.
In addition, the alarm information necessary for each situation may be provided by collecting/refining the driving information acquired from the plurality of mobilities and the electroencephalogram signal for each driving information at the server end and providing the carelessness-related alarm.
In addition, the wearable device whose performance is improved and whose weight is reduced by making up for the shortcomings of the wearable device used for the electroencephalogram measurement may be provided.
In addition, the mobile terminal to which the app that considers the electroencephalogram characteristics for each driver and processes the electroencephalogram signal based on the operation state is applied may be provided.
Additional scope of applicability of the present disclosure will become apparent from the detailed description below. However, because various changes and modifications within the spirit and scope of the present disclosure may be clearly understood by those skilled in the art, the detailed description and specific embodiments, such as preferred embodiments, of the present disclosure should be understood as being given only as examples.
Other features and aspects will be apparent from the following detailed description, the drawings, and the claims.
Description will now be given in detail according to exemplary embodiments disclosed herein, with reference to the accompanying drawings. The same or equivalent components may be provided with the same reference numbers, and description thereof will not be repeated. As used herein, the suffixes “module” and “part” are added or used interchangeably to facilitate preparation of this specification and are not intended to suggest distinct meanings or functions. In describing embodiments disclosed in this specification, relevant well-known technologies may not be described in detail in order not to obscure the subject matter of the embodiments disclosed in this specification. In addition, it should be noted that the accompanying drawings are only for easy understanding of the embodiments disclosed in the present specification, and should not be construed as limiting the technical spirit disclosed in the present specification. As such, the present disclosure should be construed to extend to any alterations, equivalents and substitutes in addition to those which are particularly set out in the accompanying drawings.
Although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are generally only used to distinguish one element from another.
It will be understood that when an element is referred to as being “connected with” another element, the element can be directly connected with the other element or intervening elements may also be present. In contrast, it will be understood that when an element is referred to as being “directly connected with” another element, there are no intervening elements present.
A singular representation may include a plural representation unless it represents a definitely different meaning from the context.
The terms such as “include” or “have” used herein are intended to indicate that features, numbers, steps, operations, elements, components, or combinations thereof used in the following description exist and it should be thus understood that the possibility of existence or addition of one or more different features, numbers, steps, operations, elements, components, or combinations thereof is not excluded.
The driver assistance system 100 according to one embodiment of the present disclosure may include an electroencephalogram measurer 110, a mobile terminal 120, and a controller 130 that controls the vehicle 140. In this regard, the electroencephalogram measurer 110 may include an electrode 111, a filter 112, a converter 113, a signal processor 114, storage 115, a communicator 116, and a power supply 117.
The driver assistance system 100 according to one embodiment of the present disclosure may measure an electroencephalogram signal of a driver via the electroencephalogram measurer 110. An electroencephalogram (EEG) is a phenomenon in which thoughts, emotions, and behaviors of an individual are generated by communication between nerve cells in a brain, and is a synchronized electrical wave generated when the nerve cells in the cerebral cortex transmit signals to each other. As described above, the electroencephalogram appears in various frequency bands, and is able to be specifically classified into a delta wave, a theta wave, an alpha wave, a beta wave, a gamma wave, and the like based on the frequency bands.
In this regard, the electroencephalogram measurer 110 may include an earset that is mounted around an ear of the driver and is able to measure an electroencephalogram signal around a left or right temporal lobe of the driver while driving. In another embodiment, the electroencephalogram measurer 110 may include a headset using a multi-channel electrode.
In addition, in the driver assistance system 100 according to one embodiment of the present disclosure, the electroencephalogram measurer 110 may include various sensors or equipment that may measure the electroencephalogram signal of the driver in real time, including the earset and headset described above.
The electrode 111 may measure the electroencephalogram signal of the driver during the driving and/or stopping of the driver in the vehicle 140 via an electrode. In this regard, the electrode 111 may include an EEG electrode, a REF electrode, and a GND electrode. The EEG electrode is a conductive electroencephalogram electrode, and is able to measure the electroencephalogram signal of the left or right temporal lobe by being in contact with an area near a left or right temporal lobe temple of the driver. The REF electrode is a reference electrode, and is able to provide a reference potential when the EEG electrode measures the electroencephalogram signal. The GND electrode is a ground electrode, is located on a back of an earflap, and is able to serve as a ground when the EEG electrode measures the electroencephalogram signal.
In one embodiment, the filter 112 may remove noise from the electroencephalogram signal of the driver measured by the electrode 111. The filter 112 may remove the noise from the electroencephalogram signal using a high pass filter, a low pass filter, or a band pass filter. As another embodiment, a filter such as a notch filter or an active filter may be used. In this regard, the noise may include noise generated by eye-blink of the driver, AC power noise, and a DC drift component as well as all elements that interfere with feature point extraction.
The converter 113 may convert the noise-removed electroencephalogram signal of the driver into a digital signal. The signal processor 114 may calculate one or more characteristics of the electroencephalogram from the electroencephalogram signal of the driver measured by the electroencephalogram measurer 110. The characteristics of the electroencephalogram calculated by the signal processor 114 may be, for example, at least one of a magnitude of the electroencephalogram signal in a time domain, a magnitude of the electroencephalogram signal in a frequency domain, and a magnitude of a signal-to-noise ratio (SNR) peak.
The storage 115 may store a reference electroencephalogram signal, that is, a preset alertness value, for determining a careless state of the driver based on the characteristics of the electroencephalogram. In this regard, the careless state of the driver may include a state in which the driver is not able to concentrate on the driving, including abnormal states such as drowsiness, reduced concentration, and surprise. In addition, the storage 115 may store the electroencephalogram signal of the driver measured via the electroencephalogram measurer 110 for a certain period of time. Such storage 115 may include an internal memory 123 and/or an external memory 123, and may include a volatile memory 123 such as a DRAM, a SRAM, or a SDRAM, a non-volatile memory 123 such as an one time programmable ROM (OTPROM), a PROM, an EPROM, an EEPROM, a mask ROM, a flash ROM, an NAND flash memory 123, or an NOR flash memory 123, and a storage device such as an SSD, a compact flash (CF) card, a SD card, a micro-SD card, a mini-SD card, an XD card, a flash drive such as a memory 123 stick, or an HDD.
The communicator 116 is a medium for transmitting and receiving information between the electroencephalogram measurer 110 and the mobile terminal 120 or the vehicle 140, and is able to transmit the electroencephalogram signal of the driver measured by the electroencephalogram measurer 110 to the mobile terminal 120 or the vehicle 140 via wireless communication. In the present embodiment, the communicator 116 may transmit and receive data with the mobile terminal 120 or the vehicle 140 via at least one communication among Bluetooth, radio frequency identification (RFID), infrared data association (IrDA), ultra-wideband (UWB), ZigBee, and wireless fidelity (Wi-Fi) technologies.
The power supply 117 may supply corresponding power throughout the electroencephalogram measurer 110. Specifically, the power supply 117 may include a converter that converts AC power to DC power and a DC/Dc converter that converts a level of the DC power. The power supply 117 may use a scheme of supplying the AC power via direct connection to an external power source, and may include a power supply 117 that includes a battery to be charged and used.
In the case of the former, a cable is connected to the power supply 117, so that it is difficult for the power supply 117 to move or a range of movement thereof is limited. In the case of the latter, the power supply 117 is free to move, but a weight and a volume thereof increase as much as those of the battery, and the power supply 117 must be directly connected to a power cable or coupled to a charging holder that supplies power for a certain period of time for charging. The charging holder may be connected to the electroencephalogram measurer 110 via a terminal thereof exposed to the outside, or the built-in battery may be charged when approaching the charging holder using a wireless scheme.
The mobile terminal 120 may include an inputter 121, an outputter 122, a memory 123, a communication module 124, and a processor 125. In this regard, the inputter 121 is for inputting video information (or signals), audio information (or signals), data, or information input by the user. When the information is input via the inputter 121, the processor 125 may control an operation of the mobile terminal 120 to correspond to the input information.
The outputter 122 is for generating an output related to a sense of sight, hearing, or touch, and is able to include a display 122a and a sound outputter 122b. In this regard, the display 122a may form a layer structure or may be integrally formed with a touch sensor. In addition, the sound outputter 122b may include a receiver, a speaker, a buzzer, and the like. In addition, information on the careless state of the driver, which will be described later, may be output via the outputter 122 and provided to the user.
The memory 123 may store data supporting various functions of the mobile terminal 120. The memory 123 may store a program for processing and controlling each signal within processor 125, and may store signal-processed video, sound, or data signals. For example, memory 123 may store an application program designed to perform various tasks that may be processed by the processor 125, and may selectively provide some of the stored application programs upon request from the processor 125.
The programs and the like stored in the memory 123 are not particularly limited as long as they may be executed by the memory 123. In addition, the memory 123 may perform a function for temporarily storing the video, audio, or data signals.
The memory 123 may include at least one of a volatile memory (e.g., a DRAM, a SRAM, a SDRAM, and the like) or a non-volatile memory (e.g., a flash memory, a hard disk drive (HDD), a solid-state drive (SSD), and the like).
The communication module 124 is a medium for transmitting and receiving information between the mobile terminal 120 and the electroencephalogram measurer 110 or the vehicle 140, and is able to receive the electroencephalogram signal of the driver measured by the electroencephalogram measurer 110 via the communication module 124 using wireless communication and is able to transmit the electroencephalogram signal to the vehicle 140.
The communicator 116 may transmit and receive wireless signals in a communication network based on wireless Internet technologies. In this regard, the wireless Internet technologies may include, for example, wireless LAN (WLAN), wireless-fidelity (Wi-Fi), wireless fidelity (Wi-Fi) direct, digital living network alliance (DLNA), wireless broadband (WiBro), world interoperability for microwave access (WiMAX), high speed downlink packet access (HSDPA), high speed uplink packet access (HSUPA), long term evolution (LTE), long term evolution-advanced (LTE-A), and the like.
In addition, the communicator 116 may support short-range communication using at least one of Bluetooth, radio frequency identification (RFID), infrared data association (IrDA), ultra wide band (UWB), ZigBee, near field communication (NFC), wireless-fidelity (Wi-Fi), Wi-Fi direct, and wireless universal serial bus (wireless USB) technologies.
Via a short-range wireless communication network, the wireless communication may be supported between the mobile terminal 120 and the communicator 116 of the electroencephalogram measurer 110, between the mobile terminal 120 and a wireless communication system, and between the mobile terminal 120 and a network where an external device is located. In addition, the above-mentioned short-range wireless communication network may be a short-range wireless personal area network.
The processor 125 may control all or some of the components included in the mobile terminal 120. Additionally, the processor 125 may control all or some of the components included in the mobile terminal 120 by running the application program stored in the memory 123.
In one embodiment, the processor 125 may determine whether the electroencephalogram measurer 110 is worn and, when the driver is not wearing the electroencephalogram measurer 110, generate a wearing recommendation signal to the driver to wear the electroencephalogram measurer 110. Such wearing recommendation signal maybe generated periodically until the driver wears the electroencephalogram measurer 110.
In addition, when the driver completes the driving, for example, when a transmission is placed in parking, the processor 125 may generate a removal recommendation signal to the driver to remove the electroencephalogram measurer 110 when the driver does not remove the electroencephalogram measurer 110. Such removal recommendation signal may be generated periodically until the driver removes the electroencephalogram measurer 110. In addition, the wearing recommendation signal and the removal recommendation signal generated by the processor 125 may be output via the outputter 122 of the mobile terminal 120 such that the driver may recognize the signals.
In particular, in the driver assistance system 100 according to one embodiment of the present disclosure, the processor 125 may analyze a frequency component of the electroencephalogram signal received via the communication module 124 and determine the careless state of the driver by quantifying the careless state of the driver using the analyzed frequency component. In addition, the controller 130 may control the vehicle 140 based on the determined careless state of the driver and provide an alarm to the driver.
The driver assistance system 100 according to one embodiment of the present disclosure may control the vehicle 140 via the controller 130 and provide the alarm using stimulation to the driver in various ways. This may be provided via the display 142, the interior light 143, the air conditioner 144, the seat 145, and the speaker 146 located in the vehicle 140.
The controller 130 may provide the alarm to the driver via visual stimulation, including displaying a screen via the display 142 of the vehicle 140 or controlling the interior light 143 of the vehicle 140. In this regard, the display 142 may be a display device that includes all of a center information display (CID), a head up display (HUD), and the like.
The controller 130 may control an intensity of the interior light 143 or a wavelength of the interior light 143. The interior light 143 may be a mood light or an ambient light inside the vehicle 140. Additionally, the interior light 143 may include an LED installed above a cluster fascia of a driver's seat or installed on a side surface of the seat 145, and the controller 130 may control ON and OFF of LED power to provide the alarm to the driver via the visual stimulation.
The controller 130 may provide the alarm to the driver via olfactory stimulation, including controlling an indoor temperature via the air conditioner 144 of the vehicle 140 or introducing external air. For example, the controller 130 may control the indoor temperature by operating an air conditioner to lower the indoor temperature when concentration of the driver decreases. To lower a carbon dioxide concentration, the controller 130 may control external air to flow into the vehicle 140 in an external air circulation mode, and may control the air conditioner 144 to generate the olfactory stimulation to the driver using an air freshener and the like.
Additionally, the controller 130 may provide the alarm to the driver via tactile stimulation, including controlling at least one of a massage device, a ventilation device, or an adjustable device installed in the seat 145 of the vehicle 140.
Additionally, the controller 130 may provide the alarm to the driver via auditory stimulation, including outputting sound via the speaker 146 of the vehicle 140. In one embodiment, the speaker 146 may be located on a headrest of the seat 145 and provide the alarm to the driver via the auditory stimulation at a location close to both ears of the driver.
As described above, because the frequency band, a spectral density, or the like of the electroencephalogram signal measured from the driver may be different for each driver, there is a limit in consistently determining the careless state of the driver via the electroencephalogram signal.
In relation, (a) in
However, as shown in (a) in
To solve the above-mentioned problem, the driver assistance system 100 according to one embodiment of the present disclosure is to ensure the objectivity of the determination on the carelessness of the driver regardless of the electroencephalogram signal that may be measured differently for each driver.
In other words, the frequency component of the electroencephalogram signal may be analyzed via the processor 125 described above with reference to
In this regard, the processor 125 may classify the frequency component by performing Fourier transform on the received electroencephalogram signal. In this regard, The Fourier transform may be a concept that includes all of Fast Fourier Transform (FFT), Short Time Fourier Transform (STFT), and the like.
In one embodiment, the frequency component classified via the processor 125 in the driver assistance system 100 according to one embodiment of the present disclosure may include at least one of theta (θ) beta (β), or SMR frequencies. In addition, the careless state of the driver may be quantified via Mathematical Formula below using the classified frequency component.
(b) in
According to one embodiment of the present disclosure, the careless state of the driver may be quantified as a value from 0 to 100 via Mathematical Formula above. Therefore, compared to the data in prior art shown in (a) in
In addition, a time point at which the carelessness of the driver occurs may be objectively and accurately determined via the same, so that the reliability of the determination on the carelessness of the driver may be increased and the unnecessary alarm may be prevented, thereby efficiently providing the alarm to driver.
Referring to
(a) in
In this regard, according to one embodiment of the present disclosure, t1 may be 4 seconds and t2 may be 1 second. In other words, the frequency component may be analyzed in the electroencephalogram signal via the 4-second window size 150 and the 1-second shift.
In this regard, the analyzing of the frequency component by the processor 125 may include classifying the frequency component by performing the Fourier transform on the received electroencephalogram signal as described above, and the classified frequency component may include at least one of the theta, beta, or SMR frequencies.
Therefore, as shown in
In this regard,
As shown in
Accordingly, the driver assistance system 100 according to one embodiment of the present disclosure may calculate the average value of the theta, beta, and SMR frequency intensities from the electroencephalogram signal every second via the window size 150 in the at least 4-second unit and quantify the careless state of the driver via Mathematical Formula above.
In addition, the processor 125 may determine the careless state of the driver by comparing the quantified value with the preset alertness value. As described above, according to one embodiment of the present disclosure, the careless state of the driver may be quantified as the value from 0 to 100 via Mathematical Formula above.
In this regard, the preset alertness value may be stored in the memory 123 as described above with reference to
Therefore, according to the above example, when the value quantified via Mathematical Formula above is 50, this corresponds to a section where the alertness is poor compared to the preset alertness value, which may be determined as the careless state of the driver. In addition, when the driver is determined to be in the careless state, the vehicle 140 may be controlled via the controller 130 to provide the alarm to the driver.
Additionally, as one embodiment of the present disclosure, continuous provision of the alarms to the driver may be restricted to prevent the unnecessary and excessive alarms from being provided to the driver. For example, after the alarm is provided to the driver, there may be a certain time interval until a next alarm is provided.
Additionally, as one embodiment of the present disclosure, the vehicle 140 may be controlled via the controller 130 such that more than the reference number of alarms are not provided within a reference time. In this case, when the reference time elapses, the reference number may also be initialized. That is, for example, when the reference time is 10 minutes and the reference number is 10 minutes, the alarm may not be provided more than 10 times in 10 minutes. In addition, the alarm may be provided by re-initializing the reference number every 10 minutes.
Referring to
The careless state of the driver may be quantified via Mathematical Formula above, and the careless state of the driver may be determined via the comparison with the preset alertness value. In this regard,
As shown in
In addition, as described above with reference to
Additionally, in the driver assistance system 100 according to one embodiment of the present disclosure, the mobile terminal 120 may output a navigation screen or the number of alarms provided to the driver via the app. (a) in
First, the electroencephalogram signal of the driver may be measured using the electroencephalogram measurer (S110). In addition, the measured electroencephalogram signal may be received on the mobile terminal (S120). In this regard, the driver assistance method according to one embodiment of the present disclosure may determine whether the driver is wearing the electroencephalogram measurer and, when the driver is not wearing the electroencephalogram measurer, generate the wearing recommendation signal to ask the driver to wear the electroencephalogram measurer. Such wearing recommendation signal may be generated periodically until the driver wears the electroencephalogram measurer. In addition, when the driver completes the driving, for example, when the transmission is placed in park, the removal recommendation signal to ask the driver to remove the electroencephalogram measurer may be generated when the driver does not remove the electroencephalogram measurer. Such removal recommendation signal may be generated periodically until the driver removes the electroencephalogram measurer. In addition, the wearing recommendation signal and the removal recommendation signal may be output via the outputter of the mobile terminal such that the driver may recognize those.
The driver assistance method according to one embodiment of the present disclosure may analyze the frequency component in the electroencephalogram signal received from the electroencephalogram measurer and quantify the careless state of the driver using the analyzed frequency component (S130). In addition, the careless state of the driver may be determined by comparing the quantified value with the preset alertness value (S140).
In this regard, the determining of the careless state of the driver may include analyzing the frequency component via the window size in the first time unit in the received electroencephalogram signal. Additionally, shifting the window size into the second time unit on the time axis and analyzing the frequency component as much as the window size may be included.
In this regard, as mentioned above, one embodiment of the present disclosure may calculate the average value of the theta, beta, and SMR frequency intensities for 4 seconds from the electroencephalogram signal every second via the 4-second window size and the 1-second shift, and quantify the careless state of the driver via Mathematical Formula above.
In this regard, the analyzing of the frequency component may include classifying the frequency component by performing the Fourier transform on the received electroencephalogram signal, and the classified frequency component may include at least one of the theta, beta, or SMR frequencies. In addition, the careless state of the driver may be quantified using the classified frequency component.
As described above, the Fourier transform may be the concept that includes all of the Fast Fourier Transform (FFT), the Short Time Fourier Transform (STFT), and the like. In addition, the careless state of the driver may be quantified as the value from 0 to 100 via Mathematical Formula above.
In this regard, the preset alertness value may be set, for example, by classifying the range equal to or lower than 55 into the range with the poor alertness, the range from 56 to 80 into the range with the good alertness, and the range from 81 to 100 into the range with the highest alertness. Therefore, according to the above example, when the value quantified via Mathematical Formula above is 50, this corresponds to the section where the alertness is poor compared to the preset alertness value, which may be determined as the careless state of the driver.
Thereafter, the determined careless state of the driver may be output via the app installed on the mobile terminal (S150). In addition, whether to provide the alarm to the driver may be determined considering whether the driver is in the carelessness state (S160), and the vehicle may be controlled via the controller to provide the alarm to the driver (S170).
The present disclosure may provide the alarm to the driver in various ways (S210), and in particular, the alarm may be provided to the driver via the various stimuli.
The alarm via the visual stimulation may be provided (S220), which may include outputting the screen via the display (S221) or controlling the interior light (S222). In this regard, the interior light may be the mood light or the ambient light inside the vehicle. Additionally, the interior light may include the LED installed above the cluster fascia of the driver's seat or installed on the side surface of the seat. In addition, the alarm may be provided to the driver via the visual stimulation by controlling ON and OFF of the LED power.
Additionally, the alarm via the olfactory stimulation may be provided (S230), which may include controlling the air conditioner (S231) to control the indoor temperature or introducing the external air. For example, the controller may control the indoor temperature by operating the air conditioner to lower the indoor temperature when the concentration of the driver decreases. To lower the carbon dioxide concentration, the external air may be controlled to flow into the vehicle 140 in the external air circulation mode, and the air conditioner may be controlled to generate the olfactory stimulation to the driver using the air freshener and the like.
Additionally, the alarm via the tactile stimulation may be provided (S240), which may include controlling the seat (S241). For example, the alarm may be provided to the driver via the tactile stimulation, including controlling the massage device installed in the seat or controlling the ventilation device or the adjustable device.
Additionally, the alarm via the auditory stimulation may be provided (S250), which may include outputting the sound via the speaker (S251). In one embodiment, the speaker may be located on the headrest of the seat and provide the alarm to the driver via the auditory stimulation at the location close to both ears of the driver.
The driver assistance system using the electroencephalogram shown in
Additionally, the driver assistance system according to the present embodiment may include a controller 150 that controls a mobility based on the carelessness alarm information received from the server 140 and provides a carelessness-related alarm to the driver.
The above-described driver assistance system may operate using various driving information acquisitors 120 and the server 140 equipped in the mobility as shown in
As shown in
It is proposed that, rather than collecting and processing such information to generate the carelessness alarm information on its own, the app 130_1 of the mobile terminal 130 according to the present embodiment utilizes the carelessness alarm information based on the information converted into the big data via the server 140. In other words, the server 140 may provide more accurate and diverse alarm information by utilizing the driving information corresponding to a plurality of mobilities and the electroencephalogram information corresponding to the driving information.
The electroencephalogram-based driver assistance service providing method shown in
First, the server 140 may acquire subscription information of the drivers of the plurality of mobilities (S210). As an example of the subscription information of the driver, subscription information of a bus driver may include information for a bus transportation company to subscribe the present driver assistance service and subscription information of an individual driver.
Next, when the plurality of mobilities are in travel, the server 140 may receive the driving information and the electroencephalogram signal information of the respective drivers of the plurality of mobilities (S220). As described above, the driving information may include the GPS information and moving path information (e.g., bus line information, current operation state information, and the like), and the electroencephalogram information for each moving path may be received. The electroencephalogram information received from the server 140 may be electroencephalogram information processed for each individual driver in the app 130_1 of the mobile terminal 130, and the processing of the electroencephalogram signal for each individual driver will be described in detail below.
The server 140 according to the present embodiment may generate the carelessness alarm information converted into the big data based on the driving information and the electroencephalogram signal information received as described above (S230), and provide the carelessness alarm information generated as such to the plurality of mobilities or the subscribers (S240).
The carelessness alarm information may include at least one of information on in which moving path or location the carelessness occurs, information on in what time period the carelessness occurs, and information on under what weather the carelessness occurs. Rather than being generated at a level of the individual mobile terminal 130 or mobility, such information is generated at the end of the server 140 based on information on the plurality of mobilities, thereby ensuring the objectivity and providing the alarm on the carelessness in more diverse aspects.
First, in the example of configuration of the system shown in
The information collection end 310 may include deep learning-based driver state monitoring (DSM) information, GPS information, and electroencephalogram (EEG) information as source data 311.
In addition, the information collection end 310 may include information acquired via the app 130_1, which may include driver information of the bus company and input information 313 of each driver. However, the driver information of the bus company and the input information 313 of each driver do not necessarily need to be input via the app 130_1 of the mobile terminal 130, and the app 130_1 is able to be installed on various computing devices to provide corresponding reference information.
In addition, the information collection end 310 may include means 314 for acquiring base information (e.g., Gyeonggi-do bus line information, bus line stop information, and vehicle information), weather information, and the like in a corresponding area in addition to the above-described information.
At the server end 320, as described above, the method may be largely divided into collecting the various information from the information collection end 310 (S321), analyzing the information collected as such (S322), and expressing and providing the alarm information to the user or a computing device of the bus company using the analyzed result (S323).
The user end 330 may include individual drivers 331 and a computing device dashboard of a bus company 332, as shown in
As described above, the subscriber information may be divided into subscription information 313_1 of each transportation company, such as the bus company, and input information 313_2 of each driver.
As shown in
Information necessary for the administrator may be displayed as shown in
As shown in
The driver information input as such may be displayed on an administrator screen 700 as shown in
A carelessness occurring section/frequency may be calculated for each line via the information such as the operating bus line 620 and the vehicle number 630 among the input information 313_2 of each driver described above. Additionally, statistics on a degree of carelessness based on an age/gender of the driver may be provided using additional personal information of the input information 313_2 of each driver.
First,
To examine a specific line in detail while displaying carelessness sections in a form of a map for the plurality of lines, the specific line may be selected and carelessness section information of the corresponding line may be displayed in an enlarged manner (840).
The carelessness alarm information as described above are illustrative, and depending on embodiments of the present disclosure, the carelessness alarm information is able to include various information as shown in <Table 1> below.
The driver assistance system according to one embodiment of the present disclosure may measure the electroencephalogram signal of the driver via the wearable device 110. The electroencephalogram (EEG) is the phenomenon in which the thoughts, the emotions, and the behaviors of the individual are generated by the communication between the nerve cells in the brain, and is the synchronized electrical wave generated when the nerve cells in the cerebral cortex transmit the signals to each other. As described above, the electroencephalogram appears in the various frequency bands, and is able to be specifically classified into the delta wave, the theta wave, the alpha wave, the beta wave, the gamma wave, and the like based on the frequency bands.
In this regard, the wearable device 110 may include the earset that is mounted around the ear of the driver and is able to measure the electroencephalogram signal around the left or right temporal lobe of the driver while driving. In another embodiment, the wearable device 110 may include the headset using the multi-channel electrode.
In addition, in the driver assistance system 100 according to one embodiment of the present disclosure, the wearable device 110 may include the various sensors or equipment that may measure the electroencephalogram signal of the driver in real time, including the earset and the headset described above.
The electrode 111 may measure the electroencephalogram signal of the driver during the driving and/or the stopping of the driver in the vehicle 140 via the electrode. In this regard, the electrode 111 may include the EEG electrode, the REF electrode, and the GND electrode. The EEG electrode is the conductive electroencephalogram electrode, and is able to measure the electroencephalogram signal of the left or right temporal lobe by being in contact with the area near the left or right temporal lobe temple of the driver. The REF electrode is the reference electrode, and is able to provide the reference potential when the EEG electrode measures the electroencephalogram signal. The GND electrode is the ground electrode, is located on the back of the earflap, and is able to serve as the ground when the EEG electrode measures the electroencephalogram signal.
In one embodiment, the filter 112 may remove the noise from the electroencephalogram signal of the driver measured by the electrode 111. The filter 112 may remove the noise from the electroencephalogram signal using the high pass filter, the low pass filter, or the band pass filter. As another embodiment, the filter such as the notch filter or the active filter may be used. In this regard, the noise may include the noise generated by the eye-blink of the driver, the AC power noise, and the DC drift component as well as all the elements that interfere with the feature point extraction.
The converter 113 may convert the noise-removed electroencephalogram signal of the driver into the digital signal. The signal processor 114 may calculate one or more characteristics of the electroencephalogram from the electroencephalogram signal of the driver measured by the wearable device 110. The characteristics of the electroencephalogram calculated by the signal processor 114 may be, for example, at least one of the magnitude of the electroencephalogram signal in the time domain, the magnitude of the electroencephalogram signal in the frequency domain, and the magnitude of the signal-to-noise ratio (SNR) peak.
The storage 115 may store the reference electroencephalogram signal, that is, the preset alertness value, for determining the careless state of the driver based on the characteristics of the electroencephalogram. In this regard, the careless state of the driver may include the state in which the driver is not able to concentrate on the driving, including the abnormal states such as the drowsiness, the reduced concentration, and the surprise. In addition, the storage 115 may store the electroencephalogram signal of the driver measured via the wearable device 110 for a certain period of time. Such storage 115 may include the internal memory and/or the external memory, and may include the volatile memory such as the DRAM, the SRAM, or the SDRAM, the non-volatile memory such as the one time programmable ROM (OTPROM), the PROM, the EPROM, the EEPROM, the mask ROM, the flash ROM, the NAND flash memory, or the NOR flash memory, and the storage device such as the SSD, the compact flash (CF) card, the SD card, the micro-SD card, the mini-SD card, the XD card, the flash drive such as the memory stick, or the HDD.
The communicator 116 is the medium for transmitting and receiving the information between the wearable device 110 and the mobile terminal or the vehicle, and is able to transmit the electroencephalogram signal of the driver measured by the wearable device 110 to the mobile terminal or the vehicle via the wireless communication. In the present embodiment, the communicator 116 may transmit and receive the data with the mobile terminal or the vehicle via at least one communication among the Bluetooth, the radio frequency identification (RFID), the infrared data association (IrDA), the ultra-wideband (UWB), the ZigBee, and the wireless fidelity (Wi-Fi) technologies.
The power supply 117 may supply the corresponding power throughout the wearable device 110. Specifically, the power supply 117 may include the converter that converts the AC power to the DC power and the DC/Dc converter that converts the level of the DC power. The power supply 117 may use the scheme of supplying the AC power via the direct connection to the external power source, and may include the power supply 117 that includes the battery to be charged and used.
In the case of the former, the cable is connected to the power supply 117, so that it is difficult for the power supply 117 to move or the range of movement thereof is limited. In the case of the latter, the power supply 117 is free to move, but the weight and the volume thereof increase as much as those of the battery, and the power supply 117 must be directly connected to the power cable or coupled to the charging holder that supplies power for a certain period of time for the charging. The charging holder may be connected to the wearable device 110 via the terminal thereof exposed to the outside, or the built-in battery may be charged when approaching the charging holder using the wireless scheme.
First, (A) in
In a case of a bus driver in a country where the driver's seat is generally located on a left side, such as Korea, when the driver is wearing the wearable device 110 for measuring the electroencephalogram on the right ear as shown in (A) in
However, the above description applies to the country where the driver's seat is located on the left side, such as Korea. On the other hand, in countries where the driver's seat is located on a right side, such as Japan/England, the wearable device that is worn on the right ear of the driver as shown in (A) in
On the other hand, the wearable device 110 according to the embodiment shown in (A) and (B) in
Therefore, in a preferred embodiment of the present disclosure, as shown in (C) in
In other words, the functional member 1320 is constructed to include a component that is heavy and does not necessarily need to be in contact with the driver among the components of the wearable device 110, and to be fixed to clothes of the driver in a form of a clip, thereby reducing the weight of the wearable device 110 while maintaining high performance of the wearable device 110.
For example, the measurer 1310 may be constructed to include the electrode 111, the filter 112, the converter 113, and the signal processor 114 among the components described above in
For measuring the electroencephalogram signal of the driver via the wearable device 110, measuring the electroencephalogram of the driver for each waveform (e.g., theta, SMR, and mid-beta in frequency ranges indicated at top of
To this end, in the embodiment shown in
The predetermined time period is a time period for extracting the electroencephalogram characteristics for each driver using the electroencephalogram during initial travel of the driver, and is assumed to be set to 5 minutes. After collecting electroencephalogram data of the driver for the first 5 minutes as such (S1520), statistics on carelessness numeric values of the corresponding individual driver may be calculated based on the collected data (S1530). For example, such statistics on the carelessness numeric values of each individual may be calculated as a distribution as shown at bottom of
When the degree of carelessness of each driver is determined using the mean (μ) and the standard deviation (σ) as described above, an alarm may be provided on the degree of carelessness of the corresponding driver in consideration of individual characteristics of the driver.
In the preferred embodiment of the present disclosure, it is proposed to additionally consider a travel state in determining the degree of carelessness of the driver. As an example, in
When the corresponding mobility is traveling on the highway, whether to provide the carelessness alarm may be determined based on a first reference value (S1550). For example, when the mobility is traveling on the highway, and when the alertness score is equal to or smaller than μ−1.96σ (the first reference value), it may be set to provide the carelessness alarm to the corresponding driver, which may correspond to a 95% confidence interval in the distribution shown at the bottom of
On the other hand, when the mobility is traveling on a general road, and when the alertness score is equal to or smaller than μ−2.58σ (a second reference value), it may be set to provide the carelessness alarm to the corresponding driver, which may correspond to 99% confidence interval in the distribution shown at the bottom of
In the above-described embodiment, it is desirable to set the carelessness alarm not to be provided for 5 minutes set to identify the electroencephalogram characteristics of each driver at start of each travel. Additionally, when the driver is not properly wearing the wearable device, a separate alarm may be provided, but it is desirable to set the carelessness alarm not to be activated in such situation.
Although the example of recording/saving the electroencephalogram characteristics of the specific driver for 5 minutes after the start of travel is illustrated in the above-described embodiment, the electroencephalogram characteristics of the driver may also be used in a scheme of being managed by the server by calculating the mean and the standard deviation in a 24-hour period.
On a left side of
On the other hand, a right side of
In addition, in the above-described embodiments of the present disclosure, the description has been made centering on the method for providing the carelessness alarm to the driver based on the electroencephalogram, but in another embodiment of the present disclosure, such an electroencephalogram-based carelessness alarm method and an image-based carelessness alarm method may be used in combination with each other.
The electroencephalogram-based carelessness alarm method has an advantage of being able to quickly determine the careless state of the driver and provide the alarm compared to the image-based carelessness alarm method. For example, the image-based carelessness alarm method may provide the alarm related to the carelessness of the driver via a change in an image of driver's actual eyelids closing when the driver is drowsy, whereas the electroencephalogram-based carelessness alarm method has an advantage of providing the alarm for the carelessness situation via the electroencephalogram of the driver before the actual eyelids are closed.
However, in a situation where the driver is not drowsy and has concentration, but is not looking forward, the image-based carelessness alarm method may have an advantage over the electroencephalogram-based carelessness alarm method. Therefore, in the embodiment of the present disclosure, it may also be set that an alarm for the situation in which the driver is not looing forward is provided based on the image in addition to the above-described carelessness alarm based on the electroencephalogram of the driver.
The above detailed description is to be construed in all aspects as illustrative and not restrictive. The scope of the present disclosure should be determined by reasonable interpretation of the appended claims and all changes coming within the equivalency range of the present disclosure are intended to be embraced in the scope of the present disclosure.
The electroencephalogram-based driver assistance system and assistance method according to the embodiments of the present disclosure as described above may be utilized by being applied to the vehicle that regularly operates on the determined moving path, such as the bus and a truck. Additionally, such concept does not necessarily have to be limited to being applied only to the vehicle, and is able to be applied to various mobilities.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2023-0001311 | Jan 2023 | KR | national |
| 10-2023-0090408 | Jul 2023 | KR | national |
