1. Field of the Invention
The present invention relates to a technique of providing safe driving assistance by determining a state of a driver by utilizing an electroencephalogram.
2. Description of the Related Art
In recent years, in connection with accident prevention apparatuses related to automobile driving, methods of determining, the state of a driver and providing driving assistance based on the result of determination are being studied. One visual perception function of a driver that is necessary for safe driving is detection of dangerous objects. Detection of dangerous objects involves noticing any dangerous motion of vehicles and pedestrians in the surroundings by peripheral vision. A deterioration in this detection function may lead to cross-collision accidents and rush-out accidents.
A “peripheral visual field” generally refers to a region spanning 130° in up and down directions and 180° in right and left directions, excluding a range of about 20° (central visual field) that is centered around a line of sight. As is known, in the peripheral visual field, it is difficult to recognize the shape and color of an object in detail, but sensitive response occurs with respect to any object that changes in time, e.g., a moving object or flickering light. In anticipation of a rushing out of a pedestrian or a motorcycle passing on the side, etc., a driver needs to pay attention to the peripheral visual field and any door mirrors or the like existing in this field. Therefore, when the amount of attention of the driver to the peripheral visual field becomes low, a remedy such as issuing an alarm to the driver is needed.
One method of determining the state of attention of a driver employs a camera which is aimed at the driver for detecting the line of sight and motions of the face of the driver, and determines an allocation of attention of the driver. For example, Japanese Laid-Open Patent Publication No. 2004-178367 (hereinafter referred to as “Patent Document 1”) discloses a technique of determining the attention allocation of a driver by comparing a fixation point, which is detected from the line of sight and motions of the face of the driver, against an optimum fixation position that the driver should pay attention to, which is determined from the ambient situation of the driver's vehicle.
Another method determines the state of attention of a driver based on changes in the traveling velocity and the steering angle of the steering wheel and the like, which reflect the manner in which the driver's vehicle is being operated. For example, Japanese Laid-Open Patent Publication No. 2002-127780 (hereinafter referred to as “Patent Document 2”) discloses a technique which determines a driver's degree of concentration on driving by using a brake response time with respect to a sudden deceleration of a preceding vehicle or the like, thus determining the level of need to output an alarm to the driver.
On the other hand, studies are under way to examine the amount of attention of a driver to driving by utilizing an event-related potential (ERP) of his or her electroencephalogram. An “event-related potential” refers to a transient potential fluctuation in the brain which occurs in temporal relationship with an external or internal event. Within an event-related potential, a positive component which appears near about 300 milliseconds based on the timing of an external visual stimulation or the like as a starting point is referred to as a P300 component, which is supposed to reflect perception of, or attention to, that stimulation.
For example, “Technique for Measuring Driver's Attention Level by Using Event-Related Potentials”, Ebe et al., Automotive Technologies, Vol. 58, No. 7, pp. 91-96, 2004 (hereinafter referred to as “Non-Patent Document 1”) discloses a study concerning the measurement of an amount of driving attention by utilizing an event-related potential. To specifically describe this study, in an experiment of trying to drive a vehicle so as to follow a preceding vehicle, the driver is asked to perform a task of stepping on a brake pedal of the driver's vehicle when brake lamps of the preceding vehicle are activated. Through a comparison of event-related potentials between the two experimental conditions of a travel (high-attention condition) during which the preceding vehicle applies sudden brakes and a travel during which this does not occur (low-attention condition), it reports that the amplitude of the P300 component of the event-related potential increases under the high-attention condition.
However, the technique described in Patent Document 1 is based on the assumption that attention is not being paid to anywhere that the line of sight is not directed, and therefore cannot accurately determine the amount of attention of the peripheral visual field of the driver. For example, in an actual driving scene, while monitoring a preceding vehicle by central vision, the driver is simultaneously detecting the motions of flanking vehicles and pedestrians by peripheral vision, and determines the direction of his or her line of sight based on the situations of the front and the surroundings. Therefore, with the conventional technique, it is difficult to cope with the case where the line of sight is being directed to the front while also paying attention to the peripheral visual field, for example.
Moreover, in the technique described in Patent Document 2, since a brake response time with respect to a sudden deceleration of a preceding vehicle or the like is used, the derived degree of concentration on driving is limited to the front of the driver's vehicle, i.e., the central visual field of the driver. In an actual driving scene, it is very rarely the case that a response to an event occurring in the peripheral visual field of a driver is straightforwardly manifested in behavior such as braking. Therefore, with the conventional technique utilizing the manner in which the driver's vehicle is operated, the amount of attention of the driver to the peripheral visual field cannot be determined with a good accuracy.
Furthermore, in a study which is described in Non-Patent Document 1, an event-related potential (ERP) with respect to activation of the brake lamps of a preceding vehicle is similarly used. Therefore, the amount of driving attention being measured is limited to that pertaining to the central visual field of the driver, and it is impossible to measure the amount of attention to the peripheral visual field.
The present invention has been made in view of the aforementioned problems, and an objective thereof is to, even when a driver is not directing his or her line of sight to objects in the surroundings, determine the amount of attention of the driver to the peripheral visual field and provide safe driving assistance in accordance with the result of determination.
A driving attention amount determination apparatus according to the present invention comprises: an electroencephalogram measurement section for measuring an electroencephalogram signal of a driver; an attention amount determination section for determining an amount of attention of the driver to a peripheral visual field by utilizing an event-related potential in the electroencephalogram signal, the event-related potential being based on a starting point which is a time point of occurrence of a visual stimulation occurring in the peripheral visual field of the driver; and an output section for calling attention of the driver by outputting a signal based on a result of the determination.
The attention amount determination section may determine the amount of attention in accordance with an amplitude level of the event-related potential of the electroencephalogram signal based on a starting point which is the time point of occurrence of the visual stimulation.
The attention amount determination section may determine that the amount of attention is small if an amplitude of a P300 component of the event-related potential is smaller than a predetermined threshold value, the P300 component being a positive component in a zone from 300 milliseconds to 600 milliseconds based on a starting point which is the time point of occurrence of the visual stimulation.
The output section may output the signal to the driver when the attention amount determination section determines that the amount of attention is small.
The attention amount determination section may determine that the amount of attention is large if the amplitude of the P300 component of the event-related potential is greater than the predetermined threshold value; and when it is determined that the amount of attention is large, the output section may not output the signal to the driver.
The attention amount determination section may determine the amount of attention in accordance with a correlation coefficient between a prestored template and the electroencephalogram signal measured based on a starting point which is the time point of occurrence of the visual stimulation.
The output section may output at least one of: a video signal for presenting a text or a symbol on a screen for presenting information; and an audio signal to be output from a loudspeaker for outputting an audio.
The driving attention amount determination apparatus may further comprise a peripheral stimulation generation section for generating the visual stimulation in the peripheral visual field of the driver.
The driving attention amount determination apparatus may further comprise: an imaging section for capturing a video of a front of a vehicle being driven by the driver; and a peripheral stimulation detection section for, from the captured video, detecting the time point of occurrence of the visual stimulation occurring in the peripheral visual field, wherein, from the peripheral stimulation detection section, the attention amount determination section may receive information identifying the detected time point of occurrence of the visual stimulation.
The driving attention amount determination apparatus may further comprise a line-of-sight measurement section for measuring a line of sight of the driver, wherein the peripheral stimulation detection section may detect whether the visual stimulation has occurred in the peripheral visual field or not in accordance with the captured video and the line of sight of the driver at the time point of occurrence of the visual stimulation measured by the line-of-sight measurement section.
The driving attention amount determination apparatus may further comprise a situation detection section for detecting a velocity or head lamp activation of the vehicle, wherein, in accordance with a result of detection by the situation detection section, the peripheral stimulation detection section ma detect whether the visual stimulation is in the peripheral visual field or not.
If a difference in timing of occurrence between a visual stimulation detected in the peripheral visual field and a visual stimulation detected in the central visual field is equal to or less than a predetermined value, the attention amount determination section may exclude, from the subject of analysis, an event-related potential of the electroencephalogram signal with respect to the visual stimulation detected in the peripheral visual field.
The peripheral stimulation generation section may cause the visual stimulation in the peripheral visual field of the driver to be generated with a timing of occurrence having a difference from a timing of occurrence of the visual stimulation occurring in the central visual field of the driver, the difference being equal to or greater than a predetermined value.
A method of determining an amount of driving attention according to the present invention comprises the steps of: measuring an electroencephalogram signal of a driver; determining an amount of attention of the driver to a peripheral visual field by utilizing an event-related potential in the electroencephalogram signal, the event-related potential being based on a starting point which is a time point of occurrence of a visual stimulation occurring in the peripheral visual field of the driver; and calling attention of the driver by outputting a signal based on a result of the determination.
A computer program for determining an amount of driving attention according to the present invention, when executed by a computer, causes the computer to execute the steps of: receiving an electroencephalogram signal of driver; determining an amount of attention of the driver to a peripheral visual field by utilizing an event-related potential in the electroencephalogram signal, the event-related potential being based on a starting point which is a time point of occurrence of a visual stimulation occurring in the peripheral visual field of the driver; and outputting a signal based on a result of the determination, thereby calling attention of the driver.
Alternatively, a driving attention amount determination apparatus according to the present invention comprises: an electroencephalogram measurement section for measuring an electroencephalogram signal of a driver; an imaging section for capturing a video of a front of a vehicle being driven by the driver; a peripheral stimulation detection section for, from the captured video, detecting a time point of occurrence of a visual stimulation occurring in a peripheral visual field of the driver, an attention amount determination section for determining an amount of attention of the driver to a peripheral visual field by utilizing an event-related potential in the electroencephalogram signal, the event-related potential being based on a starting point which is a time point of occurrence of a visual stimulation occurring in the peripheral visual field of the driver; and an output section for calling attention of the driver by outputting a signal based on a result of the determination, wherein, from the peripheral stimulation detection section, the attention amount determination section receives information identifying the detected time point of occurrence of the visual stimulation.
Alternatively, a driving attention amount determination apparatus according to the present invention comprises: an electroencephalogram measurement section for measuring an electroencephalogram signal of a driver; a peripheral stimulation generation section for generating a visual stimulation in a peripheral visual field of the driver, and outputting information identifying a time point of occurrence of the visual stimulation; an attention amount determination section for receiving the information from the peripheral stimulation generation section, and for determining an amount of attention of the driver to a peripheral visual field by utilizing an event-related potential in the electroencephalogram signal, the event-related potential being based on a starting point which is a time point of occurrence of the visual stimulation identified by the information; and an output section for calling attention of the driver by outputting a signal based on a result of the determination.
According to the present invention, from an electroencephalogram signal measured based on a starting point which is the time point of occurrence of a visual stimulation occurring in a peripheral visual field of a driver, an amount of attention of the driver to the peripheral visual field is determined. By using the electroencephalogram signal, it becomes possible to accurately determine an amount of attention to events that may possibly occur in the peripheral visual field of the driver (e.g., a sudden intrusion of a vehicle or a rushing out of a pedestrian), and based on the result of determination, a change in the state of the driver can be appropriately induced, e.g., by attention calling.
Other features, elements, processes, steps, characteristics and advantages of the present invention will become more apparent from the following detailed description of preferred embodiments of the present invention with reference to the attached drawings.
Hereinafter, with reference to the attached drawings, a concept of a driving attention amount determination apparatus according to the present invention will be described, followed by descriptions of Embodiments.
The electroencephalogram measurement section 11 measures an electroencephalogram signal from the driver 10.
The attention amount determination section 13 determines an amount of attention of the driver 10 to the peripheral visual field, from an electroencephalogram signal which is measured based on a starting point which is the time point (or temporal point) of occurrence of a visual stimulation, the visual stimulation occurring in the peripheral visual field of the driver 10.
As used herein, the “peripheral visual field” refers to a region of the visual field of a human excluding a certain visual field (central visual field) which is centered around the direction of a line of sight of the human. When contemplating a cone whose axis is the gaze direction of a human, the central visual field can be defined as a region which is contained within a certain angle constituted by the side face of this cone and the direction of the line of sight. In the following Embodiments, this certain angle is assumed to be about 20°.
The output section 14 outputs a signal based on the result of determination by the attention amount determination section 13, thus calling attention of the driver 10. As a result, the amount of attention of the driver is improved, thereby providing assistance in safe driving.
The aforementioned attention amount determination section 13 identifies the temporal point of occurrence of a visual stimulation occurring in the peripheral visual field. The visual stimulation may be provided by causing a light emission device which is provided in the driving attention amount determination apparatus 100 to emit light, or may be provided from the external environment (e.g., a lamp activated on another car). The following Embodiments will describe both cases.
Note that the specific construction of the electroencephalogram measurement section 11, the attention amount determination section 13, and the output section 14 will be described in more detail in the Embodiments.
The driving attention amount determination apparatus 1 is an apparatus which, by utilizing an electroencephalogram signal from the driver 10, determines an amount of attention to driving and provides assistance in accordance with the result of determination. For example, an amount of attention to events that may possibly occur in the peripheral visual field of a driver (e.g., a sudden intrusion of a vehicle or a rushing out of a pedestrian) is determined by utilizing an electroencephalogram, and in accordance with the result of determination, attention of the driver is called.
The driving attention amount determination apparatus 1 includes an electroencephalogram measurement section 11, a peripheral stimulation generation section 12, an attention amount determination section 13, and an output section 14. The driver 10 block is illustrated for convenience of explanation.
The outline of the hardware construction and functions of each constituent element is as follows.
The electroencephalogram measurement section 11, which is an electroencephalograph for example, measures an electroencephalogram of the driver 10.
The peripheral stimulation generation section 12, which is composed of LED light sources and a control circuit therefor, for example, generates a visual stimulation in the peripheral visual field of the driver 10. The peripheral stimulation generation section 12 gives the visual stimulation to the driver 10, and transmits information representing the timing of stimulation generation to the attention amount determination section 13.
The attention amount determination section 13, which is a microcomputer, for example, measures an electroencephalogram signal based on a starting point which is the temporal point of occurrence of the stimulation. The temporal point of occurrence of the stimulation is identified based on the information representing the timing of stimulation generation. From this electroencephalogram signal, the attention amount determination section 13 determines the amount of attention of the driver 10 to the peripheral visual field.
The output section 14 is a device which is capable of outputting at least one of an image and an audio. An image will be output by utilizing a display device such as a liquid crystal display device or a so-called organic EL display. An audio will be output by using a loudspeaker. The output section 14 calls attention of the driver 10 based on the result of determination.
Hereinafter, each constituent element will be specifically described.
By measuring potential changes at electrodes which are worn on the head of the driver 10, the electroencephalogram measurement section 11 detects an electroencephalogram signal. The inventors envisage that a wearable-type electroencephalograph will be used in future. Therefore, the electroencephalograph may be a head-mount type electroencephalograph. It is assumed that the driver 10 has worn the electroencephalograph in advance.
Electrodes are disposed on the electroencephalogram measurement section 11 so that, when worn on the head of the driver 10, the electrodes come in contact with the head at predetermined positions. The positions of the electrodes may be, for example, Pz (median parietal), A1 (earlobe), and the nasion as defined under the International 10-20 system. According to previous literature (Yo MIYATA et al., “New Physiopsychology”, 1998, p. 119, Kitaoji Shobo), a P300 component of an event-related potential, which reflects perception of or attention to an external stimulation and appears near about 300 milliseconds based on the timing of occurrence of the stimulation as a starting point, is supposed to rise to its maximum amplitude at Pz (median parietal). However, P300 component measurement is also possible at Cz (epicranium) or Oz (occiput), which are in the neighborhood of Pz, and therefore electrodes may be disposed in these positions. These electrode positions are to be determined based on reliability of signal measurements, wearing ease, and the like.
As a result of this, the electroencephalogram measurement section 11 is able to measure an electroencephalogram of the driver 10. The measured electroencephalogram is sampled so as to be computer-processable, and is sent to the peripheral visual field attention amount determination section 13. Note that, in order to reduce the influence of noises mixing into the electroencephalogram, when the event-related potential is at issue, the electroencephalogram measured by the electroencephalogram measurement section 11 is previously subjected to a 15 Hz low-pass filtering process, for example.
The peripheral stimulation generation section 12 generates a visual stimulation in the peripheral visual field of a driver. Now, the definition of the peripheral visual field will be described by way of example.
For example, in the case of a glasses-type head-mount display (Head Mounted Display: HMD) which combines a wearable-type electroencephalograph and a display as shown in
As for the flickers serving as the visual stimulation, the number of flickers per unit time is determined from the determination accuracy, interval of determination, and the like of the attention amount determination section 13 described later. For example, in the case where a change in the amount of attention is to be determined every 3 minutes, if the number of electroencephalogram data that is required for determination (number of summations) is 30, then the number of flickers is 10 times per minute.
By combining various anti-noise measures and highly accurate analysis methods that are currently employed in the studies of the event-related potential (ERP) of electroencephalograms, the required number of flickers can be further reduced. The positions to be flickered may be randomly determined, or consecutive flickers may occur in a previously-determined order. The light sources 23 are to be placed in the peripheral visual field as viewed from the driver. Moreover, the light sources 23 (e.g., LEDs) may be positioned at edge portions of the windshield inside the vehicle, or on the door mirrors; in this case, too, the light sources 23 are to be placed in the peripheral visual field as viewed from the driver.
Moreover, the peripheral stimulation generation section 12 must generate a visual stimulation in the peripheral visual field at a different timing from any visual stimulation occurring in the central visual field of the driver. A specific example thereof will be described below.
First,
In the case where the driver flickers a winker when changing the direction of travel or changing the lane, for example, a difference must be intentionally introduced between the timing of flickering a winker indication of the front panel existing in the central visual field 31 of the driver and the timing of flickering the light sources 23 existing in the peripheral visual field 32, which are disposed at edge portions of the windshield inside the vehicle, on the door mirrors, or the like.
The reason is as follows. The attention amount determination section 13 (described later) determines an amount of attention by using an event-related potential of the electroencephalogram based on a starting point which is the temporal point of occurrence of a stimulation, in particular, an event-related potential from 300 milliseconds to 600 milliseconds based on the temporal point of occurrence of a stimulation as a starting point. If visual stimulations simultaneously occur in both of the central visual field and the peripheral visual field, it is impossible to identify whether an amount of attention determined by the attention amount determination section 13 pertains to the central visual field 31 or to the peripheral visual field 32. Therefore, in order to ensure that the analysis time zones for the respective stimulations do not overlap each other, it is necessary for the peripheral visual field 32 to generate a visual stimulation with a predetermined time difference from any visual stimulation occurring in the central visual field 31.
In the case of introducing a time difference between a visual stimulation in the peripheral visual field 32 and a visual stimulation occurring in the central visual field 31, in order to be able to identify which one of the visual stimulations has induced the event-related potential (from 300 milliseconds to 600 milliseconds) that is currently at issue, it is necessary for the visual stimulation to be generated with a time difference of at least 300 milliseconds from the other visual stimulation. For example, when a winker on the front panel flickers every 600 milliseconds, the light sources 23 in the peripheral visual field may be flickered similarly every 600 milliseconds with a timing which is shifted by 300 milliseconds from the visual stimulation on the front panel.
Moreover, if visual stimulations are simultaneously detected in both of the central visual field 31 and the peripheral visual field 32, or if the difference between the detection timings of the respective stimulations is 300 milliseconds or less, the amount of attention to the peripheral visual field 32 cannot be correctly measured as described above. In this case, the data of the event-related potential with respect to the stimulation generated in the peripheral visual field 32 is excluded from the subject of analysis by the attention amount determination section 13. This process may be realized by the attention amount determination section 13 discarding the data without using it, or by the electroencephalogram measurement section 11 stopping the output of the electroencephalogram signal at that point in time.
Note that, generally speaking, a “peripheral visual field” refers to a region spanning 130° in up and down directions and 180° in right and left directions, excluding a range of about 20° (central visual field) that is centered around a line of sight (fixation point). Therefore, when providing a line-of-sight measurement section for measuring the line of sight of a driver, as shown in
In the following description, it is assumed that the central visual field and the peripheral visual field are defined as shown in
However, in actual driving, the line of sight of the driver may fluctuate. Therefore, an instance of measuring the line of sight of the driver will be described later in another Embodiment. Moreover, an instance of utilizing an external visual stimulation will also be described later in another Embodiment.
The peripheral stimulation generation section 12 transmits information representing the point in time, or timing of occurrence (trigger), of the aforementioned stimulation to the attention amount determination section 13.
Based on the information received from the peripheral stimulation generation section 12, the attention amount determination section 13 analyzes the measured electroencephalogram signal based on a starting point which is the temporal point of occurrence of a stimulation, thereby determining an amount of attention of the driver 10 to the peripheral visual field. Now, with reference to
At step S51 in
At step S52, from the peripheral stimulation generation section 12, the attention amount determination section 13 receives information of the point in time at which a stimulation occurred.
At step S53, in the electroencephalogram data received at step S51, the attention amount determination section 13 cuts out electroencephalogram data from −100 milliseconds to 600 milliseconds, based on each point in time of occurrence acquired at step S52 as a starting point.
At step S54, the attention amount determination section 13 applies a baseline correction to the electroencephalogram data having been cut out. For example, a baseline correction is performed with respect to an average potential from −100 milliseconds to 0 milliseconds, based on the point in time at which the stimulation occurred as a starting point.
At step S55, the attention amount determination section 13 temporarily stores the electroencephalogram data which has been subjected to the baseline correction at step S54.
At step S56, the attention amount determination section 13 determines whether the number of electroencephalogram data stored at step S55 has reached a predetermined, necessary number of summations to be made. If this value is not reached, the process returns to S51; if this value is reached, the process proceeds to S57.
Note that, in the studies of event-related potentials in general, an analysis is performed after deriving an arithmetic mean of electroencephalogram data. As a result, random action potentials of the brain that are not related to the event which is at issue are counteracted, thus making it possible to detect an event-related potential (e.g., a P300 component) that has a certain latency (i.e., the amount of time in which an action potential occurs since the starting point which is the temporal point of occurrence of a stimulation) and polarity.
For example, according to previous literature (Yo MIYATA et al., “New Physiopsychology, 1998, p. 110, Kitaoji Shobo), a process of taking 30 arithmetic means is performed.
In the present embodiment, the number of summations is 20 to 30, for example. By increasing this number, the SN ratio can be improved. However, this number of summations is only exemplary, and the present invention is not limited thereto. The amount of attention may be determined from a non-cumulative electroencephalogram (i.e., a single piece of electroencephalogram data).
At step S57, the attention amount determination section 13 performs an arithmetic mean process for the electroencephalogram data from the necessary number of times stored at step S55.
Furthermore, from the electroencephalogram data after taking the arithmetic mean, the amplitude of the event-related potential from 300 milliseconds to 600 milliseconds is analyzed, and based on whether the amplitude is large or small, a determination of an amount of attention is performed. At this time, based on ERP characteristics which are unique to the peripheral visual field identified by the inventors, the amount of attention to the peripheral visual field is determined. The details of the determination process will be described later with reference to the experimental results shown in
Now, the relationship between the range of electroencephalogram data whose sum is taken and the range of amount-of-attention determination is explained. For example, in the case where a sum of the electroencephalogram data for the flickering of all of the light sources 23 shown in
At step S58 in
The output section 14 presents a result of determination by the attention amount determination section 13 in the form of an image or an audio. Alternatively, based on the result of determination, the output section 14 may output a signal for calling attention of the driver from the apparatus side when the amount of attention is low. As a result, the amount of attention of the driver can be improved.
The signal(s) which is output from the output section 14 in order to call attention of the driver may be either one or both of a video signal and an audio signal, for example. Specifically, the driver may be spoken to with an audio signal, or an operating noise or an alarm sound may be presented with an audio signal; or with a video signal, a text or image presentation may be made on a car navigation system or a head-up display. As a result, it is possible to achieve an attention calling for improving the amount of attention of the driver.
The signals which the output section 14 outputs in order to call attention may include any control signal that causes an action for calling attention of the driver. Examples thereof include: a control signal for a direct information presentation using an AR (Augmented Reality) technique, such that an image is displayed in overlay on an object which needs attention; a control signal for causing an indirect intervention through a vibration of the steering wheel, a smell, or adjustment of an amount of fanned air; and the like. The actions for calling attention may include various actions such as the aforementioned examples. Any of the aforementioned examples can be considered as calling attention by exerting an external action to the driver.
For example,
Now, results of an experiment performed by the inventors in connection with the aforementioned determination of an amount of attention will be described. Through an experiment described below, the inventors have found characteristics such that the amplitude of an event-related potential with respect to a stimulation occurring in the peripheral visual field from 300 milliseconds to 600 milliseconds greatly changes depending on whether the amount of attention is large or small.
A total of 4 test subjects were involved, including one male and three females, with an average age of 21±1.5 years. The experimental details will be described with reference to
The inventors performed the experiment by a dual task method, in which each test subject was asked to perform two tasks concurrently. The first task was a central task 71 of counting to oneself a number of times that symbols (◯/Δ/□/X) presented on a screen center in
Each test subject was wearing an electroencephalograph (Polymate AP-1124 manufactured by TEAC Corporation), and the electrodes were positioned according to the International 10-20 electrode system, with a recording electrode at Pz (median parietal), a reference electrode at A1 (right earlobe), and a ground electrode at the metopic. Electroencephalogram data which was measured with a sampling frequency of 200 Hz and a time constant of 3 seconds were subjected to a bandpass filtering process from 1 to 6 Hz, and electroencephalogram data from −100 milliseconds to 600 milliseconds was cut out based on the flickering of a peripheral lamp as a starting point, and a baseline correction was performed with respect to an average potential from −100 milliseconds to 0 milliseconds.
The first condition concerns a classification with respect to the visual fields. In this experiment, the classification was made as shown in
The second condition concerns a classification with respect to each test subject's response time regarding a button press. In this experiment, in order to classify the amount of attention (large or small) as an experimental condition, a response time before achieving a button press was used. In physiopsychological experiments, response time is supposed to reflect the amount of attention; for example, in Patent Document 2, too, a degree of concentration of attention to driving is calculated by using a brake response time.
In this experiment, a relationship between an electroencephalogram and an amount of attention was analyzed, where the amount of attention served as an index of button-press response time. Among all response times in this experiment, very many samples were found between 400 milliseconds and 600 milliseconds. Therefore, a classification was made between: cases where a response was attained within 600 milliseconds (fast response time, i.e., a state of high attention to the stimulation); and cases where a response was not attained within 600 milliseconds (slow response time, i.e., a state of low attention to the stimulation). In each of the graphs in
Through determining the amplitude level of an event-related potential by utilizing the aforementioned ERP characteristics in the peripheral visual field, it becomes possible to accurately determine an amount of attention to an event that may occur in the peripheral visual field of a driver, e.g., a sudden intrusion of a vehicle or a rushing out of a pedestrian, based on an electroencephalogram.
Furthermore, advantages of the construction according to the present embodiment will be specifically described based on results of a trial calculation of an attention amount distinction rate in this experiment.
In the method of distinction, a threshold value of ERP maximum amplitude that maximizes the distinction rate in each visual field is chosen, and a determination as to whether the amount of attention is large or small is made based on whether or not the ERP amplitude of each non-cumulative electroencephalogram is equal to or greater than this threshold value. As the threshold value that maximizes the distinction rates, a threshold value is chosen at which a largest average between the correctness rate of the case where the amount of attention is large and the correctness rate of the case where the amount of attention is small is obtained. In the cases of
From
On the other hand, in region 3 (peripheral visual field) with a viewing angle of 20° or more of
Instead of employing threshold processing with respect to the amplitude of an event-related potential as described above, the amount of attention to the peripheral visual field may be determined based on correlation coefficient values with respect to prestored templates. Herein, the templates are meant to be the arithmetic mean waveform data of the electroencephalogram signal (c) in the case where the amount of attention is large and the arithmetic mean waveform data of the electroencephalogram signal (f) in the case where the amount of attention is small, both pertaining to region 3 (peripheral visual field) in
With the construction and procedure of processing according to the present embodiment, in an apparatus which provides safe driving assistance by determining the state of a driver, a visual stimulation is generated in the peripheral visual field of the driver, and the amount of attention of the driver to the peripheral visual field is determined from an event-related potential of an electroencephalogram signal based on the temporal point of occurrence of the stimulation as a starting point. This makes it possible to determine an amount of attention with respect to an event that may possibly occur in the peripheral visual field of the driver, e.g., a sudden intrusion of a vehicle or a rushing out of a pedestrian, even in the absence of any behavior index such as braking. Then, based on the result of determination, assistance for inducing a change in the state of the driver, e.g., attention calling, can be appropriately provided.
A driving attention amount determination apparatus according to the present embodiment includes an imaging section for imaging the front of a driver's vehicle. From the video that is captured by the imaging section, the driving attention amount determination apparatus detects an occurrence of a visual stimulation, which serves as a starting point when analyzing an event-related potential of the electroencephalogram, and discerns a central visual field and a peripheral visual field from the position in the captured video at which the visual stimulation occurred. Then, an amount of attention to the peripheral visual field is determined.
As a result, without having to purposely provide a visual stimulation from the driving attention amount determination apparatus as in Embodiment 1, it is possible to determine an amount of attention to the peripheral visual field by taking from within the captured video of the front a natural visual stimulation that occurs in front of the driver during driving.
The imaging section 15 is a camera which is capable of moving picture imaging, for example. The imaging section 15 is disposed at the vehicle front (on the dashboard, behind the rear-view mirror, etc.), and images the vehicle front with an angle of view of 105° along the vertical direction and 135° along the lateral direction, at 30 frames per second, for example. The imaging section 15 can capture an image such as that shown in
From the video captured by the imaging section 15, the peripheral stimulation detection section 16 detects the temporal point of occurrence of a visual stimulation to serve as a starting point when analyzing an event-related potential of the electroencephalogram, and simultaneously determines (identifies) a region of the captured video at which the visual stimulation has occurred. As used herein, a visual stimulation is anything of which an amount of change of the luminance in the video exceeds a predetermined threshold value. Note that it is only exemplary to employ an amount of change. For example, a rate of change in luminance may be employed; in this case, an occurrence of a visual stimulation may be determined when there is a rate of change in luminance of 50% or more. For example, brake lamps of a preceding vehicle, a winker of a flanking vehicle, head lights of an oncoming vehicle, switching of a traffic light, and the like may be visual stimulations. The point in time at which any such change has occurred is detected as the temporal point of occurrence of a visual stimulation.
The peripheral stimulation detection section 16 detects the temporal point of occurrence of a visual stimulation as defined above, and determines whether the position of the stimulation, i.e., the position of luminance change, is in the central visual field or the peripheral visual field. As the determination method, any stimulation that exists in the region of the lane in which the driver's vehicle is present in the captured video as shown in
Hereinafter, with reference to
At step S161, in the vehicle front video having been captured by the imaging section 15, the peripheral stimulation detection section 16 calculates a luminance image difference between frames.
At step S162, from the aforementioned difference, the peripheral stimulation detection section 16 determines whether or not a luminance change has occurred that is equal to or greater than a predetermined threshold value Th1. If such a luminance change has occurred, control proceeds to step S163; if not, control returns to step S161 to calculate a next inter-frame luminance difference.
At step S163, the peripheral stimulation detection section 16 stores the temporal point of luminance change and the position at which the luminance change occurred in that image.
At step S164, the peripheral stimulation detection section 16 detects white lines from the inter-frame luminance difference calculated at step S161. Specifically, in a road image taken from a vehicle which is traveling at a certain rate, the asphalt on the road surface and the structures and vegetation around the road appear to be moving, but the white lines, which maintain essentially constant luminance values in their places, appear still in the image. Therefore, from the inter-frame luminance difference, the peripheral stimulation detection section 16 detects any region which is equal to or less than a predetermined threshold value Th2 to be a region of a non-moving white line.
At step S165, by using the detected white lines, the peripheral stimulation detection section 16 extracts, as a lane region, a region where the distance between both white lines is equal to or greater than a certain width. In the example of
At step S166, the peripheral stimulation detection section 16 determines whether the position of luminance change stored at step S163 falls outside the lane region extracted at step S165 or not. If it is determined as falling outside the lane region, the luminance change is determined to have occurred in the peripheral visual field 32 (
At step S167, to the attention amount determination section 13, the peripheral stimulation detection section 16 transmits the temporal point of luminance change which has been determined as a luminance change in the peripheral visual field 32 (
The above description assumes that the peripheral visual field 31 and the peripheral visual field 32 do not significantly change, but are substantially fixed.
However, the central visual field and the peripheral visual field of a driver are actually not fixed, and will presumably change depending on the driving situation (velocity of the driver's vehicle or lightness of the neighborhood of the driver's vehicle). For example, when the driver's vehicle is traveling on an expressway at 100 km per hour or more, the field of vision of the driver is narrower than when the vehicle is stopped. Moreover, when the neighborhood of the driver's vehicle is dark, e.g., at night, the field of vision of the driver is narrower than in the daytime. When the field of vision of the driver becomes narrow, detection of dangerous objects is delayed even in a visual field which is closer to the center, thus increasing the possibility of cross-collision accidents and rush-out accidents.
Therefore, by detecting the driving situation and changing the central visual field and peripheral visual field, an amount-of-attention determination which is truer to the actual circumstances is made possible. With reference to
In accordance with the velocity of the driver's vehicle and the activation/inactivation of the head lamps of the driver's vehicle as detected by the situation detection section 17, the peripheral stimulation detection section 16 changes the definitions of the central and peripheral visual fields.
Table 2 shows an exemplary relationship between the velocity of the driver's vehicle and an area ratio of the central visual field relative to that when the vehicle is stopped.
In the above table, (a) at less than 50 km per hour, the area ratio relative to that when the vehicle is stopped is set to 1; (b) at 50 km per hour or more but less than 100 km per hour, the area ratio is set to 0.8; and (c) at 100 km per hour or more, the area ratio is set to 0.6. In
Table 3 shows an exemplary relationship between activation/inactivation of the head lamps of the driver's vehicle and an area ratio of the central visual field relative to that in the daytime.
In the above table, (a) when no lamps are activated, the area ratio relative to that in the daytime is set to 1; (b) when the small lamps (clearance lights) are activated, the area ratio is set to 0.8; and (c) when the head lamps (head lights) are activated, the area ratio is set to 0.6. This makes it possible to determine an amount of attention to the peripheral visual field in accordance with activation/inactivation of the head lamps of the driver's vehicle, thus reducing the danger of cross-collision accidents and rush-out accidents.
With the construction and procedure of processing according to the present embodiment, occurrence of a visual stimulation is detected from within a captured video of the front of the driver's vehicle; the central visual field and the peripheral visual field are discerned from the position in the captured video at which the stimulation occurred; and an amount of attention to the peripheral visual field is determined. Thus, without having to purposely provide a visual stimulation from the driving attention amount determination apparatus, it is possible to determine an amount of attention to the peripheral visual field by taking from within the captured video of the front a natural visual stimulation that occurs in front of the driver during driving. Furthermore, it is possible to determine an amount of attention to the peripheral visual field in accordance with the driver's vehicle and the surrounding situation.
In Embodiment 2, a region in which a stimulation occurs is determined based on the assumption that a driver is basically looking in the front center during driving. However, the driver may not always have his or her line of sight directed in the front center when a visual stimulation occurs, and therefore the peripheral visual field may always be fluctuating.
Therefore, in the present embodiment, a line-of-sight measurement section for measuring the line of sight of the driver is provided in the driving attention amount determination apparatus. The driving attention amount determination apparatus determines a region in which a visual stimulation occurs in accordance with the position of a fixation point of the driver.
The line-of-sight measurement section 18 may be a head-mount type measuring instrument which is worn by the driver in advance, or an onboard-type measuring instrument which is disposed near the rear-view mirror of the vehicle.
The peripheral stimulation detection section 16 detects the temporal point of occurrence of a visual stimulation, and determines whether the position of the stimulation is in the central visual field or the peripheral visual field. As the determination method, the region in which the stimulation has occurred is determined based on the position of the fixation point 41 (
With the construction and procedure of processing according to the present embodiment, the line of sight of a driver is measured, and a peripheral visual field is determined in accordance with the position of a fixation point, thus making it possible to accurately determine whether the stimulation is in the peripheral visual field or not even if the driver is not directing his or her line of sight in the front center when the visual stimulation occurs. As a result, an amount of attention to the peripheral visual field can be determined with a higher accuracy.
In the case where the driving attention amount determination apparatus according to the present invention is implemented as a head-mount display type apparatus to be worn by a user, an amount of attention to the peripheral visual field can be determined even while the user is riding a bicycle or walking, without being limited to safety assistance while driving an automobile. For example, when the user is watching television on a sub-screen of a wearable-type display while walking, an amount of attention to the peripheral visual field of the user may be determined based on an event-related potential of his or her electroencephalogram, thus making it possible to appropriately call attention to an obstacle in walking, etc.
The aforementioned line-of-sight measurement section 18 may be provided in the driving attention amount determination apparatus 1 (
For example,
By providing the line-of-sight measurement section 18 in the driving attention amount determination apparatus 1a, it becomes possible for the driving attention amount determination apparatus 1a to dynamically identify the incessantly-changing central visual field and peripheral visual field of a driver. Thus, the peripheral stimulation generation section 12 is able to selectively flicker a light source which is positioned in the peripheral visual field of the driver.
Taking the glasses-type head-mount display of
Thus, by providing the line-of-sight measurement section 18 to control presentation of a visual stimulation, it is ensured that a visual stimulation is presented in the peripheral visual field of a driver. Thus, it is possible to determine whether or not attention is being paid to the peripheral visual field with a higher accuracy.
Furthermore,
When the driving attention amount determination apparatus 2b detects the velocity of the driver's vehicle and the activation/inactivation of the head lamps of the driver's vehicle by using the situation detection section 17, the peripheral stimulation detection section 16 changes the definitions of the central and peripheral visual fields in accordance with the velocity of the driver's vehicle and the activation/inactivation of the head lamps of the driver's vehicle as detected by the situation detection section 17.
With respect to each of the above-described Embodiments, any process that was described by employing a flowchart can be implemented as a program to be executed by a computer. Such a computer program may be distributed on the market in the form of a product recorded on a storage medium, such as a CD-ROM, or transmitted via telecommunication lines such as the Internet.
All or some of the constituent elements composing the driving attention amount determination apparatus may be implemented as a general-purpose processor (semiconductor circuit) executing a computer program. Alternatively, they may be implemented as a special processor in which such a computer program and a processor are integrated. For example, a processor executing a computer program receives an electroencephalogram signal of a driver which is measured by the electroencephalogram measurement section 11. Then, from the electroencephalogram signal measured based on a starting point which is the temporal point of occurrence of a visual stimulation occurring in the peripheral visual field of the driver, the processor determines an amount of attention of the driver to the peripheral visual field, and outputs a signal based on the result of determination. As a result, attention of the driver can be called.
Otherwise, as a processor executes a computer program, the processor may control the operations of the peripheral stimulation generation section 12, the imaging section 15, the peripheral stimulation detection section 16, the situation detection section 17, the line-of-sight measurement section 18, and the like, or the processor may function as each such constituent element.
The driving attention amount determination apparatus according to the present invention is useful for preventing accidents in connection with events that may occur in the peripheral visual field of the driver, e.g., a sudden intrusion of a vehicle or a rushing out of a pedestrian. In the case where it is implemented as a head-mount display type apparatus, it is also applicable to safety assistance while riding a bicycle or while walking.
While the present invention has been described with respect to preferred embodiments thereof, it will be apparent to those skilled in the art that the disclosed invention may be modified in numerous ways and may assume many embodiments other than those specifically described above. Accordingly, it is intended by the appended claims to cover all modifications of the invention that fall within the true spirit and scope of the invention.
Number | Date | Country | Kind |
---|---|---|---|
2008-201520 | Aug 2008 | JP | national |
This is a continuation of International Application No. PCT/JP2009/003724, with an international filing date of Aug. 4, 2009, which claims priority of Japanese Patent Application No. 2008-201520, filed on Aug. 5, 2008, the contents of which are hereby incorporated by reference.
Number | Date | Country | |
---|---|---|---|
Parent | PCT/JP2009/003724 | Aug 2009 | US |
Child | 12718326 | US |