Patent Application
20200090518
Embodiments relate to a drive assist device, a drive assist system, and a drive assist method.
Drive assist devices that notifies drivers of the presence of hazardous places, obstacles, and the like (hazard objects) in vehicles or the like to call drivers' attention are provided.
Related art is disclosed in Japanese Laid-open Patent Publication No. 2008-40974, Japanese Laid-open Patent Publication No. 2003-99899, Japanese Laid-open Patent Publication No. 7-167668 and Japanese Laid-open Patent Publication No. 2005-4414.
According to an aspect of the embodiments, an information processing device includes: a memory; and a processor coupled to the memory and configured to: detect an environment around a vehicle which is driven by a driver; generate a hazard list of an object to be a hazard based on the detected environment; detect a gaze of the driver; evaluate a risk regarding driving of the driver for each object included in the hazard list based on a frequency at which the object included in the hazard list is included in a field of view of the driver based on the detected gaze; and output drive assist information corresponding to the object with the evaluated risk that is equal to or larger than a threshold.
The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention.
For example, the drive assist device displays a name, a code, and the like for specifying the hazardous place or the like in a gaze direction of when the driver looks at the hazardous place or the like, and determines whether a sound uttered by the driver matches a sound pattern of the displayed name, code, or the like. Then, in a case where the uttered sound does not match the sound pattern within a predetermined time after the name, code, or the like for specifying the hazardous place or the like is displayed, the drive assist device issues a warning to the driver.
However, warnings are sometimes excessively issued for cognitive activities of the driver, and drive assist is sometimes excessively performed.
For example, in a case where the drive assist is excessively performed, such as Issuance of warnings to the driver, although the driver sufficiently recognizes the hazardous place or the like, the driver's tendency to depend on the drive assist becomes stronger. The driver who has strong dependence on the drive assist has less cognitive activities for hazardous places, and for example, the prediction ability for the hazardous places is less improved. Thus, the excessive drive assist may not lead to assist for the driver's cognitive activities.
A drive assist device, a drive assist system, and a drive assist method for realizing appropriate drive assist for the driver may be provided.
Hereinafter, a drive assist device, a drive assist system, and a drive assist method according to embodiments will be described with reference to the drawings. Configurations having the same functions in the embodiments are denoted by the same reference signs, and redundant description will be omitted. Note that the drive assist device, the drive assist system, and the drive assist method to be described in the following embodiment are merely examples and thus do not limit the embodiments. In addition, the embodiments below may be appropriately combined unless otherwise contradicted.
Note that, in the following embodiments, “call attention message” refers to drive assist information for calling driver's attention regarding drive operation in a vehicle or the like.
In addition, a “driver” refers to an object person to which sounds of the call attention message are output to assist driving. In the following embodiments, cases where the object person is located at a position of a driver's seat for driving and operating the vehicle will be described. However, even if the driver is not at the position of the driver's seat, a person who is located at the position where the person can perform the drive operation is also included in the driver.
In addition, a “virtual outputting person” refers to a virtual sound source that outputs various types of sound information including the call attention message by sounds. When outputting the sound information via a plurality of sound output units (sound output devices such as speakers) arranged around the driver who is performing the drive operation, the virtual sound source can be arranged in an arbitrary direction around the driver by controlling a volume (sound pressure) and a phase. For example, the driver may be caused to recognize as if persons having predetermined attributes exist in respective directions in which the virtual sound sources are arranged by changing voice sound for each of the virtual sound sources.
Note that, a virtual sound source arranged at a position corresponding to the position of the seat around the driver, among the virtual outputting persons, is referred to as a “virtual occupant”.
In addition, “check action” refers to an action (a motion for grasping a driving environment or situation such as grasping of an external object by visual observation, check of an operation panel or a rearview mirror, for example) performed by the driver for check according to a situation (right or left turn, overtaking, or the like, for example) of the vehicle operated by the driver.
Further, “call” refers to a check action performed by utterance among check actions of the driver. Note that the check actions include a pointing action, a movement of a gaze, and the like, in addition to the call. Furthermore, the “call” includes information provision from the “virtual occupant” by a peripheral monitoring function by a vehicle sensor.
Further, “repeated call” means that virtual occupants output information (repeated call information) corresponding to content of the call by sounds at the same time after the call (note that the output may not be necessarily perfectly at the same time, and there may be a gap to some extent in sound output timing among the virtual occupants). Furthermore, “continuous call” means that the virtual occupants sequentially output the information corresponding to the content of the call by sounds after the call. Here, the “continuous call” includes the “repeated call” by the driver corresponding to preceding “call” by a virtual occupant, and further includes “continuous call” “repeated” by the “virtual occupants” at the same time. Note that, in the following embodiments, a case of mainly performing the “repeated call” will be described. However, similar processing is performed in the case of the “continuous call”.
As illustrated in
The drive assist device 100 is a device which is installed in a vehicle driven by a driver and assists cognitive activities of the driver at the time of driving by providing the driver with drive assist information using a sound output by the sound output unit 131 and a display output by the display output unit 132.
Specifically, the driver recognizes hazardous places such as a pedestrian, and vehicles including a leading vehicle, a following vehicle, and an oncoming vehicle under the surrounding environment of the vehicle and the situation as hazard objects (S1). Then, the driver monitors the driver's own vehicle (such as a lane and a speed) and monitors surrounding hazard objects (such as distances between the vehicle and front, rear, right, and left vehicles) (S2).
In addition, the driver searches (S4) for the hazard object in searching (S3) a peripheral field of view in a view, and checks situation change of the hazard object (S5). In a case where there is the situation change (S5: YES), the driver searches for the hazard object in a central field of view (S6), and supplements the hazard object with the central field of view (S7) to update cognition of the hazard object (S8). In a case where there is no situation change (S5: NO), the driver does not supplement the hazard object in the central field of view, and forgetting of the hazard progresses (S9).
Next, the driver makes a safety determination on the basis of the recognized hazard object or the like (S10). For example, in a case where the driver does not recall lane change (S11: NO) according to presence or absence of recall of lane change (change from a driving lane to an overtaking lane) by safety determination (S11), the driver performs an operation (for an accelerator or a steering wheel) to maintain the inter-vehicle distance and speed (S12).
In a case where the driver has recalled the lane change (S11: YES), the driver performs risk prediction for the hazard object (S13), and performs safety check for risk avoidance (S14). Specifically, the driver performs safety check of a hazard ahead, safety check of a right rear hazard, and absence check of a hazard in a right blind spot according to a drive procedure.
Next, the driver makes a safety check decision and determines lane change (S15). The driver who has determined the lane change presents a blinker (S16), and rechecks the absence of the hazard in the right blind spot (S17), and then performs an operation for the lane change (for the accelerator or the steering wheel) (S18). Next, the driver performs safety check (S19) and completes the lane change (S20).
The drive assist device 100 detects the environment around the vehicle driven by the driver and generates a hazard list of an object to be a hazard on the basis of the detected environment, with respect to the cognitive structure model of the driver at the time of driving. Then, the drive assist device 100 detects a gaze of the driver and evaluates a risk (low interest) regarding driving of the driver for each object included in the hazard list on the basis of a frequency at which the object included in the hazard list is included in a field of view of the driver based on the detected gaze. Then, the drive assist device 100 outputs drive assist information corresponding to the object having the evaluated risk that is equal or larger than a threshold, using a sound output by the sound output unit 131 or a display output by the display output unit 132.
Therefore, the drive assist device 100 evaluates the risk for the object included in the hazard list detected by a peripheral monitoring sensor on the basis of the frequency of being included in the field of view of the driver, and provides the drive assist information corresponding to the object having the risk that is equal to or larger than the threshold, thereby suppressing excessive drive assist and realizing appropriate drive assist for the driver. For example, the drive assist device 100 does not provide the drive assist information for an object having a high frequency of being included in the field of view of the driver and recognized by the driver as an object of sufficiently high interest so as not to disturb the cognitive activities of the driver for the object. In addition, the drive assist device 100 provides the drive assist information for an object having a low frequency of being included in the field of view of the driver, and insufficiently recognized by the driver due to a decrease in interest, thereby suppressing the driver forgetting the object.
Returning to
Further, the navigation device 111 detects whether the driver's own vehicle travels straight, turning to the right, or turning to the left when the vehicle has reached the intersection on the basis of a route search result to a destination and a vehicle situation (such as a history of the drive operation and the speed of the vehicle). Similarly, the navigation device 111 detects whether the vehicle is branching to the left side or branching to the right side when the vehicle has reached the branching point on the basis of the route search result to the destination. Further, the navigation device 111 detects whether the vehicle is merging from the right side or merging from the left side when the vehicle has reached the meeting point on the basis of the route search result to the destination. The navigation device 111 includes these detection results (course detection results) in the position information and transmits the position information to the drive assist ECU 120.
The vehicle control ECU 112 acquires drive operation information (a steering angle of the steering wheel, an opening degree of an accelerator pedal, a depression amount of a brake pedal, and the like, for example) indicating the drive operation performed by the driver for the vehicle, vehicle information (a vehicle speed, for example) indicating a state of the vehicle, and the like. The vehicle control ECU 112 transmits the acquired drive operation information and vehicle information to the drive assist ECU 120.
The gesture detection unit 113 detects a gesture of the driver on the basis of an image of a vehicle interior captured by an on-vehicle camera (not illustrated) or the like. In addition, when detecting that the driver has performed a pointing action, the gesture detection unit 113 determines a pointing direction and transmits pointing direction information to the drive assist ECU 120.
The sound detection unit 114 detects call by the driver on the basis of a sound inside the vehicle collected by an on-vehicle microphone (not illustrated) or the like and transmits call information to the drive assist ECU 120.
The gaze detection unit 115 detects a gaze direction of the driver on the basis of a face image of the driver captured by the on-vehicle camera (not illustrated) or the like and transmits gaze direction information to the drive assist ECU 120.
The surrounding environment detection unit 116 detects obstacles such as pedestrians and other vehicles around the vehicle driven by the driver on the basis of detection information by the on-vehicle sensor (not illustrated), the on-vehicle camera (not illustrated), and the like, for example, and transmits surrounding environment information to the drive assist ECU 120. Specifically, the surrounding environment detection unit 116 transmits identification information for identifying the detected obstacles (the pedestrians, vehicles, and the like, for example), and the surrounding environment information including position information, speed information, and the like of the obstacles with respect to the vehicle driven by the driver to the drive assist ECU 120. That is, the surrounding environment detection unit 116 is an example of an environment detection unit. Note that, in the example in
In addition, the surrounding environment detection unit 116 may acquire the surrounding environment information via road-to-vehicle communication or vehicle-to-vehicle communication and transmit the surrounding environment information to the drive assist ECU 120. In this case, the surrounding environment detection unit 116 functions as communication means.
The drive assist ECU 120 performs control for outputting the call attention message by sounds or by display. The drive assist ECU 120 stores a program for realizing various functions. The drive assist ECU 120 executes the program to function as a situation determination unit 121, an information specifying unit 122, an action determination unit 123, a hazard list generation unit 124, a risk evaluation unit 125, and an output control unit 126. In addition, the drive assist ECU 120 includes an action information storage unit 127 for storing various types of information such as action information, a hazard list, and a risk table, a hazard list storage unit 128, and a risk table storage unit 129.
Here, a hardware configuration of the drive assist ECU 120 will be described.
As illustrated in
The CPU 201 is a computer that executes various programs stored in the ROM 202.
The ROM 202 is a non-volatile memory, and stores various programs executed by the CPU 201 and the information (the action information, the hazard list, the risk table, and the like) used when the various programs are executed.
The RAM 203 is a volatile memory, and provides a work area in which the various programs stored in the ROM 202 are expanded when executed by the CPU 201.
The connection device 204 connects the drive assist ECU 120 to an external device, and transmits and receives various types of information to and from the external device. Examples of the external device include the navigation device 111, the vehicle control ECU 112, the gesture detection unit 113, the sound detection unit 114, the gaze detection unit 115, the surrounding environment detection unit 116, the sound output unit 131, the display output unit 132, the communication unit 133, and the like.
Returning to
When receiving the information regarding the current situation of the vehicle from the situation determination unit 121, the information specifying unit 122 specifies information used when the call attention message is output by sounds to the driver by reference to the action information in the action information storage unit 127, and notifies the output control unit 126 of the information. That is, the information specifying unit 122 is an example of a message specifying unit.
Specifically, the information specifying unit 122 acquires the call attention message and an ID of the message on the basis of the information regarding the current situation of the vehicle. In addition, the information specifying unit 122 specifies a direction that the driver should check on the basis of the acquired call attention message, and specifies a virtual outputting person defined in association with the specified direction. In addition, the information specifying unit 122 specifies timing and order to output sounds defined in association with the acquired call attention message. Further, the information specifying unit 122 notifies the output control unit 126 of these pieces of information as information used when the call attention message is output by sounds.
In addition, when acquiring the information regarding the current situation of the vehicle from the situation determination unit 121, the information specifying unit 122 specifies information used when appropriateness of a check action of the driver is determined by reference to the action information of the risk evaluation unit 125, and notifies the action determination unit 123 of the information.
Specifically, the information specifying unit 122 acquires the call attention message and an ID of the message on the basis of the information regarding the current situation of the vehicle. In addition, the information specifying unit 122 specifies order, pointing direction, call content, and gaze direction defined in association with the acquired call attention message. Then, the information specifying unit 122 notifies the action determination unit 123 of these pieces of information as the information used when the appropriateness of the check action of the driver is determined.
When acquiring the information used when the appropriateness of the check action of the driver is determined from the information specifying unit 122, the action determination unit 123 receives the pointing direction information from the gesture detection unit 113, receives the call information from the sound detection unit 114, and receives the gaze direction information from the gaze detection unit 115.
In addition, the action determination unit 123 determines whether at least any one of the received pointing direction information, call information, and gaze direction information corresponds to the information used when the appropriateness of the check action of the driver is determined, which has been acquired from the information specifying unit 122. In the case of determining that the at least any one of the information corresponds to the information, the action determination unit 123 determines that the driver of the vehicle has performed an appropriate check action.
In addition, the action determination unit 123 acquires, on the basis of a determination result as to whether the driver of the vehicle has performed the appropriate check action, a determination result message according to the determination result, and a virtual outputting person of the determination result message, from preset setting information and the like. Next, the action determination unit 123 notifies the output control unit 126 of the acquired determination result message and virtual outputting person.
The hazard list generation unit 124 generates the hazard list of the object to be a hazard around the vehicle on the basis of the surrounding environment information detected by the surrounding environment detection unit 116. Specifically, the hazard list generation unit 124 recognizes an obstacle detected by the surrounding environment detection unit 116 as the object to be a hazard and provides an object ID for identifying the object, and then stores the hazard list listing information (identification information, position information, speed information, and the like) for each object included in the surrounding environment information in the hazard list storage unit 128.
The risk evaluation unit 125 refers to the hazard list in the hazard list storage unit 128, and evaluates a risk regarding driving of the driver for each object included in the hazard list on the basis of the gaze direction information detected by the gaze detection unit 115. Next, the risk evaluation unit 125 outputs an evaluation result of the risk for each object included in the hazard list to the output control unit 126.
Specifically, the risk evaluation unit 125 obtains, for each object included in the hazard list, a frequency of being included in the field of view of the driver indicated by the gaze direction information. Then, the risk evaluation unit 125 calculates an oblivion degree indicating the degree of forgetfulness of the driver about the object according to the obtained frequency. For example, the risk evaluation unit 125 calculates the oblivion degree to be higher for the object having a lower frequency of being included in the field of view of the driver.
In addition, the risk evaluation unit 125 obtains hazard information objectively indicating the degree of hazard of the object with respect to the vehicle according to the position information and the speed information for each object included in the hazard list. The hazard information includes, for example, time-to-collision (TTC) that is a collision time to the vehicle for each object and the like. TTC is calculated by dividing the distance between the vehicle and the object by a speed difference (relative speed) on the basis of the position information and the speed information for each object.
Next, the risk evaluation unit 125 multiplies the oblivion degree for each object included in the hazard list by the hazard information such as TTC to obtain an evaluation value of the risk regarding driving of the driver for each object (hereinafter the evaluation value is referred to as a risk value). For example, an object having a higher oblivion degree and a higher risk has a higher risk value. Next, the risk evaluation unit 125 outputs the risk value for each object included in the hazard list to the output control unit 126.
The output control unit 126 acquires the call attention message, the ID of the message, the virtual outputting person, the timing, and the order notified from the information specifying unit 122. The output control unit 126 outputs the acquired call attention message by sounds via the sound output unit 131 at the acquired timing.
In addition, the output control unit 126 outputs the call attention message regarding the object having the risk value that is equal to or larger than a preset threshold by sounds via the sound output unit 131 on the basis of the risk value for each object included in the hazard list notified from the risk evaluation unit 125.
Specifically, the output control unit 126 acquires the position information of the object having the risk value that is equal to or larger than the present threshold by reference to the hazard list storage unit 128, and specifies the call attention message in which the direction to be checked in the call attention message and the position of the object match on the basis of the acquired position information.
Next, the output control unit 126 outputs the specified call attention message as the call attention message regarding the object by sounds via the sound output unit 131. Thereby, the drive assist device 100 can provide the driver with the call attention message about the object having a low frequency of being included in the field of view of the driver and insufficiently recognized by the driver, for example.
The action information storage unit 127 is, for example, a data table or the like, and stores the action information for each identification information such as an ID.
As illustrated in
In the “position”, a position at which the call attention message should be output to the driver by sounds is defined. The example in
In the “situation”, a situation of when the vehicle has reached any position of the intersection, the branching point, and the meeting point is defined. This is because the direction that the driver should check is different depending on the situation. For example, in a case where the situation of when the vehicle has reached the intersection is “straight ahead”, the drive assist device 100 causes the driver to check left side and right side. In a case where the situation of when the vehicle has reached the intersection is “right turn”, the drive assist device 100 causes the driver to check right front and ahead of right turn. Further, in a case where the situation of when the vehicle has reached the intersection is “left turn”, the drive assist device 100 causes the driver to check ahead of left turn and left rear.
In the “ID”, an identifier for identifying each “call attention message” output by sounds according to the “position” and the “situation” is stored.
In the “call attention message”, a message to be output by sounds according to the “position” and the “situation” is stored. For example, in a case of the position=“intersection” and the situation=“straight ahead”, “caution, intersection. stop and check”, “check left”, and “check right” are stored as messages to be output by sounds.
In the “direction to be checked”, a direction that the driver should check by outputting the “call attention message” by sounds is defined. For example, in a case of the call attention message=“intersection. please stop”, the direction that the driver should check is front. Therefore, in the “direction to be checked”, “front” is defined. In a case of the call attention message=“check left side”, the direction that the driver should check is a left direction. Therefore, in the “direction to be checked”, “left side” is defined. Further, in a case of the call attention message=“check right side”, the direction that the driver should check is a right direction. Therefore, in the “direction to be checked”, “right side” is defined.
In the “virtual outputting person”, a virtual outputting person arranged at a position corresponding to the direction that the driver should check is defined by outputting the call attention message by sounds. For example, in a case of the direction to be checked=“front”, a virtual outputting person arranged at a position outside and in front of the vehicle is defined in the “virtual outputting person”. Further, in a case of the direction to be checked=“left side”, a virtual occupant arranged at a position of a passenger's seat is defined in the “virtual outputting person”. Furthermore, in a case of the direction to be checked=“right side”, a virtual outputting person arranged at the position outside and on the right side of the vehicle is defined in the “virtual outputting person”.
In the “timing”, timing to output the call attention message by sounds is defined. In the “order”, an order to output the call attention message by sounds is defined.
Note that “L” represents a distance from the current position of the vehicle to the “position” (for example, the intersection) and “V” represents the current vehicle speed of the vehicle, in the expressions stored in the “timing”.
For example, in the case of the position=“Intersection” and the situation=“straight ahead”, the call attention message of “caution, intersection. stop and check” is output by sounds (L/V+10) seconds before the vehicle reaches the intersection. Further, the call attention message of “check right” is output by sounds (L/V+5) seconds before the vehicle reaches the intersection. Furthermore, the call attention message of “check left” is output by sounds (L/V) seconds before the vehicle reaches the intersection. That is, in the case of the action information of the action information storage unit 127, the timing of sound output of the call attention message is determined on the basis of the current vehicle speed of the vehicle, and respective call attention messages are output by sounds in a defined order at predetermined intervals, in this case, at intervals of 5 seconds. A speech pause time of punctuation may be expanded or reduced depending on the situation.
In the “pointing direction”, a direction of a pointing action that the driver should perform in response to the sound output of the call attention message is stored. In the “call content”, a call content that the driver should perform in response to the sound output of the call attention message is stored. In the “gaze direction”, a direction of a gaze that the driver should give in response to the sound output of the call attention message is stored.
Returning to
The hazard list storage unit 128 stores label information, position information, inter-vehicle information, speed information, and hazard information of the object for each object ID.
The label information is information labeled on the basis of identification content of the object (for example, a pedestrian, a vehicle, or the like), a positional relationship (front and back), and a relative speed. For example, label information such as “leading vehicle” is given to a vehicle traveling in front in the same direction, and “following vehicle” is given to a vehicle traveling behind in the same direction.
The position information, the inter-vehicle information, and the speed information indicate the position, inter-vehicle distance, a relative speed, and the like with respect to the vehicle of the driver based on the position information and the speed information of the object detected by the surrounding environment detection unit 116. The hazard information is, for example, TTC regarding the object calculated on the basis of the position information, inter-vehicle distance, and relative speed.
The risk table storage unit 129 stores, for each object ID, check frequency information, the oblivion degree, the risk value, and the flag information of the object. The check frequency information is information indicating the frequency at which the object is captured in the field of view of the driver, and is, for example, time when the presence of the object in the field of view is checked or the like.
The oblivion degree is a value indicating the degree of forgetfulness of the driver about the object, which is calculated by the risk evaluation unit 125. Similarly, the risk value is an evaluation value of the risk regarding driving of the driver for the object, which is calculated by the risk evaluation unit 125. The flag information is a flag assigned to the object. An example of the flag information includes a risk prediction flag indicating that the driver has predicted (recognized) a risk to the object by performing a drive operation (accelerator reduction or the like) for the object and the like, for example.
Returning to
The display output unit 132 is a display, an indicator light, or the like that outputs information by display under the control of the output control unit 126. For example, as the display output unit 132, a liquid crystal display (LCD) provided on a dashboard or the like, a projection device for projecting information on a front windshield or the like can be applied. Further, the display output unit 132 may include an indicator light 132a (light emitting diode (LED), see
The communication unit 133 is a communication device that communicates with an external device by wireless communication under the control of the drive assist ECU 120. Specifically, the communication unit 133 performs road-to-vehicle communication, vehicle-to-vehicle communication, communication with a server device 501 (see
Next, a flow of drive assist processing executed by the drive assist ECU 120 will be described.
The drive assist processing illustrated in
Here, details of the situation determination processing will be described.
Next, the situation determination unit 121 extracts the position detection result and the course detection result included in the received position information. Note that the position detection result includes any of the intersection, the branching point, and the meeting point, and the course detection result includes any of straight ahead, right turn, left turn, branching to the right side, branching to the left side, merging from the right side, and merging from the left side.
Next, the situation determination unit 121 determines whether the current situation of the vehicle corresponds to the situation defined in the action information of the action information storage unit 127 on the basis of the position detection result and the course detection result (S31).
In the case where the current situation is determined not to correspond to the situation defined in the action information (S31: NO), the situation determination unit 121 returns the processing to S30. On the other hand, in the case where the current situation is determined to correspond to the situation defined in the action information (S31: YES), the information specifying unit 122 acquires the ID corresponding to the situation determined by the situation determination unit 121 from the action information of the action information storage unit 127 (532).
For example, in the case where the vehicle is determined to be in the situation of traveling straight through the intersection, the information specifying unit 122 acquires ID=“1”, “2”, and “3” from the action information (see
Next, the information specifying unit 122 acquires the call attention message corresponding to the acquired ID (S33). For example, in the case where the acquired IDs are “1”, “2”, and “3”, the information specifying unit 122 acquires “caution, Intersection. stop and check”, “check left”, and “check right”, as the corresponding call attention message (see
Next, the information specifying unit 122 specifies the direction that the driver should check on the basis of the acquired call attention message by reference to the action information of the action information storage unit 127, and specifies the virtual outputting person defined in association with the specified direction. In addition, the information specifying unit 122 specifies the timing and order defined in association with the acquired call attention message by reference to the action information (S34).
Referring to the example in
Next, the information specifying unit 122 notifies the output control unit 126 of the acquired ID and call attention message, and the specified virtual outputting person, timing, and order, as the information used when the call attention message is output by sounds (S35).
Further, the information specifying unit 122 specifies the order, and the pointing direction, the call content, and the gaze direction of the driver defined in association with the acquired call attention message by reference to the action information (S36) in parallel to the processing in S34 and S35.
Referring to the example in
Next, the information specifying unit 122 notifies the action determination unit 123 of the acquired ID, and the specified order, pointing direction, call content, and gaze direction, as the information used when the appropriateness of the check action of the driver is determined (S37). Thereafter, the information specifying unit 122 returns the processing.
Returning to
Here, details of the check action determination processing will be described.
Next, the action determination unit 123 reads the pointing direction corresponding to the order=“n” out of pointing directions included in the information used when the appropriateness of the check action of the driver is determined, which has been notified from the information specifying unit 122 (S41). For example, in the case of the order=“1”, the action determination unit 123 reads the pointing direction=“front”.
Next, the action determination unit 123 determines whether the gesture of the driver is appropriate (S42). Specifically, the action determination unit 123 determines whether the pointing direction information acquired in S40 matches the “pointing direction” read in S41. In the case where the pointing direction information is determined to match the read pointing direction (S42: YES), the action determination unit 123 advances the processing to S49.
In the case where the pointing direction information is determined not to match the read pointing direction (S42: NO), the action determination unit 123 determines that the gesture (pointing action) of the driver is not appropriate and advances the processing to S43. In S43, the action determination unit 123 receives the call information from the sound detection unit 114.
Next, the action determination unit 123 reads the call content corresponding to the order=“n” out of call contents included in the information used when the appropriateness of the check action of the driver is determined, which has been notified from the information specifying unit 122 (S44). For example, in the case of the order=“1”, the action determination unit 123 reads the call content=“stop clear”.
Next, the action determination unit 123 determines whether the call of the driver is appropriate (S45). Specifically, the action determination unit 123 determines whether the call information acquired in S43 matches the “call content” read in S44. In the case where the call information is determined to match the call content (S45: YES), the action determination unit 123 determines that the call of the driver is appropriate and advances the processing to S49.
In the case where the call information is determined not to match the call content (S45: NO), the action determination unit 123 determines that the call of the driver is not appropriate and advances the processing to S46. In S46, the action determination unit 123 receives the gaze direction information from the gaze detection unit 115.
Next, the action determination unit 123 reads the gaze direction corresponding to the order=“n” out of gaze directions included in the information used when the appropriateness of the check action of the driver is determined, which has been notified from the information specifying unit 122 (S47). For example, in the case of the order=“1”, the action determination unit 123 reads the gaze direction=“front”.
Next, the action determination unit 123 determines whether the gaze direction of the driver is appropriate (S48). Specifically, the action determination unit 123 determines whether the gaze direction information received in S46 matches the “gaze direction” read in S47. In the case where the gaze direction information is determined to match the gaze direction (S48: YES), the action determination unit 123 determines that the gaze direction of the driver is appropriate and advances the processing to S49.
In the case where the gaze direction information is determined not to match the gaze direction (S48: NO), the action determination unit 123 determines that the gaze direction of the driver is not appropriate and advances the processing to S52.
In S49, the action determination unit 123 determines that the appropriate check action has been performed by the driver. Next, the action determination unit 123 reads the determination result message and the virtual outputting person corresponding to the case where the appropriate check action has been performed from the preset setting information or the like and notifies the output control unit 126 of the read information (S50). Examples of the determination result message include sounds such as “good” praising for the appropriate check action and “check”, a clear sound, and the like. Further, the virtual outputting person may be “all virtual occupants”, or the like. Further, another “repeated call” by the virtual occupant (“continuous call”) may be performed in time with the last call “good” so as not to disturb the continuous call by the driver.
Next, the output control unit 126 outputs by sounds the determination result message notified from the action determination unit 123 via the sound output unit 131 (S51). Thereby, the appropriate check action having been performed can be notified to the driver by the sounds such as “good” and “check”, the clear sound, and the like. Thereafter, the output control unit 126 returns the processing.
In S52, the action determination unit 123 determines that the appropriate check action has not been performed by the driver. Next, the action determination unit 123 reads the determination result message and the virtual outputting person corresponding to the case where the appropriate check action has not been performed from the preset setting information or the like and notifies the output control unit 126 of the read information (S53). For example, examples of the determination result message include a sound such as “unchecked” Indicating that the appropriate check action has not been performed, a beep sound, and the like. Further, the virtual outputting person may be “all virtual occupants”, or the like.
Next, the output control unit 126 outputs by sounds the determination result message notified from the action determination unit 123 via the sound output unit 131 (S54). Thereby, the appropriate check action having not been performed can be notified to the driver by the sound such as “unchecked”, the beep sound, and the like. Thereafter, the output control unit 126 returns the processing.
Returning to
Here, details of hazard list update (generation) processing will be described.
As illustrated in
In the case where the change has been supplemented (S61: YES), the hazard list generation unit 124 gives a label such as “leading vehicle” or “following vehicle” on the basis of the surrounding environment information (S62) and measures the inter-vehicle distance and the relative speed (S63) for the vehicle (nearby vehicle) with supplemented change. Next, the hazard list generation unit 124 updates the hazard list in the hazard list storage unit 128 regarding the vehicle (object) with the supplemented change on the basis of the label given in S62, the inter-vehicle distance and the relative speed measured in S63, and the like (S64).
Note that, in the case where the change is not supplemented (S61: NO), the hazard list generation unit 124 skips S62 and S63 and advances the processing to S64. In this case, since there is no vehicle with supplemented change, the content of the hazard list in the hazard list storage unit 128 is not updated and kept unchanged. After S64, the hazard list generation unit 124 returns the processing.
Returning to
Here, details of the risk evaluation processing will be described.
As illustrated in
Next, the risk evaluation unit 125 checks whether there is a stay in which the gaze stops for a predetermined period in the central field of view (571) and determines the presence or absence of the stay (S72). In the case where there is a stay in the central field of view (S72: YES), the risk evaluation unit 125 checks the object (nearby vehicle) in the central field of view from the objects in the hazard list on the basis of the position information stored in the hazard list storage unit 128 (S73).
Next, the risk evaluation unit 125 evaluates the risk for the object checked to be in the central field of view. Specifically, the risk evaluation unit 125 obtains the hazard information such as TTC according to the position information and the speed information regarding the object included in the hazard list in the hazard list storage unit 128. Next, the risk evaluation unit 125 clears the oblivion degree (resets the value to a value indicating no oblivion) for the object most recently checked in the central field of view, and then multiplies the oblivion degree by the hazard information to estimate the risk value (S74).
In the case where there is no stay in the central field of view (S72: NO), the risk evaluation unit 125 checks whether there is a stay in which the gaze stops for a predetermined period in the peripheral field of view (S75) and determines the presence or absence of the stay (S76).
In the case where there is no stay in the peripheral field of view (S76: NO), the risk evaluation unit 125 evaluates the risk for the object outside the field of view. Specifically, the risk evaluation unit 125 obtains the hazard information such as TTC according to the position information and the speed information regarding the object included in the hazard list in the hazard list storage unit 128. Next, the risk evaluation unit 125 reads the previous risk value from the risk table storage unit 129, increases the oblivion degree from the previous value as the object outside the field of view, and then multiplies the oblivion degree by the hazard information to estimate the risk value (S77).
In the case where there is a stay in the peripheral field of view (S76: YES), the risk evaluation unit 125 checks the object (nearby vehicle) in the peripheral field of view from the objects in the hazard list on the basis of the position information stored in the hazard list storage unit 128 (S78).
Next, the risk evaluation unit 125 evaluates the risk for the object checked to be in the peripheral field of view. Specifically, the risk evaluation unit 125 obtains the hazard information such as TTC according to the position information and the speed information regarding the object included in the hazard list in the hazard list storage unit 128. Next, the risk evaluation unit 125 reads the previous risk value from the risk table storage unit 129, maintains the oblivion degree (the same value as previous time) as the object most recently checked in the peripheral field of view, and then multiplies the oblivion degree by the hazard information to estimate the risk value (S79).
Following S74, S77, and S79, the risk evaluation unit 125 updates the risk table in the risk table storage unit 129 on the basis of the risk value estimated for the object (S80) and returns the processing.
Returning to
In the case where there is the object having a risk value that is equal or larger than a preset threshold (S25: YES), the output control unit 126 outputs by sounds the call attention message regarding the determined object via the sound output unit 131 to guide the driver to the call (526). In the case where there is no object having a risk value that is equal or larger than a preset threshold (S25: NO), the output control unit 126 returns the processing.
Specifically, the output control unit 126 acquires the position information of the object having the risk value that is equal to or larger than the present threshold by reference to the hazard list storage unit 128, and specifies the call attention message in which the “direction to be checked” in the call attention message and the position of the object match on the basis of the acquired position information.
As described above, in the case where the position of the object 314 having the risk value that is equal or larger than the preset threshold is “right side”, the output control unit 126 specifies a message with the “direction to be checked” being “right side” among the call attention messages notified from the information specifying unit 122.
As an example, assuming that messages with the ID of “1” to “3” among the call attention messages illustrated in
Next, the output control unit 126 outputs the specified call attention message as the call attention message regarding the object by sounds via the sound output unit 131. Thereby, the drive assist device 100 can provide the driver with the call attention message regarding the object having a low frequency of being included in the field of view of the driver and insufficiently recognized by the driver, for example. For example, the drive assist device 100 can provide the call attention message such as “caution, right” to guide the driver to the call for right side check. Of course, it goes without saying that emphasis of the meaning of words is possible by emphasizing punctuation.
Further, the output control unit 126 outputs a determination result message such as a clear sound as in “#5”, for the appropriate check action in “#4”. Thereby, the drive assist device 100 can notify the driver that the appropriate check action is being performed.
The risk evaluation processing may be performed on the basis of not only the gaze of the driver but also whether the driver has predicted (recognized) the risk for the object by performing the drive operation (accelerator reduction or the like) for the object.
Next, the risk evaluation unit 125 calculates the risk value according to the presence or absence of the risk prediction flag in S74, S77, and S79. Specifically, in the case where there is the risk prediction flag, the risk evaluation unit 125 calculates the risk value to be lower than the case of no risk prediction flag because the driver predicts (recognizes) the risk for the object.
Further, the output control unit 126 may cause the display device provided outside the vehicle to display the presence or absence of the check action on the basis of the results in the check action determination processing (S40 to S54).
Conversely, in the case where the appropriate check action having not been performed by the driver D is determined in the check action determination processing, the output control unit 126 sets the indicator light 132a to lighting display (for example, red display) indicating uncheck (S92). As described above, the indicator light 132a provided outside the vehicle outputs the display indicating the presence or absence of the check action by the driver D, whereby safety check can be performed between vehicles, for example.
In addition, the risk evaluation unit 125 may set a threshold for evaluating the risk value according to the drive mode (automatic drive mode, high-speed drive mode, leading vehicle following mode, manual drive mode, or the like) set by the driver D or the like. For example, immediately after the automatic drive mode is switched to the manual drive mode or the like, the cognitive activity may be a slow cognitive activity familiar to the automatic drive mode. Therefore, immediately after the automatic drive mode is switched to the manual drive mode or the like, the threshold may be set to be low to increase the frequency of outputting the call attention message. Further, since the cognitive activity as in the manual drive mode is not necessary in the automatic drive mode, the threshold may be set to be high to decrease the frequency of outputting the call attention message.
In the above-described first embodiment, a configuration of the drive assist device 100 alone has been illustrated. However, a server device having a similar function to the drive assist ECU 120 may be separately prepared from the drive assist device 100, and the drive assist device 100 and the server device connected to each other via a communication network may cooperate to realize the above-described functions.
With such a configuration of the drive assist system 500, the server device 501 has an advantage of mutually interpolating parts of respective pieces of information of the drive assist devices 100A, 1008, and the like. For example, in the hazard lists of vehicles close to each other, an object lacked in one hazard list may be interpolated from content of the other hazard list.
As described above, the drive assist device 100 includes a surrounding environment detection unit 116, the hazard list generation unit 124, a gaze detection unit 115, the risk evaluation unit 125, and an output control unit 126. The surrounding environment detection unit 116 detects an environment around a vehicle driven by a driver. The hazard list generation unit 124 generates a hazard list of an object to be a hazard on the basis of the detected environment. The gaze detection unit 115 detects a gaze of the driver. The risk evaluation unit 125 evaluates a risk regarding driving of the driver for each object included in the hazard list on the basis of a frequency at which the object included in the hazard list is included in a field of view of the driver based on the detected gaze. The output control unit 126 outputs drive assist information corresponding to the object having the evaluated risk that is equal to or larger than a threshold.
Thereby, the drive assist device 100 can suppress excessive drive assist and realize appropriate drive assist for the driver. For example, the drive assist device 100 does not provide the drive assist information for an object having a high frequency of being included in the field of view of the driver and sufficiently recognized by the driver so as not to disturb the cognitive activities of the driver for the object. In addition, the drive assist device 100 provides the drive assist information for an object having a low frequency of being included in the field of view of the driver, and insufficiently recognized by the driver, thereby suppressing the driver forgetting the object.
In addition, the risk evaluation unit 125 calculates an oblivion degree indicating the degree of forgetfulness of the driver about the object on the basis of whether the object included in the hazard list is included in the field of view of the driver based on the detected gaze. Then, the risk evaluation unit 125 evaluates a risk on the basis of the calculated oblivion degree and a collision time to the vehicle for each object included in the hazard list. The risk evaluation is performed on the basis of the oblivion degree of the driver about the object, that is, the recognition degree of the object, and the collision time when the object collides with the vehicle, that is, an objective risk of the object in this manner, whereby the risk evaluation of the object included in the hazard list can be appropriately performed.
Further, in a case where the object is included in a central field of view of the driver based on the gaze of the driver, the risk evaluation unit 125 resets a value of the oblivion degree of the object. Further, in a case where the object is included in a peripheral field of view of the driver based on the gaze of the driver, the risk evaluation unit 125 maintains the value of the oblivion degree of the object. Further, in a case where the object is not included in the central field of view and the peripheral field of view of the driver based on the gaze of the driver, the risk evaluation unit 125 increases the value of the oblivion degree of the object. By changing the evaluation of the oblivion degree of the driver for the object in the case where the object is not included in both the central field of view and the peripheral field of view of the driver in this manner, the risk evaluation of each object can be appropriately performed.
Note that the respective constituent elements of the illustrated apparatus and devices are not necessarily physically configured as illustrated in the drawings. That is, the specific aspects of separation and integration of each of the apparatus and devices are not limited to the illustrated aspects, and all or part of the apparatus or devices can be functionally or physically separated and integrated in any unit, in accordance with various loads, use status, and the like.
In addition, various processing functions performed by the drive assist device 100, 100A, 100B, or the server device 501 may be entirely or optionally partially executed on a central processing unit (CPU) (or a microcomputer, such as a microprocessor unit (MPU) or a micro controller unit (MCU)). In addition, it is needless to say that whole or any part of various processing functions may be executed by a program to be analyzed and executed on a CPU (or a microcomputer, such as an MPU or an MCU), or on hardware by wired logic. In addition, various processing functions performed by the server device 501 may be executed by a plurality of computers in cooperation though cloud computing.
The various processing functions described in the above embodiment can be realized by execution of a prepared program on a computer. This program may not be stored in a memory of the computer or the like. For example, the computer may read the program stored in a computer-readable storage medium and may execute the program. The computer-readable storage medium corresponds to, for example, a portable recording medium such as a CD-ROM, a DVD disk, and a universal serial bus (USB), a semiconductor memory such as a flash memory, a hard disk drive, and the like. Alternatively, the program may be prestored in a device connected to a public line, the Internet, a LAN, or the like, and the computer may read the program from the device and execute the program.
All examples and conditional language provided herein are intended for the pedagogical purposes of aiding the reader in understanding the invention and the concepts contributed by the inventor to further the art, and are not to be construed as limitations to such specifically recited examples and conditions, nor does the organization of such examples in the specification relate to a showing of the superiority and inferiority of the invention. Although one or more embodiments of the present invention have been described in detail, it should be understood that the various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the invention.
This application is a continuation application of International Application PCT/JP2017/020364 filed on May 31, 2017 and designated the U.S., the entire contents of which are incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2017/020364 | May 2017 | US |
| Child | 16694453 | US |
