Patent Grant
11458841
This application claims priority to and the benefit of Japanese Patent Application No. 2020-046808 filed on Mar. 17, 2020, the entire disclosure of which is incorporated herein by reference.
The present invention relates to a display control apparatus, a display control method, and a computer-readable storage medium storing a program with which an image can be displayed such that the image is superimposed on a visual field region of a driver.
Japanese Patent Laid-Open No. 2017-223666 describes that a POI in which an occupant is interested is specified using a direction of interest of the occupant obtained by an eye tracker and position information that is based on a GPS, and information indicating a name, an address, a description, business hours, contact details, and so on, of the specified POI is presented to the occupant. Japanese Patent Laid-Open No. 2014-216714 describes that only information extracted from information on an in-vehicle HUD in accordance with a first rule is transmitted to a passenger's mobile terminal.
The present invention provides a display control apparatus, a display control method, and a computer-readable storage medium storing a program for presenting information in which a driver is interested in to the driver, using an appropriate presentation method.
The present invention in its first aspect provides a display control apparatus including: a detection unit configured to analyze a line of sight of a driver of a vehicle, and detect a viewpoint of the driver in a visual field region of the driver that is obtained as a result of the analysis; a display control unit configured to display an image such that the image is superimposed on the visual field region, based on a result of the detection by the detection unit; and a transmission control unit configured to, if a condition is satisfied, transmit related information that is related to information represented by the image to an external terminal, in addition to or instead of the display control unit displaying the image.
The present invention in its second aspect provides a display control method including: analyzing a line of sight of a driver of a vehicle, and detecting a viewpoint of the driver in a visual field region of the driver that is obtained as a result of the analysis; displaying an image such that the image is superimposed on the visual field region, based on a detection result; and if a condition is satisfied, transmitting related information that is related to information represented by the image to an external terminal, in addition to or instead of displaying the image.
The present invention in its third aspect provides a computer-readable storage medium storing a program for causing a computer to perform functions of: analyzing a line of sight of a driver of a vehicle, and detecting a viewpoint of the driver in a visual field region of the driver that is obtained as a result of the analysis; displaying an image such that the image is superimposed on the visual field region, based on a detection result; and if a condition is satisfied, transmitting related information that is related to information represented by the image to an external terminal, in addition to or instead of displaying the image.
According to the present invention, information in which a driver is interested in can be presented to the driver using an appropriate presentation method.
Hereinafter, embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention, and limitation is not made to an invention that requires all combinations of features described in the embodiments. Two or more of the multiple features described in the embodiments may be combined as appropriate. Furthermore, the same reference numerals are given to the same or similar configurations, and redundant description thereof is omitted.
In Japanese Patent Laid-Open No. 2017-223666, it is possible that an occupant cannot sufficiently check the presented detailed information during driving. Japanese Patent Laid-Open No. 2014-216714 does not mention that information is transmitted to a driver's mobile terminal.
According to one aspect of the present invention, information in which a driver is interested can be presented to the driver using an appropriate presentation method.
The control apparatus in
Functions or the like assigned to the respective ECUs 20 to 29 will be described below. Note that the number of ECUs and functions assigned thereto can be designed as appropriate, and can be further segmented than in the present embodiment, or can be integrated.
The ECU 20 executes control associated with automated driving of the vehicle 1. During automated driving, the ECU 20 automatically controls at least either steering or acceleration/deceleration of the vehicle 1. In a later-described control example, the ECU 20 automatically controls both steering and acceleration/deceleration.
The ECU 21 controls an electric power steering device 3. The electric power steering device 3 includes a mechanism for steering front wheels in accordance with a driver's driving operation (steering operation) to a steering wheel 31. The electric power steering device 3 also includes a motor that exerts a driving force for assisting in the steering operation or automatically steering the front wheels, a sensor for detecting a steering angle, and so on. If the driving state of the vehicle 1 is automated driving, the ECU 21 automatically controls the electric power steering device 3 in response to an instruction from the ECU 20, and controls the traveling direction of the vehicle 1.
The ECUs 22 and 23 control detection units 41 to 43 for detecting the surrounding situation of the vehicle and perform information processing on their detection results. The detection units 41 are cameras (hereinafter referred to as “cameras 41” in some cases) for capturing images of the front of the vehicle 1. In the present embodiment, the detection units 41 are attached to the vehicle interior on the inner side of the windscreen, at a front portion of the roof of the vehicle 1. Analysis of the images captured by the cameras 41 makes it possible to extract an outline of a target and extract a lane marker (white line etc.) of a traffic lane on a road.
The detection units 42 are Light Detection and Ranging (LIDARs), and detect a target around the vehicle 1 and measure the distance to the target. In the present embodiment, five detection units 42 are provided, one on each corner of the front part of the vehicle 1, one at the center of the rear part, and one on each side of the rear part. The detection units 43 are millimeter wave radars (hereinafter referred to as “radars 43” in some cases), and detect a target around the vehicle 1 and measure the distance to the target. In the present embodiment, five radars 43 are provided, one at the center of the front part of the vehicle 1, one at each corner of the front part, and one on each corner of the rear part.
The ECU 22 controls one of the cameras 41 and the detection units 42 and performs information processing on their detection results. The ECU 23 controls the other camera 41 and the radars 43 and performs information processing on their detection results. As a result of two sets of devices for detecting the surrounding situation of the vehicle being provided, the reliability of the detection results can be improved. Also, as a result of different types of detection units such as cameras and radars being provided, manifold analysis of the surrounding environment of the vehicle is enabled.
The ECU 24 controls a gyroscope sensor 5, a GPS sensor 24b, and a communication device 24c, and performs information processing on their detection results or communication results. The gyroscope sensor 5 detects rotational motion of the vehicle 1. A path of the vehicle 1 can be determined based on the results of detection by the gyroscope sensor 5, the wheel speed, or the like. The GPS sensor 24b detects the current position of the vehicle 1. The communication device 24c wirelessly communicates with a server that provides map information, traffic information, and weather information, and acquires such information. The ECU 24 can access a database 24a of map information that is built in the storage device, and the ECU 24 searches for a route from the current location to a destination, for example. Note that a database of the aforementioned traffic information, weather information, or the like may also be built in the database 24a.
The ECU 25 includes a communication device 25a for vehicle-to-vehicle communication. The communication device 25a wirelessly communicates with other vehicles in the surrounding area and exchanges information between the vehicles.
The ECU 26 controls a power plant 6. The power plant 6 is a mechanism that outputs a driving force for rotating drive wheels of the vehicle 1, and includes, for example, an engine and a transmission. For example, the ECU 26 controls the output of the engine in response to the driver's driving operation (acceleration pedal operation or accelerating operation) detected by an operation detection sensor 7a provided on an acceleration pedal 7A, and switches the gear ratio of the transmission based on information such as vehicle speed detected by a vehicle speed sensor 7c. If the driving state of the vehicle 1 is automated driving, the ECU 26 automatically controls the power plant 6 in response to an instruction from the ECU 20 and controls acceleration/deceleration of the vehicle 1.
The ECU 27 controls lighting devices (headlight, tail light etc.) including direction indicators 8 (blinkers). In the example in
The ECU 28 controls an input/output device 9. The input/output device 9 outputs information to the driver and accepts input of information from the driver. A sound output device 91 notifies the driver of information using a sound. A display device 92 notifies the driver of information by means of a display of an image. The display device 92 is, for example, disposed in front of the driver seat and constitutes an instrument panel or the like. Note that although an example of using a sound and a display is described here, information may alternatively be notified using a vibration and/or light. Further, information may be notified by combining two or more of a sound, a display, a vibration, and light. Furthermore, the combination may be varied or the notification mode may be varied in accordance with the level (e.g., degree of urgency) of information to be notified. The display device 92 includes a navigation device.
An input device 93 is a switch group that is disposed at a position at which it can be operated by the driver and gives instructions to the vehicle 1, and may also include a sound input device.
The ECU 29 controls brake devices 10 and a parking brake (not shown). The brake devices 10 are, for example, disc brake devices and provided on the respective wheels of the vehicle 1, and decelerate or stop the vehicle 1 by applying resistance to the rotation of the wheels. For example, the ECU 29 controls operations of the brake devices 10 in response to the driver's driving operation (braking operation) of the driver detected by an operation detection sensor 7b provided on a brake pedal 7B. If the driving state of the vehicle 1 is automated driving, the ECU 29 automatically controls the brake devices 10 in response to an instruction from the ECU 20 and controls deceleration and stop of the vehicle 1. The brake devices 10 and the parking brake can also be operated to maintain the stopped state of the vehicle 1. If the transmission of the power plant 6 includes a parking lock mechanism, it can also be operated to maintain the stopped state of the vehicle 1.
A description will be given of control associated with automated driving of the vehicle 1 executed by the ECU 20. If an instruction of a destination and automated driving is given by the driver, the ECU 20 automatically controls the travel of the vehicle 1 to a destination in accordance with a guided route searched for by the ECU 24. During automated control, the ECU 20 acquires information (external information) associated with the surrounding situation of the vehicle 1 from the ECUs 22 and 23, and gives instructions to the ECUs 21, 26, and 29 based on the acquired information to control steering and acceleration/deceleration of the vehicle 1.
The external recognition unit 201 recognizes external information regarding the vehicle 1 based on signals from external recognition cameras 207 and external recognition sensors 208. Here, the external recognition cameras 207 are, for example, the cameras 41 in
The in-vehicle recognition unit 203 identifies an occupant of the vehicle 1 and recognizes the state of the occupant based on signals from an in-vehicle recognition camera 209 and an in-vehicle recognition sensor 210. The in-vehicle recognition camera 209 is, for example, an infrared camera installed on the display device 92 inside the vehicle 1, and detects a line-of-sight direction of the occupant, for example. The in-vehicle recognition sensor 210 is, for example, a sensor for detecting a biological signal of the occupant. The in-vehicle recognition unit 203 recognizes that the occupant is in a dozing state or a state of doing work other than driving, based on those signals.
The action planning unit 204 plans actions of the vehicle 1, such as an optimal path and a risk-avoiding path, based on the results of recognition by the external recognition unit 201 and the self-position recognition unit 202. The action planning unit 204 plans actions based on an entrance determination based on a start point and an end point of an intersection, a railroad crossing, or the like, and prediction of behavior of other vehicles, for example. The drive control unit 205 controls a driving force output device 212, a steering device 213, and a brake device 214 based on an action plan made by the action planning unit 204. Here, for example, the driving force output device 212 corresponds to the power plant 6 in
The device control unit 206 controls devices connected to the control unit 200. For example, the device control unit 206 controls a speaker 215 to cause the speaker 215 to output a predetermined sound message, such as a message for warning or navigation. Also, for example, the device control unit 206 controls a display device 216 to cause the display device 216 to display a predetermined interface screen. The display device 216 corresponds to the display device 92, for example. Also, for example, the device control unit 206 controls a navigation device 217 to acquire setting information in the navigation device 217.
The control unit 200 may also include functional blocks other than those shown in
A head-up display (HUD) control unit 218 controls a head-up display (HUD) 219 that is attached to the vehicle interior near the windscreen of the vehicle 1. The HUD control unit 218 and the control unit 200 can communicate with each other, and the HUD control unit 218 acquires, for example, captured image data obtained by the external recognition cameras 207 via the control unit 200. The HUD 219 projects an image onto the windscreen under the control of the HUD control unit 218. For example, the HUD control unit 218 receives captured image data obtained by the external recognition cameras 207 from the control unit 200, and generates image data to be projected by the HUD 219 based on the captured image data. This image data is, for example, image data to be overlapped (superimposed) with the landscape that can be seen from the driver through the windscreen. Due to the projection onto the windscreen by the HUD 219, the driver can feel that an icon image (destination information etc.) for navigation is overlapped with the landscape of a road ahead, for example. The HUD control unit 218 can communicate with an external device via a communication interface (I/F) 220. The external device is, for example, a mobile terminal 221 such as a smartphone held by the driver. The communication I/F 220 may be configured such that it can be connected to a plurality of networks, and may be, for example, configured such that it can be connected to the Internet.
Operations in the present embodiment will be described below.
It is desirable for the driver that information regarding a point of interest is displayed, but there may be a situation where the driver has to move the viewpoint away from that point depending on the driving situation. In the present embodiment, information regarding a point of interest is appropriately presented based on two methods, namely HUD display on the windscreen by the HUD 219 and transmission of the information to the driver's mobile terminal 221.
In the present embodiment, if it is detected that the driver has placed a viewpoint at a point for a predetermined time or more, HUD display is executed, and information is transmitted to the mobile terminal 221 held by the driver. With this configuration, information regarding a point of interest can be displayed for the driver during driving, and the driver can also check the information after the driving.
In step S101, the HUD control unit 218 analyzes the driver's line of sight. For example, the in-vehicle recognition unit 203 of the control unit 200 may analyze the driver's line of sight using the in-vehicle recognition camera 209 and an in-vehicle recognition sensor 210, and the HUD control unit 218 may acquire the analysis results.
In step S102, the HUD control unit 218 detects the driver's viewpoint on the windscreen. That is to say, the HUD control unit 218 specifies the coordinates of the viewpoint in the visual field region 300 based on the analysis results obtained in step S101. For example, the HUD control unit 218 specifies the coordinates of the viewpoint in the visual field region 300 to be subjected to HUD display, based on captured image data that corresponds to the visual field region 300 and is obtained by the external recognition camera 207. For example, the HUD control unit 218 may detect a viewpoint based on an overlap of viewpoints on AR content displayed in the visual field region 300. Here, an overlap of viewpoints may be determined based on an area around the center of the AR content, or may be determined based on a predetermined region including the surroundings of the AR content. The predetermined region may be determined in accordance with the number of pieces of other displayed AR content or the details thereof.
In step S103, the HUD control unit 218 performs later-described display control. After step S103, processing from step S101 is repeated.
In step S202, the HUD control unit 218 determines whether or not a predetermined time has elapsed. Here, if it is determined that the predetermined time has not elapsed, it is decided that the position of the viewpoint is not a point of interest of the driver, and the processing in
In step S203, the HUD control unit 218 generates information (HUD display information) to be displayed on the windscreen by the HUD 219. For example, the HUD control unit 218 extracts information corresponding to the point of interest, based on the current position of the vehicle 1 and the map information acquired from the control unit 200, and object information corresponding to the position of the viewpoint on the captured image data. The HUD display information is, for example, a name such as that indicated as the information 301 or the information 302 in
The content of the HUD display information is not limited to a name, but any other information may also be added thereto. For example, the content of the HUD display information is not limited to name information obtained from the map information, and related information that may be acquired from the Internet may also be added. For example, the HUD control unit 218 may acquire related information from the Internet based on a name. At this time, for example, in the case of the name of a restaurant such as the information 301, information such as business hours, SNS information, a genre such as Japanese food or Western food, and evaluation of this restaurant may be acquired and displayed together with the name. Also, the HUD control unit 218 may newly generate related information based on the information acquired from the Internet. For example, information indicating whether the restaurant is open or closed may be generated as related information based on the business hours acquired from the Internet and information on the current date and time information. The related information may also be generated based on the external recognition results obtained by the external recognition unit 201. For example, congestion information or the like may be generated as related information based on the image recognition results, such as an image showing that there is a waiting line of people in front of a restaurant corresponding to the information 301.
The amount of related information may be changeable. For example, if it is inferred that the driver is more interested based on the driver's preference information, the related information may be increased. The driver's preference may be analyzed in advance by an external server by pairing the driver's mobile terminal 221 with the vehicle 1 (control unit 200), for example. For example, the server analyzes and acquires the driver's eating and drinking preference based on search history information on the Internet of the mobile terminal 221 or the like. Also, a configuration may be employed in which each configuration of the above-described HUD display information can be set through user settings.
In step S204, the HUD control unit 218 displays the HUD display information generated in step S203 based on the coordinates of the viewpoint of which a stationary state is detected in step S201. For example, the HUD control unit 218 executes HUD display so as not to overlap with a travel path in the visual field region 300 via a lead line from the coordinates of the viewpoint of which a stationary state is detected in step S201.
In step S205, the HUD control unit 218 generates information (terminal display information) to be displayed on the mobile terminal 221 held by the driver. The terminal display information may be the same information as, or may differ from, the HUD display information generated in step S203. For example, the terminal display information may be information related to the HUD display information generated in step S203.
The terminal display information may also include the following information that is less relevant to the content of the HUD display information. For example, information regarding the time for which the area of the HUD display information was stared at may be included. Position information regarding the vehicle 1 and navigation information at the time when the area of the HUD display information was stared at may also be included. For example, a destination and a goal of the vehicle 1 and path information may be included as navigation information. Weather information and time information may also be included. As a result of the terminal display information thus including the aforementioned information, the driver can readily recall at which point the driver saw the information, based on the information displayed on the mobile terminal 221 after driving.
The terminal display information may also be generated as information that is simplified HUD display information. For example, if it is inferred that the driver is not interested, as a result of analyzing a face image and a voice of the driver after the HUD display information has been displayed in step S204, the terminal display information may be generated as simplified HUD display information, e.g., a name only. The results of analyzing a face image and a voice of the driver may be obtained by acquiring the recognition results obtained by the in-vehicle recognition unit 203. Furthermore, a configuration may be employed in which the terminal display information is not generated depending on the results of analyzing a face image and a voice of the driver after the HUD display information has been displayed in step S204.
In step S206, the HUD control unit 218 transmits the terminal display information generated in step S205 to the mobile terminal 221 held by the driver. For example, the terminal display information may be transmitted by e-mail. Here, along with the transmission of the terminal display information, a sound notification indicating that the information has been transmitted may be given to the driver. The sound notification may be given by the speaker 215, or may be given by the mobile terminal 221. After step S206, the processing in
Thus, according to the present embodiment, if an area at which the driver places a viewpoint for a predetermined time or more is detected, HUD display is executed, and information is transmitted to the mobile terminal 221 held by the driver. With this configuration, information regarding a point of interest can be displayed for the driver during driving, and the driver can also check the information after the driving.
Although the present embodiment has described that the mobile terminal 221 is a mobile terminal held by the driver, it may alternatively be a mobile terminal held by a passenger other than the driver, e.g., a person in a passenger seat or a back seat. Also, the mobile terminal 221 need not be present in the vehicle 1. That is to say, a PC or the like that is outside the vehicle 1, e.g., that is present in the driver's house may be used as the mobile terminal 221 in the present embodiment due to being paired with the vehicle 1 (control unit 200). Also, a device mounted in the vehicle 1 may be used as the mobile terminal 221. For example, a display audio device or a meter display mounted in the vehicle 1 may be used as the mobile terminal 221.
Although the present embodiment has described a configuration in which the driver's line of sight is analyzed and the terminal display information is transmitted to the driver's mobile terminal 221, the person whose line of sight is to be analyzed is not limited to the driver, and a configuration may alternatively be employed in which a line of sight of a passenger other than the driver, such as a person in a passenger seat or a back seat, is analyzed, and information is transmitted to a corresponding (paired) mobile terminal. At this time, the terminal display information may be transmitted to any of the aforementioned various devices, rather than the mobile terminal.
Although the present embodiment has described that whether or not a predetermined time has elapsed is determined in step S202 after a stationary state of a viewpoint has been detected in step S201, this predetermined time may be changeable. For example, if the current time information is 11:00 to 14:00, it is inferred that the driver is having lunch, and therefore the predetermined time is set shorter than that in other time periods. With this configuration, while the driver is moving the viewpoint over a plurality of points to search for a restaurant, HUD display can be readily executed at a point at which the driver stops the viewpoint with interest. Also, for example, the predetermined time may be changeable based on the driver's preference. For example, upon a stationary state of the viewpoint being detected in step S201, information corresponding to the point of interest, as mentioned above. If it is inferred that that information matches the driver's preference; for example, if there is a signboard of “ABC golf” and it has been analyzed that the driver likes golf, the predetermined time is set shorter.
Whether or not to execute the operations in the present embodiment may be set in accordance with a control state of the vehicle 1. For example, the operations in the present embodiment may be executed in the case of an eyes-off automated driving mode.
The second embodiment will be described below regarding differences from the first embodiment. In the present embodiment, HUD display is executed if it is detected that the driver places a viewpoint at a point for a predetermined time or more. On the other hand, if the time for which the driver places the viewpoint at a point is less than the predetermined time, information is transmitted to the mobile terminal 221 held by the driver. With this configuration, if, for example, the driver moves the viewpoint away from a point in a short time, the driver can check the information after driving.
In step S303, the HUD control unit 218 determines whether or not a second predetermined time has elapsed. Note that the second predetermined time is set longer than the first predetermined time. If it is determined that the second predetermined time has elapsed, in step S304, the HUD control unit 218 generates HUD display information, and in step S305, the generated HUD display information is displayed based on the coordinates of the viewpoint of which a stationary state is detected in step S301. Thereafter, the processing in
If it is determined in step S303 that the second predetermined time has not elapsed, in step S306, the HUD control unit 218 generates terminal display information. The terminal display information may be the same as, or may differ from, the HUD display information generated in step S304. For example, the terminal display information may be information related to the HUD display information generated in step S304. In step S307, the HUD control unit 218 transmits the terminal display information generated in step S306 to the mobile terminal 221 held by the driver. For example, the terminal display information may be transmitted by e-mail. Here, along with the transmission of the terminal display information, a sound notification indicating that the information has been transmitted may be given to the driver. The sound notification may be given by the speaker 215, or may be given by the mobile terminal 221. After step S307, the processing in
Thus, according to the present embodiment, if an area at which the driver places a viewpoint for a predetermined time or more is detected, HUD display is executed. On the other hand, if the time for which the driver places the viewpoint to that area is less than the predetermined time, HUD display is not executed, and information is transmitted to the mobile terminal 221 held by the driver. Also, as described as the first predetermined time, as for an area at which the driver does not place a viewpoint for a fixed time, it is decided that the driver is not interested in this area, and the HUD display information and the terminal display information are not generated. With this configuration, if, for example, it can be decided that the driver is interested in an area but the driver moves the viewpoint away therefrom in a short time, the driver can check information regarding the area after driving.
The third embodiment will be described regarding differences from the first and second embodiments. In the present embodiment, even if the time for which the viewpoint is placed at a point is less than a predetermined time, if there is a specific reason, information is transmitted to the mobile terminal 221 held by the driver. If there is no specific reason, information is not transmitted to the mobile terminal 221. With this configuration, in a situation where the viewpoint needs to be necessarily moved away, e.g., for the purpose of risk avoidance during driving, information is transmitted to the mobile terminal 221. With this configuration, the driver can check that information after driving.
In step S403, the HUD control unit 218 determines whether or not a second predetermined time has elapsed. Note that the second predetermined time is set longer than the first predetermined time. If it is determined that the second predetermined time has elapsed, in step S404, the HUD control unit 218 generates HUD display information, and in step S405, the generated HUD display information is displayed based on the coordinates of the viewpoint of which a stationary state is detected in step S401. Thereafter, the processing in
If it is determined in step S403 that the second predetermined time has not elapsed, in step S406, the HUD control unit 218 determines whether or not a specific event has occurred before the second predetermined time elapses. For example, it is determined whether or not there has been, as a specific event, a sudden change in the behavior of the vehicle 1 (sudden acceleration, sudden deceleration etc.), a change in the amount of steering, or a hand-off warning notification. If it is determined that any of these events has occurred, it is decided that the driver has necessarily moved the viewpoint away from the point of interest, and the processing proceeds to step S407. On the other hand, if it is determined that no specific event has occurred, it is decided that information corresponding to the position of this viewpoint is not information that the driver is to check with the mobile terminal 221 after driving, and the processing in
Note that, in step S406, processing other than determination of whether or not a specific event has occurred may be performed. For example, a reason that the driver has moved the viewpoint away may be inferred based on the behavior of the vehicle 1, the recognition results obtained by the external recognition unit 201, the map information, or the like. For example, when another vehicle or a moving object is approaching, or a near-miss point is recognized, it may be decided that the driver has moved the viewpoint away due to risk avoidance during driving, and the processing may proceed to S407. Also, if the recognition results obtained by the in-vehicle recognition unit 203, e.g., a state of the driver himself (an operation of a meter, a predetermined gesture etc.) is detected, the processing may proceed to step S407.
In step S407, the HUD control unit 218 generates terminal display information. The terminal display information may be the same as, or may differ from, the HUD display information generated in step S404. For example, the terminal display information may be information related to the HUD display information generated in step S404. In step S408, the HUD control unit 218 transmits the terminal display information generated in step S407 to the mobile terminal 221 held by the driver. For example, the terminal display information may be transmitted by e-mail. Here, along with the transmission of the terminal display information, a sound notification indicating that the information has been transmitted may be given to the driver. The sound notification may be given by the speaker 215, or may be given by the mobile terminal 221. After step S408, the processing in
Thus, according to the present embodiment, even if the time for which the viewpoint is placed at a point is less than a predetermined time, if there is a specific reason (e.g., a specific event has occurred), information is transmitted to the mobile terminal 221 held by the driver. On the other hand, if there is no specific reason, information is not transmitted to the mobile terminal 221. With this configuration, in a situation where the viewpoint needs to be necessarily moved away, e.g., for the purpose of risk avoidance during driving, information is transmitted to the mobile terminal 221. Thus, the driver can check the information after driving.
The fourth embodiment will be described below regarding differences from the first to third embodiments. In the present embodiment, if it is detected that the driver has not been able to check the content of the information, in a predetermined time after HUD display was executed and then the display disappeared, information is transmitted to the mobile terminal 221 held by the driver. With this configuration, if it is decided that the driver has not been able to sufficiently check information displayed by the HUD, information is transmitted to the mobile terminal 221 such that the driver can check this information after driving.
In step S503, the HUD control unit 218 generates HUD display information, and in step S504, the generated HUD display information is displayed based on the coordinates of the viewpoint of which a stationary state is detected in step S501. Then, upon the HUD display information being displayed, the HUD control unit 218 starts measurement using a timer function.
In step S505, the HUD control unit 218 determines whether or not the HUD display information displayed in step S504 has been checked by the driver. For example, if it is determined that the display has disappeared for some reason before a predetermined time elapses after the HUD display information is displayed in step S504, the HUD control unit 218 determines that the HUD display information has not been checked by the driver. Also, the HUD control unit 218 may perform the determination based on the recognition results obtained by the in-vehicle recognition unit 203. For example, whether or not the information has been checked by the driver may be determined based on the results of analyzing the driver's image and voice. The driver's image is, for example, an image of the driver's gesture. At this time, a gesture (such as shaking the palm) meaning that the driver could not check the information may be predetermined, and the determination may be performed based on whether or not the driver has performed this gesture. Also, the determination may be performed based on whether or not the time for which the driver places the viewpoint at the HUD display information is a predetermined time or more. If it is determined in step S505 that the information has been checked by the driver, the processing in
In step S506, the HUD control unit 218 generates terminal display information. The terminal display information may be the same as, or may differ from, the HUD display information generated in step S503. For example, the terminal display information may be information related to the HUD display information generated in step S503. In step S507, the HUD control unit 218 transmits the terminal display information generated in step S506 to the mobile terminal 221 held by the driver. For example, the terminal display information may be transmitted by e-mail. Here, along with the transmission of the terminal display information, a sound notification indicating that the information has been transmitted may be given to the driver. The sound notification may be given by the speaker 215, or may be given by the mobile terminal 221. After step S507, the processing in
Thus, according to the present embodiment, if it is detected that the driver has not been able to checked the content of display in a predetermined time after the HUD display was executed and this display disappeared, information is transmitted to the mobile terminal 221 held by the driver. With this configuration, if it is decided that the driver has not been able to sufficiently check information displayed by the HUD, information is transmitted to the mobile terminal 221 such that the driver can check this information after driving.
Each of the above embodiments may be selected as an operation mode by the driver. In this case, corresponding predetermined time in
A display control apparatus of one of the above embodiments includes: a detection unit (209, 210, 203, S102) configured to analyze a line of sight of a driver of a vehicle, and detect a viewpoint of the driver in a visual field region of the driver that is obtained as a result of the analysis; a display control unit (218) configured to display an image such that the image is superimposed on the visual field region, based on a result of the detection by the detection unit; and a transmission control unit (
With this configuration, information in which the driver is interested can be presented to the driver using an appropriate presentation method.
As the condition, if the detection unit has detected a stationary state of the viewpoint for a first predetermined time, the transmission control unit transmits the related information to the external terminal in addition to the display control unit displaying the image.
With this configuration, for example, information in which the driver is interested can be displayed by the HUD and also transmitted to the driver's mobile terminal.
As the condition, if the detection unit does not detect a stationary state of the viewpoint for a second predetermined time, the transmission control unit transmits the related information to the external terminal instead of the display control unit displaying the image. If the detection unit has detected the stationary state of the viewpoint for the second predetermined time, the display control unit displays the image, and the transmission control unit does not transmit the related information to the external terminal. The condition is that the detection unit has further detected the stationary state of the viewpoint for a third predetermined time that is shorter than the second predetermined time.
With this configuration, if, for example, the driver cannot sufficiently see the information in which the driver is interested, the information can be transmitted to the driver's mobile terminal such that the driver can check the information after driving.
The display control apparatus further includes a determination unit (S406) configured to determine whether or not a specific event has occurred if the detection unit does not detect the stationary state of the viewpoint for the second predetermined time, wherein if it is determined by the determination unit that the specific event has occurred, the transmission control unit transmits the related information to the external terminal, instead of the display control unit displaying the image.
With this configuration, if, for example, the driver cannot sufficiently see information in which the driver is interested due to unavoidable reasons such as risk avoidance, the information can be transmitted to the driver's mobile terminal such that the driver can check the information after driving.
If it is determined by the determination unit that the specific event has not occurred, the transmission control unit does not transmit the related information to the external terminal.
With this configuration, if, for example, it is decided that the driver is not interested in information, a configuration in which the information is not transmitted to the driver's mobile terminal can be employed.
The display control apparatus further includes a decision unit (S505) configured to decide whether or not the image has been checked by the driver, after the image was displayed by the display control unit, wherein, as the condition, if it is decided by the decision unit that the image has not been checked by the driver, the transmission control unit transmits the related information to the external terminal. If it is decided by the decision unit that the image has been checked by the driver, the transmission control unit does not transmit the related information to the external terminal. The decision unit decides whether or not the image has been checked by the driver, based on whether or not the detection unit has detected the stationary state of the viewpoint for a fourth predetermined time.
With this configuration, if, for example, it is decided that information in which the driver is interested has not been checked by the driver, the information can be transmitted to the driver's mobile terminal such that the driver can check the information after driving.
The external terminal is a mobile terminal held by the driver. With this configuration, for example, information can be transmitted to the driver's mobile terminal.
The invention is not limited to the foregoing embodiments, and various variations/changes are possible within the spirit of the invention.
| Number | Date | Country | Kind |
|---|---|---|---|
| JP2020-046808 | Mar 2020 | JP | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 7642931 | Sato | Jan 2010 | B2 |
| 8134594 | Nagamine et al. | Mar 2012 | B2 |
| 8953011 | Lang et al. | Feb 2015 | B2 |
| 9513702 | Cho | Dec 2016 | B2 |
| 10456072 | Okabe et al. | Oct 2019 | B2 |
| 10868976 | Kuehne | Dec 2020 | B2 |
| 20050107952 | Hoshino | May 2005 | A1 |
| 20070229310 | Sato | Oct 2007 | A1 |
| 20100128128 | Nagamine et al. | May 2010 | A1 |
| 20100134340 | Sotomaru | Jun 2010 | A1 |
| 20120200664 | Lang et al. | Aug 2012 | A1 |
| 20150015479 | Cho | Jan 2015 | A1 |
| 20150031352 | Yi | Jan 2015 | A1 |
| 20170350718 | Schulz | Dec 2017 | A1 |
| 20180088323 | Bao et al. | Mar 2018 | A1 |
| 20180177446 | Okabe et al. | Jun 2018 | A1 |
| 20180288337 | Kuehne | Oct 2018 | A1 |
| 20200051529 | Higashiyama et al. | Feb 2020 | A1 |
| 20200269759 | Watanabe et al. | Aug 2020 | A1 |
| 20210260999 | Masuya et al. | Aug 2021 | A1 |
| Number | Date | Country |
|---|---|---|
| 2005-135037 | May 2005 | JP |
| 2014216714 | Nov 2014 | JP |
| 2017-223666 | Dec 2017 | JP |
| 2016166791 | Oct 2016 | WO |
| Entry |
|---|
| Office Action for U.S. Appl. No. 17/185,113 dated Nov. 9, 2021 (Related Case). |
| File history of related U.S. Appl. No. 17/185,113, filed Feb. 25, 2021 (Related Case). |
| Number | Date | Country | |
|---|---|---|---|
| 20210291653 A1 | Sep 2021 | US |
