The present application claims priority to European Patent Application No. 23 218 534.8, entitled “A DRIVER ASSISTANCE SYSTEM”, and filed on Dec. 20, 2023. The entire contents of the above-listed application is hereby incorporated by reference for all purposes.
The disclosure relates to a driver assistance system, in particular to a driver assistance system for a vehicle.
Driver assistance may include any relief that is provided to an individual associated with a vehicle to increase individual protection and enhance driver experience. For instance, a driver of a vehicle may want to obtain information about locations of interest in a surrounding environment of the vehicle. When a driver retrieves such information by performing a search on a mobile device, for example, the driver may be distracted when performing the search while driving. If a driver stops in order to safely perform a search on a mobile device, this will delay the driver's arrival at a final destination. Instead of or in addition to obtaining information about locations or objects of interest, a driver may desire to select any other objects in a surrounding environment and, e.g., communicate with the selected object. This may require a driver to perform unauthorized and/or potentially dangerous actions.
There is a need for a driver assistance system that is able to reliably identify an object of interest and to subsequently perform a desired action in a safe manner in order to prevent accidents.
The driver assistance system and related method according to the various embodiments described herein provide a mechanism that allows to search, identify, and select one or more targets of interest in the surrounding environment of a vehicle. In order to do so, a driver of a vehicle does not have to perform hazardous actions such as, e.g., picking up their phone or handling a touch display of the vehicle while driving. The driver assistance system and related method are very user friendly and provide a very satisfying user experience. At the same time, road safety is significantly increased.
A driver assistance system for a vehicle is disclosed herein. The driver assistance system includes a processing unit, a memory, a user interface, an external camera unit, and a display unit, wherein the user interface is configured to receive a first user input from an occupant of the vehicle and transmit the first user input to the processing unit, the processing unit is configured to process the first user input and transmit a corresponding request to the external camera unit, the external camera unit, in response to the request, is configured to capture one or more images of a surrounding environment of the vehicle, the processing unit is further configured to, based on the first user input, identify one or more objects in the one or more images captured by the external camera unit, the display unit is configured to display the one or more images captured by the external camera unit, and to highlight the one or more identified objects in the displayed one or more images, the user interface is further configured to receive a second user input from the occupant of the vehicle, the second user input including a selection of at least one of the one or more highlighted objects, and the processing unit is further configured to cause an action to be performed in response to the second user input.
A method according to embodiments of the disclosure includes receiving a first user input from an occupant of a vehicle; in response to the first user input, obtaining one or more images of a surrounding environment of the vehicle captured by an external camera unit; based on the first user input, identifying one or more objects in the one or more images; annotating the one or more images to highlight the one or more objects; displaying at least one of the one or more annotated images by means of a display unit; receiving a second user input from the occupant of the vehicle, the second user input including a selection of at least one of the one or more highlighted objects; and causing an action to be performed in response to the second user input.
Other systems, features and advantages of the disclosure will be or will become apparent to one with skill in the art upon examination of the following detailed description and figures. It is intended that all such additional systems, methods, features and advantages included within this description, be within the scope of the invention and be protected by the following claims.
The arrangements may be better understood with reference to the following description and drawings. The components in the figures are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention. Moreover, in the figures, like referenced numerals designate corresponding parts throughout the different views.
The processing unit 102 is further configured to, based on the first user input, identify one or more objects in the one or more images captured by the external camera unit 106. For example, as will be described herein, the processing unit 102 may be configured to apply a segmentation algorithm, a machine learning model, or the like to identify one or more objects in an image and may be configured to annotate the images, such as highlighting the identified object(s) with a colored box and/or corresponding number. The display unit 108 is configured to display the one or more annotated images captured by the external camera unit 106.
User inputs may include voice prompts, for example, to start the process of obtaining images, identifying objects, and displaying annotated images. In some examples, secondary inputs such as eye gaze tracking, as determined by an internal vehicle camera, may be additionally used to provide additional information to a prompt, like a direction in which the speaking occupant is looking, thus indicating a region to be searched to identify objects. Further, user inputs may also include voice prompts, eye tracking, or touch inputs to the display to select at least one of the one or more highlighted objects displayed in an annotated image. For example, the processing unit 102 may identify more than one object in an image in response to a first user input. A second user input may then be received that selects one of the more than one object, the selected one being the object to which the occupant wishes action be taken about. Such actions may include obtaining and displaying additional information about an object (e.g., a building, geographic feature, etc.), transmitting a message to a vehicle, road-side unit, emergency services, or the like, and adjusting one or more vehicle operation states. For example, an advanced driver assistance system (ADAS) may be adjusted to change an adaptive cruise control setting, to turn on lane assist protocols, or the like, in response to identification of an object, such as a swerving car.
According to one example, the selected at least one object may be a place or object of interest (e.g., a building, tourist attraction, public place, etc.). In this case, the action may comprise at least one of providing information about the selected at least one object to the occupant of the vehicle, and adding a location of the selected at least one object as a destination in a navigation system of the vehicle. This will be described in further detail by means of several examples below.
According to another example, the selected at least one object may be a vehicle. In this case, the action may correspond to the first user input (e.g., the initial user prompt). For example, a voice prompt regarding a vehicle's headlights being off may indicate an action of transmitting a message to said vehicle to turn on their headlights. As another example, a voice prompt regarding a vehicle's driving state (e.g., excess speed, swerving, etc.) may indicate an action of changing ADAS settings and/or transmitting a message to emergency services or other appropriate authority regarding the vehicle's driving behavior and the current location.
In some examples, content of the first user input may inform which images are processed by the processing unit. For example, the first user input may specify a region relative to the vehicle or an internal vehicle camera may track the occupant's eye gaze at the time when the occupant voices the prompt. As an example, an occupant may ask “What is that building in front of us on the right?”. The processing unit may receive the first user input and identify which cameras of the external camera unit 106 to activate to acquire images or which images from memory to obtain that correspond to the indicated region. As another example, an occupant may provide the first user input, for example by voice, while looking out a window in a particular direction. Internal cameras equipped with eye-gaze tracking capabilities may detect the eye gaze of the occupant at the time of the first user input (e.g., either in real-time or retroactively to account for the time it takes for the occupant to provide the input and for the processing unit to receive and process the input). The processing unit may receive the first user input as well as eye-gaze tracking data and may identify which cameras of the external camera to activate to acquire images in the particular direction or to obtain previously acquired images acquired in the particular direction.
In this way, the number of images that are processed by the processing unit may be reduced. By filtering obtained images according to the content of the user input to identify relevant images, overall processing efficiency may be increased, thereby decreasing the time needed to generate an output (e.g., an annotated image for display).
The driver assistance system 100 may then present one or more of the obtained relevant images on a display of the vehicle and highlight the one or more identified objects in the displayed one or more images, as exemplarily illustrated in
In some examples, the location of the occupant voicing the prompt may affect how the images are displayed. For example, when microphones in the driver zone or front passenger zone detect the voice prompt, the images may be displayed in a first manner, such as on the infotainment head unit. Conversely, when microphones in a rear passenger zone detect the voice prompt, the images may be displayed on a posterior headrest display for viewing by the rear passengers. In this way, the output of the driver assistance system 100 may be targeted to the specific occupant who provided the prompt.
In the example illustrated in
Alternatively or additionally, an occupant gaze determined to be lingering at the desired object (e.g., for at least 1.5-2 seconds) may also be taken as selection of the desired object. This gaze may be captured by means of an internal camera unit, for example (e.g., eye-pupil-tracking camera). That is, a direction of view of the occupant may be identified, followed by an analysis by means of suitable target detection algorithms that are able to identify a target the occupant is gazing at. Alternatively or additionally, a specific object may be selected by pointing at it (e.g., for at least 1.5-2 seconds). A hand (i.e. a finger) of the occupant may be captured by means of an internal camera unit and the direction the occupant is pointing at may be determined by means of suitable image processing algorithms, for example. Alternatively or additionally, an object may be selected by means of electroencephalography. In order to be able to do so, an occupant may be required to wear a special cap comprising sensors connected by wires, the sensors being in direct contact with the occupant's head. However, brain signals may be captured in any other suitable way, e.g., remotely by means of sensors arranged close to (but not in direct contact with) an occupant's head, e.g., in a headrest of the vehicle or in the roof above the head of the occupant.
According to one example, an object the occupant is gazing/pointing at or thinking of may be additionally highlighted (e.g., by increased brightness, by using a special color, by thickening the line forming the frame, by flashing the frame, etc.). If the additionally highlighted object corresponds to the object the occupant has selected, the occupant can confirm the choice, for example. This can be done in any suitable way such as, e.g., continuing to gaze at the object, pressing a specific button on the steering wheel, on a specific control panel, or on the dashboard, by means of voice confirmation (e.g., by saying “yes” or “select”), etc. Such confirmation, however, is optional. Requesting confirmation from the driver/occupant, however, increases the accuracy of the selection, and may further decrease the risk of distracting the driver of a vehicle as a result of an erroneous recognition of the occupant's selection.
In the example described with respect to
Now referring to
It is generally also possible that no object can be identified in the one or more images which matches the occupant's request. In this case, the occupant can be informed that no matches could be found. The occupant may then reformulate their request, for example. In such examples, the driver assistance system 100 may reprocess images from the time of the first user input. As an example, the subject of the occupant's request may have passed by the time the reformulated request is provided. It is also possible that the occupant requests the driver assistance system that all objects in the one or more images be highlighted, even if they have not been identified to match the request.
Once a specific object has been selected, an action is performed. In examples in which the object is an environmental feature, such as a building, a geographic feature, a tourist attraction, or the like, the action may comprise providing information about the specific object to the occupant(s) of the vehicle. For example, the driver assistance system 100 may output information about the object in any suitable way (e.g., spoken or in writing on a display). Alternatively or additionally, the action may comprise transmitting a location of the selected object to a navigation system and adding the location as a destination. In yet further examples, the action may comprise memorizing a selected object (target) in the driver assistance system for later retrieval. That is, an occupant may not receive information about the object while driving. The object may be memorized such that information about the object can be presented to the driver/occupant at a later point in time.
According to another example, the selected at least one object may be a vehicle or person, for example. This will be described by means of different examples in the following. Referring to
That is, a message may be transmitted to the second vehicle, e.g., by means of inter-vehicle communication. For example, a message may be transmitted stating “Your lights are off”, or similar. The driver of the second vehicle may react to such a message, e.g., by switching on their lights, and/or by answering by transmitting a response. Instead of or in addition to transmitting a message to another vehicle, one or more images captured by the external camera unit 106 may be stored by the driver assistance system 100 for later retrieval. It is even possible, that a message to an appropriate authority is transmitted. If, for example, an occupant of the vehicle recognizes a hazardous driving behavior of a second vehicle, they may want to inform the police about it. This may also apply, for example, if an occupant of the vehicle witnesses a crime or heavy violation of traffic regulations, for example.
If, for example, an occupant of the vehicle witnesses a crime or the behavior of a vehicle is dangerous (e.g., swerving, excess speed, etc.), they may request to the driver assistance system 100 “Do you see that person wearing a yellow jacket and carrying a black bag?” or similar. The driver assistance system 100 may then identify and highlight any vehicles or persons present in the surrounding environment of the vehicle, similar to what has been described with respect to the objects, places of interest, and vehicles above. An occupant of the vehicle may select one or more of a plurality of highlighted persons or vehicles in the one or more images, or may confirm that one person or vehicle highlighted in the one or more images is the correct person or vehicle. Images may be transmitted to an authority or emergency service (e.g., the police) instantaneously, or may be stored in the system for later transmission (e.g., if an authority requests further evidence of a reported crime or unauthorized behavior). It is even possible that the driver assistance system 100 in such cases continues capturing images or even capture video footage, in order to collect further evidence, as long as the concerned person or vehicle is within the field of view of the external camera unit 106. When one or more images are transmitted to an authority, an exact geolocation of the position at which the one or more images had been captured may additionally be transmitted.
According to an even further example, one or more images or video data captured within a certain time frame before the occupant provided first user input may be stored for later retrieval and/or transmission to an authority. An occupant may request the driver assistance system 100 “Please memorize the black vehicle in front of us and track its behavior”, for example. The concerned vehicle may then be identified in one or more images captured by means of the external camera unit 106, highlighted, and, after a respective selection has been provided by the occupant, the requested action may be performed, similar to what has been described above.
In such an example, in response to the user request for tracking the behavior, images acquired by the external camera unit may be stored in non-transitory memory or other longer-term memory than other images acquired. As described above, images acquired by the external camera unit may be stored in temporary or otherwise transitory memory for a predefined length of time. In some examples, the buffer length of time may be adjusted based on the speed of the vehicle. For example, at a first, slower speed, the buffer of time for which images are stored in memory may be a first time, and at a second, higher speed, the buffer of time may be a second time that is longer than the first time. For example, at the first slower speed an object may be passed by slower than at the second higher speed. As such, the processing unit may retrieve images from longer ago in the case of higher speed to find images in which the relevant object may be identified. Conversely however, when the user requests a particular object (e.g., another vehicle driving in front of occupant) be tracked over time, images acquired before the request (e.g., extending the length of the buffer) and images acquired after the request may be stored in non-transitory memory.
Another alternative or additional action that may be performed in response to a second user input from the occupant comprises identifying a license plate of another vehicle. If, for example, an occupant experiences another vehicle performing unauthorized and/or hazardous driving actions, they may identify the concerned vehicle in the one or more images captured by the external camera unit 106, similar to what has been described above, and the license plate may subsequently be identified in the one or more images. The identified license plate number may be stored and/or transmitted to an authority, for example. It may be possible that a license plate of another vehicle cannot be identified on the one or more captured images, e.g., at night time. The driver assistance system 100, therefore, may additionally comprise a night vision camera, for example, in order to be able to identify license plates also at night. It is, however, generally also possible that license plates are not identified by means of the one or more captured images. For example, it is also possible that vehicles transmit specific vehicle IDs between each other, e.g., by means of transponders. Such vehicle IDs could be the same or different than the license plate. A vehicle ID, however, also allows that messages between vehicles be sent to the correct recipient.
As mentioned above, wireless communication between vehicles may be initiated after a vehicle has been selected in the one or more images captured by the external camera unit 106. Wireless communication may include sending a message, for example via vehicle-to-everything communication, the message including the sender's plate number, addressee's (i.e. target vehicle's) plate number and an individual message or a code of a pre-defined message (for example, code 001 may mean “Turn on the lights”) to other vehicles, road-side units, or emergency services. For example, all vehicles within a defined radius around the sending vehicle (e.g. 200 m), may receive the message, and may decide whether they are the correct recipients of the message. If the addressee's plate number included in the message corresponds to the plate number of a vehicle which has received the message, a receiver unit mounted on the receiving vehicle may generate an acknowledgement message and transmit it to the sending vehicle via radio channel. An acknowledgement message may have any suitable form and content. For example, an acknowledgement message may have the same structure as the original message: Sender plate number, followed by addressee plate number (e.g., may be copied from the sender field of the received message), followed by an individual message or a generally known message code (for example, code 801 may mean “message received”). In addition to sending an acknowledgement message, the receiver unit of the receiving vehicle may decode the initial message and report it to the driver of the receiving vehicle, e.g., using one or more information tools (head-up display, head-unit, dashboard, audio-system, etc.). The message interpretation may be displayed or pronounced in a language selected by the user as communication language.
After having received a message from a vehicle, the driver of the receiving vehicle can acknowledge the message to stop receiving the same messages from drivers of other vehicles. For example, having received a message that the vehicle is noisy and there might be a problem, the driver of the concerned vehicle can acknowledge that they have understood the situation, thus, ignoring the same messages from other drivers during the same driving session.
Message codes may be unified between car manufacturers, in order to allow vehicles of different makes to easily communicate with each other by means of standardized codes. In addition, a list of pre-defined standard messages may be updatable. It is also possible that some messages are country or area dependent. A message codes database in a vehicle may be updated regularly or when crossing the border from a local transportation agency server/cloud, for example. According to one example, country dependent messages may have dedicated codes (e.g. the range between 200 and 599).
Pre-defined messages generally may concern any possible situations: dangerous maneuvers, disturbing/impolite behavior, compliments for being polite (e.g. “Thank you!”, “You are welcome!”), which are otherwise often signalized by misusing the lights and alarm lights, non-proper state of the vehicle (e.g., “door open”, “trunk open”), dangerous state of the vehicle (e.g., “flat tire”, “smoke”, “fire”, “strange noise”), intentions (e.g., message “Please, go first!” may be transmitted if only one lane is available and/or several vehicles have the same priority), requests (e.g., message “Please, give me the way!” may be transmitted if another vehicle is using a wrong lane on a motorway, or if a vehicle is blocking the road), etc. Sending pre-defined messages (e.g., specific message codes) allows that a received message is presented to occupants of the receiving vehicle in a desired language. If, for example, an English speaking occupant of a first vehicle sends an individual, verbally pronounced message to a French speaking occupant of another vehicle, they might not be able to understand the message. If pre-defined messages are sent, the messages can be presented to the occupants of the receiving vehicle in a language that has been selected as standard language. That is, the pre-defined message may be presented in French to the occupants of the receiving vehicle. By allowing only pre-defined messages to be sent to other vehicles, it may also be possible to prevent impolite or rude messages to be sent to other vehicles.
If vehicles are required to exchange more than one message for any reason, a target generally does not have to be searched and selected again. The last target vehicle can be memorized, and a communication can be continued with the same target vehicle, unless a new target vehicle is selected. For example, the driver of a vehicle that was given the way from another vehicle may send a “Thank you!” message without selecting the other vehicle again. This is valid for any vehicle participating in a communication. For example, when the other vehicle receives the “Thank you!” message, it is not required to identify that vehicle again. It may simply reply, e.g., with another message “You are welcome!” and the vehicle previously participating in the communication may be chosen by default. It is also possible to display (e.g., using a head-up display, dashboard, or display of the head-unit) a list of vehicles in the near-zone (within a certain radius) which have recently communicated with the occupant in order to provide the occupant with an opportunity to choose one vehicle out of the list of vehicles for transmitting a message to.
According to another example, some pre-defined messages may be designated as broadcast messages that are to be received to all vehicles within a defined radius. For example, a message “Danger in front”, may be sent to a plurality of vehicles, e.g., if a radar control has been seen or identified, or an accident happened blocking the road, etc. Such type of message is often illegally signalized by drivers by blinking the front lights, i.e. by misusing the lights.
In some cases, selecting vehicles or persons as targets may be too slow when the first user input is a spoken input (e.g., “Please select that blue vehicle and send a Give me the way, please! message to him”). In some cases, providing a spoken first user input may also be inconvenient for any reasons. Therefore, the driver assistance system 100 optionally may comprise or may be coupled to a dedicated button or other control located in some convenient area (e.g. on the steering wheel or dashboard). By simply pressing the dedicated button, automatic search of all ground vehicles located in direct visibility in front of, beside and behind the vehicle may be activated. All identified vehicles may be highlighted, similar to what has been described above. Corresponding images may be presented on a windscreen of the vehicle and/or on any other suitable displays such as, e.g., on the side windows, exterior mirrors, on the rear view mirror, or on the display of a head-unit. According to one example, if a driver wants to say “Give me the way, please!” to a vehicle driving in front, they may be simply required to press a dedicated “Search vehicle targets button”, e.g., on the steering wheel, followed by pointing towards the desired target, and subsequently confirming the selection by verbally pronouncing the desired message. According to another example, if a message is to be sent to a vehicle towards the right, it may be enough to press a “Search vehicle targets button” on the steering wheel followed by rotating the head to the right side and verbally pronouncing the desired message.
It may even be possible that some defined messages that are used regularly are associated with a dedicated “hot key” or with some defined tactile combination on the steering wheel (e.g. double finger hit), or on the touchpad (e.g. drawing a cross or hitting the touchpad with two fingers simultaneously at horizontal disposition three consequent times, etc.). Such “hot keys” or “hot combinations” may be individually programmable by a user. Further, it may be possible that the driver assistance system automatically identifies one or more pre-defined messages that may be suitable in a certain situation. For example, the driver assistance system 100, based on the image(s) captured by the external camera unit 106 may assess a current situation, e.g., by means of any suitable image processing techniques and artificial intelligence algorithms, and may present one or more messages that are considered suitable in the situation to the occupant of the vehicle. The occupant may then select one of the one or more messages presented to them to be transmitted to one or more other vehicles. This similarly applies for a vehicle receiving such a message. A driver assistance system of the receiving vehicle may similarly assess the received message and provide one or more possible response messages to an occupant of the receiving vehicle to be transmitted in response to the original message. According to one example, an occupant of a vehicle receiving a message “Give me the way, please”, may be offered one or more pre-defined messages to respond such as, e.g., “I am sorry!”, or “I apologize, but I cannot do it now”, depending on whether the current traffic situation allows, or does not allow them to move to the right lane.
A particular type of pre-defined messages may also be, for example, “Like” and “Dislike”, which can be received, if other drivers like your driving style or a certain behavior. An occupant may store any “likes” and “dislikes” they received during a driving session in a database (e.g., at personal cloud location, in social networks for sharing with others, etc.). A collection of likes/dislikes can be permanently stored and may be sorted, e.g., by weeks/months/years. Such collections could be used, e.g., for self-analysis and consideration of self-improvement.
In yet further examples, based on the content of a user input, the processing unit may cause one or more operating states of the vehicle to be adjusted. For example, when an occupant states “Do you see the vehicle speeding behind us?”, and the vehicle is identified as discussed above, in addition to sending one or more messages to the speeding vehicle, other vehicles in the region, and/or emergency services, the processing unit may adjust ADAS settings. For example, the processing unit may engage lane assist and/or may reduce a chosen speed of adaptive cruise control. As another example, the processing unit may cause autonomous driving to change lanes away from the identified speeding vehicle or autonomous driving may be disengaged with an alert to the driver stating as such.
Now referring to
Additionally or alternatively, the user interface 104 may comprise a voice processing unit 1044. By means of a voice processing unit 1044, spoken commands of an occupant may be recorded. Additionally or alternatively, the user interface 104 may comprise an electroencephalography analyzing unit 1046, in order to be able to detect brain activities of an occupant (e.g., occupant thinking of a specific object). Alternatively or additionally, the user interface 104 may comprise any other kind of interface unit 1048 such as, e.g., one or more buttons and/or knobs.
The driver assistance system 100, e.g., the processing unit 102 of the driver assistance system 100, may be coupled to a navigation system 1142, an inter-car communication unit 1144, and/or any other external subsystems 1146, 1148. The driver assistance system 100, e.g., the processing unit 102, may additionally or alternatively be coupled to or may comprise a wireless connection unit 110. By means of a wireless connection unit 110, the driver assistance system 100 may communicate, e.g., with a voice assistant 200 (e.g., an in-car voice assistant or an external voice assistant implemented in a cloud) and/or an external database or cloud 202 (e.g., sightseeing data base and recognition system). The driver assistance system 100, e.g., the processing unit 102, may be further coupled to an audio unit 112 comprising one or more loudspeakers. The display unit 108 may comprise or may be coupled to at least one of a head-up display 1082, and a display 1084 (e.g., a touch display) arranged at any other position in the vehicle. The external camera unit 106 may comprise or may be coupled to an image analysis unit 1062, for example.
The external camera unit 106 may comprise one or more cameras mounted at suitable positions in or on the vehicle. According to one example, a single 360° camera may be arranged on the roof of the vehicle. According to other examples, a plurality of cameras may be mounted at different positions in or on the vehicle. One or more cameras may be arranged inside the vehicle and close to the windscreen, for example, in order to be able to monitor a surrounding environment in front of the vehicle. Additionally or alternatively, one or more cameras may be arranged in or on the front lights or in or on the front bumper of the vehicle, for example. In order to be able to monitor a surrounding environment towards the sides of the vehicle, one or more cameras may be arranged in or on the doors of the vehicle, in the A or B pillars, or in or on the roof of the vehicle facing sideways, for example. In order to be able to monitor a surrounding environment towards the rear of the vehicle, one or more cameras may be arranged behind or anywhere close to the rear window, in or on the trunk of the vehicle, or in or on the rear bumper, for example. Any other locations in or on the vehicle are generally possible.
The external camera unit 106 may be coupled to an external image analysis unit, for example. The external camera unit 106 may transmit any captured images to the external image analysis unit and the external image analysis unit may analyze the one or more images and detect potentially interesting objects (buildings, places of interest, geographical objects, vehicles, persons, etc.). The results of such analysis may be transmitted to the processing unit 102, for example, to be presented to the occupant of the vehicle. An analysis performed by an external image analysis unit may optionally comprise coordinates of identified objects in respect to some given point, e.g. in respect to the occupant's eyes, or to a head-up display. Coordinates can be angular positions in horizontal and vertical planes, or positions, or Cartesian coordinates of one of a plurality of points of the concerned object. In addition, an external image analysis unit may deliver horizontal and vertical dimensions of the object.
Parameters (e.g., coordinates, sizes, . . . ) of one or more identified objects may be transmitted from an external image analysis unit to the processing unit 102, which may store the parameters and/or may try to identify a geolocation of the object, by considering a current geolocation of the vehicle (e.g., obtained from a navigation system), a direction of travel of the vehicle, and a direction of the object with respect to vehicle, for example. Optionally, if the driver assistance system 100 is connected to a global network (e.g., the Internet) providing maps with street view possibilities, the processing unit 102 may identify the object by means of comparison of the street view images with the image(s) captured by the external camera unit 106. The processing unit 102 may further acquire more information about an object, including, e.g., a name of the object, its exact geolocation, parking possibilities, an official website, phone number(s), and/or any other information, which can be obtained from an open source. Artificial Intelligence algorithms located in the same or a different cloud, may perform such a search on request, using cloud resources and then deliver information to the processing unit 102.
When inter-vehicle communication is required, an external image analysis unit may continuously analyze the image(s) captured by the external camera unit 106, and identify other vehicles in the surrounding environment of the vehicle as potentially interesting targets. When one or more potentially interesting targets have been found, such targets may be continuously tracked. During the tracking process different parameters of the targets may be determined: speed (relative or absolute), a lane it is driving on, its location with respect to other vehicles on the road, etc. In other words, any parameters may be determined which, may help Artificial Intelligence (AI)-based algorithms to identify some vehicles as potentially interesting targets for the driver in order to be able to establish a communication with them. Such an AI-based algorithm can be integrated into the processing unit 102. After one or more potentially interesting targets have been identified, the processing unit 102 may proceed to highlight (e.g., frame) them while presenting them by means of the display unit 108.
When one or more potentially interesting targets have been identified, the occupant of the vehicle may be able to select one or more of them for performing one or more specific actions. One or more objects may be selected in any suitable way, as has been described above. For example, one or more objects may be selected by means of on eye and/or finger tracking cameras connected to an eye and finger tracking unit (EFTU), which may use suitable algorithms in order to detect motion and direction of view of the eye, or a finger pointing direction. Alternatively or additionally, one or more objects may be selected using microphones connected to a voice processing unit (VPU), which may extract voice signals from a recorded signal, the recorded signal further including surrounding noises, and optionally analyze the speech. It is also possible that processed speech is not analyzed by a voice processing unit of the driver assistance system 100, but is “outsourced” instead to a dedicated cloud. For example, local processing may be useful for systems, where there is no connection to an in-car voice assistant, for example, where only potentially interesting targets are automatically identified and where the occupant only needs to select one of them. If connection to in-car voice assistant cloud-based algorithms is present, a voice processing unit may only purify the speech signal, and then forward it to the processing unit 102, which again may further forward it to an in-car voice assistant cloud via Wireless Connection Unit (WCU).
A potentially interesting target may additionally or alternatively also be selected by the occupant using Electroencephalography (EEG). This may require the occupant to put on a special cap that is supplied with special EEG-sensors. Such sensors may be wirelessly connected to the driver assistance system 100, for example. In this way, user comfort may be increased as there are no disturbing wires inside the vehicle. Any kind of wired connection, however, is generally also possible. According to some embodiments, EEG-sensors may be used which do not require the occupant wearing a cup. EEG-sensors may be used that are configured to remotely read brain signals without being in direct contact with a user's head. Irrespective of how brain signals are transmitted from one or more EEG-sensors to the driver assistance system 100, the driver assistance system 100 may analyze and interpret the signals, e.g., by means of an EEG-analyzer. In this way, EEG-signals may eventually be transformed into commands, which may be routed to the processing unit 102.
Potentially interesting targets may be selected by an occupant in any other suitable way, e.g. via buttons or via a touchpad and/or using pre-defined tactile combinations. Different ways of selecting a potentially interesting target may be provided in a vehicle. An occupant may choose individually, which method they prefer, e.g., whether they want to point at a potentially interesting target, gaze at a potentially interesting target, formulate a verbal command to select a potentially interesting target, press a button, operate a touch display, perform a tactile combination of a steering wheel of the vehicle, using EEG-sensors, etc. One or more methods of selecting a potentially interesting target may be combined with each other (e.g., finger pointing and gazing). This makes the driver assistance system 100 very user friendly, as it allows to individually choose one or more methods, and the accuracy of the system may be increased by combining two or more different selection methods.
For a user-initiated potentially interesting target search and selection, an in-car voice assistance may be used. As mentioned above, an occupant's command (first and/or second user input) may be received via one or more internal microphones, and may optionally pass through a preliminary speech signal purification algorithm, which removes interior car noises from the received microphone signal. The command may be sent from the processing unit 102 to a voice assistant 200, e.g., via a wireless connection unit 110. In the voice assistant 200, speech recognition algorithms may be applied, for example, and potentially interesting targets may be searched.
The voice-assistant 200 may transmit a single command, or a series of commands to the processing unit 102, in order to prompt the processing unit 102 to have at least one image captured by means of the external camera unit 106. According to one example, a driver may request information about an object located ahead of the vehicle. After having received the command from the voice-assistant 200, the processing unit 102 may request the external camera unit 106 to capture one or more images of a surrounding environment of the vehicle, e.g., of a surrounding environment in front of the vehicle. In the meantime, the processing unit 102 may prepare for further potentially interesting target tracking (object identification) and highlighting. The external camera unit 106 may capture one or more images by means of one or more cameras. The one or more images may optionally be stored for further use by a potentially interesting target tracking algorithm, and may be sent to the processing unit 102. The processing unit 102 may forward the one or more captured images to the voice assistant 200, via a wireless connection unit 110, for example. After having received the one or more images, the voice assistant 200 may identify one or more potentially interesting targets (objects) in the one or more images, e.g., based on AI (artificial intelligence) image analysis algorithms, which can be physically located in the same or in a different cloud. The voice assistant 200 may further request that the correct object be found on the image (the object matching the occupant's initial request). If one or more objects matching the occupant's initial request are found in the one or more images, the AI-based algorithm may provide the coordinates of the one or more objects on the image as well as other information such as, e.g., its size, e.g. in pixels.
After having identified one or more objects matching the requested objects, the voice assistant 200 may send one or more commands to the processing unit 102 to highlight the one or more objects in the one or more images when presented by means of a display unit 108 (e.g., on a head-up display). The voice assistant 200 may additionally transmit one or more parameters to the processing unit 102 relevant for highlighting the one or more objects (e.g., position of the object(s) on the image(s), and size of a frame that is to be used to frame the object(s)). Optionally, the voice assistant 200 may transmit a voice message to the processing unit 102. The processing unit 102 may cause a voice message to be output, e.g., by means of one or more loudspeakers in the vehicle. The voice message may request the occupant to confirm that the correct object has been identified, or to select one object, if more than one object has been identified. The voice message may be routed to an audio unit 112 of the vehicle, in order to be played back by means of one or more loudspeakers of the audio unit 112. Outputting a voice message, however, is only one example. It is also possible that one or more objects are highlighted in the one or more images and the occupant has to confirm/select an object without any further request to do so. According to a further embodiment, it is also possible that the occupant is requested to confirm/select an object by means of a message presented on a display of the vehicle.
Signal and command transmission/routing as well as image processing generally require a certain amount of time. When the vehicle moves on along its route, the coordinates of the potentially interesting target(s) on the one or more images as well as their size may change. The driver assistance system, therefore, may be configured to correct (update) the image(s) presented to the occupant accordingly. In particular, the position and/or size of any markings that are used to highlight the one or more objects may change as the vehicle moves and before the occupant confirmed/selected an object. Therefore, after the processing unit 102 has received a command from the voice assistant 200 to highlight one or more objects in the one or more images using specific coordinates and sizes, it may forward the commands to the external camera unit 106. The external camera unit 106 may adjust parameters of the potentially interesting target(s) accordingly, followed by starting an image tracking algorithm. The external camera unit 106 may then send corrected parameters of the potentially interesting target(s) to the display unit 108 (e.g., via the processing unit 102). If the one or more images are presented to an occupant by means of a head-up display, the display unit 108 may transform the parameters of the potentially interesting target(s) on the image(s) in consideration of coordinates that are used by the head-up display concerning the occupant's head position. A potentially interesting target tracking algorithm may be used such that the location and size of the potentially interesting target(s) are always known to a head-up display controller for the visible highlighting of the object(s) on the windscreen. The external camera unit 106 and the display unit 108 may be in direct communication with each other or may communicate with each other via the processing unit 102. Direct communication between the units, however, is generally more efficient with respect to latency minimization. It is generally possible that an audio signal (voice message) received by the processing unit 102 is intentionally delayed before outputting the message, in order to synchronize the audio signal with a point in time at which presentation of the one or more images by means of the display unit 108 is initiated.
Once one or more objects have been highlighted in the presented image(s), the occupant may be requested to confirm the highlighted object, or to select one of a plurality of highlighted objects. When the occupant confirms/selects an object, the processing unit 102 may initiate an appropriate action. For example, a request may be sent to the voice assistant, 200 for object identification. When the object has been identified, a search for information about the object may be initiated, e.g., using a suitable search engine. Any known search engine can be used for this purpose. Information about the object may be presented to the occupant in any suitable way. For example, (basic) information about the object may be presented by means of an audio signal. Presenting a written description of the object to a driver while the vehicle is moving may be hazardous. However, information about an object may be presented in written form to another occupant of the vehicle who is not the driver, for example. The occupant could verbally report or summarize the information to the driver. Optionally, if they are not satisfied with the information presented by the driver assistance system 100, the occupant may request further (e.g., more detailed) information about the respective object. Alternatively or additionally, an occupant may request the driver assistance system 100 to use the location of the respective object as a destination for a navigation system of the vehicle and start navigating to the location of the object. Any other action is generally possible.
By means of the examples presented above, it is demonstrated that there are generally many different possibilities of implementing the driver assistance system 100. Commands of an occupant may be received in many different suitable ways. Object(s) may be highlighted in images in many different suitable ways. An occupant may then confirm or select an object highlighted in the image(s) in any suitable way. Even further, signals and commands may be routed between the different units of the driver assistance system 100 in any suitable way. Even further, the action that is to be performed in response to an object being confirmed/selected by an occupant can be suitably chosen for any possible situation.
Identifying vehicles as potentially interesting targets is a special case when identifying potentially interesting targets (objects) in one or more images captured by the external camera unit 106. Vehicles are almost permanently present on roads, and there are only few situations when there are no other vehicles close by. Road situations are also generally changing rapidly. Therefore, establishing an inter-vehicle communication with another vehicle may be implemented in a very fast and ergonomic way. According to some examples, other vehicles may be identified in the one or more images and displayed accordingly by means of the display unit 108 automatically, without even requiring any user action (user input). Automatic identification and displaying may be individually activated by the occupant, for example. In some traffic situations, an AI-based algorithm may prepare and suggest sending pre-defined messages corresponding to a current traffic situation to specific other vehicles. For example, if on a highway another vehicle occupies the left lane, while the right lane is free, this situation may be captured by the driver assistance system 100 automatically, e.g., by using a suitable vehicle search algorithm when analyzing the one or more images. A suitable action may then be suggested to the driver, e.g., it may be suggested sending a pre-defined message: “Please, give me the way!” to the respective vehicle. The identified vehicle may be automatically highlighted in the image(s) presented to the occupant, and the suggested message may be displayed on a display.
Generally, automated vehicle identification may be implemented using the same technical implementations as described above. However, the algorithms used to identify vehicles instead of objects may differ somewhat. For automated vehicle identification, the external camera unit 106 may continuously capture images (e.g., a video stream may be captured), and perform vehicle search, and (optionally) number plate recognition. In addition, the driver assistance system 100 may automatically identify situations in which inter-vehicle communication may be suitable, for example, potentially incorrect or dangerous situations detected on the images of the surrounding environment (e.g., other vehicles occupying the wrong lane, vehicles without lights on in the dark, road accidents, speed controls, dangerous maneuvers of other vehicles, etc.). The driver assistance system 100 may have access to a list of situations in which inter-vehicle communication may be required (the list may be regularly updated), and algorithms for vehicle identification may be updated regularly. According to one example, exterior cameras may be used for vehicle identification. However, other vehicles in the surrounding environment may generally be detected by means of any other suitable sensors and data sources, e.g., in combination with exterior cameras (e.g., LIDAR sensors), geolocation, time of the day, season, etc.
Now referring to
When the voice processing unit 1044 receives a voice command from the occupant to send a message to a specific vehicle within a near-zone using inter-vehicle communication, a verification may be performed (e.g., by means of the processing unit 102) whether this is the first voice command for inter-vehicle communication during a present driving session. If yes, the driver assistance system 100 (e.g., the voice processing unit 1044 or the processing unit 102) may request the inter-vehicle communication unit 300 (e.g., a message code data base 306) to provide message code data via the bus 302. Message code data may include message code only. Including message transcripts into text into message code data is generally not required. In order to decrease the data volume to be sent, only text in the default language may be transmitted. For example, if the message code data base 306 contains a message code in a first column, its transcript into English in a second column, its transcript into German in a third column, . . . its transcript into simplified Chinese in a xyz column, then only the contents of some columns, e.g., columns 1 and 3, may be sent, if the default language of the vehicle is “German”.
An ordinal number of the vehicle's default language may be stored in a settings unit 310, for example. After having received the message code data, the driver assistance system 100 (e.g., the voice processing unit 1044 or the processing unit 102) may identify which pre-defined message corresponds to the voice request of the occupant after passed through a speech to text module 304 (which however is not necessarily a part of the inter-vehicle communication unit 300). Such a correspondence may be established using AI-based algorithms, for example. As a result, the voice assistant 200 may receive the message code that is to be sent, or, if a corresponding message is not found, the occupant may be informed and asked to re-formulate the message. In case of ambiguities, an AI algorithm may even suggest two or more messages to the occupant to select one message to be sent. If this is the case, the occupant can select a specific message from a proposed list of messages by means of any suitable technique, e.g., using a scrolling wheel, or a mini-touchpad on the steering wheel, or verbally pronouncing the number of the desired message in the list. If, for example, an occupant of a vehicle wants to inform occupants of oncoming vehicles that there is a wild animal on the road, there may not be a suitable pre-defined message covering this situation. The AI algorithm may present alternatives such as, e.g., “General hazard in front”, or “Serious traffic accident in front”, of which the occupant may select one message. Even if none of the presented messages exactly capture the situation, they will still be able to warn drivers of oncoming vehicles and raise their awareness.
Generally, there may be different types of messages. One type of messages may include messages with a fixed addressee, another type of message may include broadcast messages. A broadcast message is a message, which is addressed to more than one vehicle, e.g., to all vehicles in the near-zone (within a defined radius), or to all vehicles fulfilling defined properties, for example, all vehicles going in the opposite direction. In order to prepare a message to be sent, the inter-vehicle communication unit 300 may require the plate number of the selected target vehicle. If it is the first communication with that specific vehicle, its plate number converted to text may be provided by a plate recognition module, which may be a component of the external camera unit 106, for example. If this is not the first message to or from a certain vehicle, its plate number may already be known and may be stored in a communication list. By default, (if a target vehicle is not explicitly selected), a message may be sent to the last vehicle that has been communicated with. For example, if a message (e.g., “Thank you!”) has been received from a vehicle with the plate number “M K 324” and a response to this message is to be sent, no further search for the respective plate number of this vehicle is required. A reply message may simply be sent (e.g., “You are welcome!”), as the recipient is stored in the list of the last communicated vehicles still located in the near-zone. Moreover, its plate number will be in the first position of an inter-vehicle communication list. Therefore, the plate number can be omitted when a response is sent. It will be automatically sent to the last communicated car—“M K 324”.
When a composing and sending unit 312 receives a message code and a target vehicle plate number, it may start composing the message, which may comprise, e.g., message code (e.g., 2-4 bytes), plate number of the target vehicle (e.g., 10-12 bytes), and plate number of the sender (may be stored in the settings unit 310). After the message has been composed, it can be send to a transmission unit 314, which may be configured to prepare the message for radio-transmission to the target vehicle via an antenna 316. The antenna 316 may transform the electrical signal with the encoded message into radio-waves. In
When the antenna 316 receives incoming signals 42, the received radio signals are transmitted it into electrical signals, which are provided to the transmission unit 314. The transmission unit 314 may be configured to transform an analog electrical signal into a digital format which may then be fed to a decoding unit 308. The decoding unit 308 may be configured to decode and analyze a received message. The decoding unit 308, for example, may first determine whether the received message is addressed to the receiving vehicle, or whether it is addressed to another vehicle. If it is addressed to another vehicle, the message may be dismissed. The received message may also be determined to be a broadcast message. In this case, the receiving vehicle may be determined to be one of several intended recipients. This can be determined, e.g., by analyzing an addressee field of the message, which may be compared with a plate number of the receiving vehicle. If the addressee plate number indicated in the received message is identical with the plate number of the receiving vehicle, or if no addressee plate number is indicated in the received message (i.e. broadcast message), the message may be further processed.
As there are various different formats and symbol types of plate numbers worldwide, comparing the plate number included in the message and the plate number of the receiving vehicle may be challenging. However, certain common rules and standards may generally be defined for plate number recognition format which may be applicable for all car manufacturers. Further processing of a received message may include sending an automatic acknowledgement by generating and feeding an acknowledgment command from the decoding unit 308 to the composing and sending unit 312, extracting message code and translating it into text by means of the message code data base 306, extracting a sender's plate number as text and routing it as text to the processing unit 102 via communication bus 302. Additionally, or alternatively, it may be possible to convert the text of the message and the sender's plate number into a speech signal by means of the speech to text module 304. The speech signal may be routed via the processing unit 102 to the audio unit 112 and then played back via the one or more loudspeakers.
If receipt of a message sent to other vehicles by means of the inter-vehicle communication unit 300 is not acknowledged by the receiver within certain pre-defined time (which can be hard coded or be stored in the settings unit 310), the same message may be sent again. After several unsuccessful attempts, the occupant may be informed that the message could not successfully be delivered to the target vehicle. This may be the case, for example, if a target vehicle does not support inter-vehicle communication, or if inter-vehicle communication in the receiving vehicle has been deactivated for any reason. The maximum number of attempts to send the same message may be defined in the settings unit 310, for example.
The inter-vehicle communication unit 300 as illustrated in
Now referring to
At 702, method 700 includes receiving a first user input from an occupant of a vehicle. As is described herein, in one example, the first user input may be a voice prompt received by a voice processing unit. The voice prompt may be a request, such as a request for additional information of an object or geographic feature, a question regarding another vehicle on the road, or the like as is herein described.
In some examples, one or more secondary inputs may be obtained regarding the first user input, as noted at 704. For example, the vehicle may be equipped with an internal camera unit equipped with eye-gaze tracking capabilities. In such an example, the secondary inputs may be determinations of the eye gaze of the occupant that may provide additional information about the first user input, such as which direction the occupant is looking toward, thus indicating the direction in which the object may be. As another example, the first user input may include a prompt that includes additional details. For example, a prompt may be “what is that building ahead on my left?”. While the main portion of the prompt is “what is that building”, the additional information of “ahead on my left” may be considered secondary inputs to the first user input. As will be described below, the secondary input(s) may allow for filtering of images to identify relevant images.
At 706, method 700 includes, in response to receiving the first user input, obtaining one or more images. In some examples, obtaining the one or more images may include instructing one or more cameras of an external camera unit (e.g., external camera unit 108) to capture images. In other examples, obtaining the one or more images may include obtaining previously captured images from memory. As is herein described, acquired images may be stored in memory temporarily for a buffer period of time. In some examples, the buffer period may be adjustable, such as based on the speed of the vehicle. Images may be obtained from memory in examples when the delay of the occupant speaking the voice prompt would render newly captured images unusable (e.g., if the object would be gone from view).
In examples in which secondary input(s) regarding the first user input are also received or determined, relevant images from available images may be identified, as noted at 708. For example, when the first user input indicates a region in which the subject of the request is located, or when eye-gaze tracking indicates the region, a subset of images may be obtained from memory that were captured by cameras facing in the indicated direction. Similarly, when cameras are instructed to capture new images, only a subset of the cameras, for example those facing in the indicated direction, may be activated to capture new images or only images from those cameras may be taken for processing. In this way, the number of images that are to be processed for object identification may be reduced, thus increasing overall processing efficiency.
At 710, method 700 includes identifying one or more objects in the one or more images. As is described above, identifying one or more objects may be accomplished by application of a segmentation process to the one or more images, as noted at 712. Such segmentation processes may include any feasible segmentation algorithm, a machine-learning model trained to identify objects, or the like. Further, following object identification, the one or more images may be annotated based on the identified images, as noted at 714. For example, each object within an image may be highlighted with a colored frame to allow for easy differentiation by the occupant. Other annotations, such as numbers, symbols, and the like, may also be included.
At 716, method 700 includes displaying one or more images on the display device. In some examples, a single representative image, such as an image that includes all of the identified objects, may be displayed. In other examples, more than one of the one or more images may be displayed, for example in a grid design. The display device may be a display device of an infotainment head unit, a dashboard, or the like. In other examples, the one or more images may be displayed on a personal device, such as a smart phone, that is communicatively coupled to the processing unit (e.g., via radio communication like Bluetooth). When the images are annotated, annotated versions of the one or more images may be displayed.
At 718, method 700 includes receiving a second user input from the occupant selecting at least one of the one or more objects in the displayed image(s). In some examples, the second user input may comprise a manual selection of an object (e.g., via the touchscreen of the infotainment unit), visual selection (e.g., via eye-gaze tracking from internal cameras), via voice selection (e.g., a voice prompt indicating one of the objects), or the like, as described above. In examples in which only one object was identified, the second user input may be a confirmation that the object identified is the desired object.
At 720, method 700 includes causing an action to be performed in response to the second user input. The action that is taken may be dependent upon the type of request. When the first user input regards a concerning behavior of another vehicle in the vicinity, as a non-limiting example, the action may include adjusting one or more vehicle operation states, as noted at 722. Adjusting one or more vehicle operation states may include altering ADAS settings, in one example, such as enabling lane assist, disabling autonomous driving, reducing adaptive cruise control speed, or changing lanes when in autonomous driving mode. When the first user input regards another vehicle's state (e.g., headlights/brake lights status, etc.), a person, or an abnormal road feature (e.g., a malfunctioning traffic light, a downed power line, etc.), the action may include transmitting one or more messages, as noted at 724. As non-limiting examples, a message may be transmitted to a specific identified vehicle telling them they have a headlight out, a message may be broadcast to multiple vehicles in the vicinity informing them of a speeding vehicle, a message may be transmitted to emergency services informing them of a crime or of an abnormal road feature, or the like. Messages may be transmitted via any reasonable technology, such as vehicle-to-everything methods. When the first user input regards a geographic feature, tourist attraction, a building, or the like, the action may include obtaining and displaying information regarding the selected at least one of the one or more objects, as noted at 726. For example, as is described with respect to
The technical effect of the systems and methods described herein is that via reception of voice prompts, a vehicle may obtain images of a surrounding environment, identify objects within the images, and then take actions based on the voice prompt and the object(s) identified. Thus, occupants of a vehicle may be provided additional information about their surroundings, may communicate easily with other vehicles in the vicinity, and/or report abnormalities they have witnessed all without being distracted from driving. Further, based on particular prompts and identified objects, operation of a vehicle may be adjusted. Further still, by retrieving images stored in memory for object identification, wherein the memory is configured to store images for an adjustable period of time (e.g., adjustable based on speed of the vehicle), object identification may be performed even when the object is no longer in active view of the occupant. For example, the occupant may ask “what was that store we just passed” and images stored temporarily in memory may be analyzed. Additionally, image analysis may be performed specifically for relevant images based on secondary inputs such as additional information in a user input or from in vehicle sensors, which may reduce the processing demands on the system, thus increasing efficiency and speed of output.
The general concept has been described with respect to a passenger car above. Generally, however, the vehicle may be any kind of vehicle such as, e.g., a passenger car, a van, a camper van, a caravan, a truck, a bus, a tractor, a motorbike, or a train, etc. It may be understood, that the illustrated systems are merely examples. While various embodiments of the invention have been described, it will be apparent to those of ordinary skill in the art that many more embodiments and implementations are possible within the scope of the invention. In particular, the skilled person will recognize the interchangeability of various features from different embodiments. Although these techniques and systems have been disclosed in the context of certain embodiments and examples, it will be understood that these techniques and systems may be extended beyond the specifically disclosed embodiments to other embodiments and/or uses and obvious modifications thereof. Accordingly, the invention is not to be restricted except in light of the attached claims and their equivalents.
The disclosure also provides support for a driver assistance system of a vehicle, comprising: a processing unit, and non-transitory memory storing instructions that when executed cause the processing unit to: receive a first user input from an occupant of the vehicle, in response to the first user input, obtain one or more images captured by one or more cameras of an external camera unit, identify, within the one or more images, one or more objects corresponding to the first user input, and take one or more actions based on the one or more objects and the first user input. In a first example of the system, the memory stores further instructions that when executed cause the processing unit to: display the one or more images on a display device, and receive second user input selecting at least one object of the one or more objects, wherein the one or more actions are taken based on the selected at least one object of the one or more objects. In a second example of the system, optionally including the first example, the selected at least one object is a place or object of interest, and wherein the action comprises at least one of: providing information about the selected at least one object to the occupant of the vehicle, and adding a location of the selected at least one object as a destination in a navigation system of the vehicle. In a third example of the system, optionally including one or both of the first and second examples, the selected at least one object is a vehicle or person, and the action comprises transmitting a message to the selected at least one object. In a fourth example of the system, optionally including one or more or each of the first through third examples, the selected at least one object is a vehicle or person, and the action comprises at least one of: storing one or more images captured by the external camera unit in the non-transitory memory, adjusting one or more vehicle operation states, and transmitting a message to an appropriate authority. In a fifth example of the system, optionally including one or more or each of the first through fourth examples, the one or more image are obtained from temporary memory, wherein the temporary memory is configured to store images for a buffer period of time that is adjustable based on a speed of the vehicle. In a sixth example of the system, optionally including one or more or each of the first through fifth examples, the non-transitory memory stores further instructions that when executed cause the processing unit to: identify the one or more images from a larger set of images based on one or more secondary inputs relating to the first user input, wherein the one or more secondary inputs include at least one of: eye-gaze tracking information obtained by interior vehicle cameras, the eye-gaze tracking information indicating a region in which the one or more objects are located, and region information included in the first user input. In a seventh example of the system, optionally including one or more or each of the first through sixth examples, to obtain the one or more images, the processing unit is configured to cause the one or more cameras of the external camera unit to capture new images specific to the first user input.
The disclosure also provides support for a method for a driver assistance system, comprising: receiving a first user input from an occupant of a vehicle, in response to the first user input, obtaining one or more images of a surrounding environment of the vehicle captured by an external camera unit, based on the first user input, identifying one or more objects in the one or more images, annotating the one or more images to highlight the one or more objects, displaying at least one of the one or more annotated images by means of a display unit, receiving a second user input from the occupant of the vehicle, the second user input including a selection of at least one of the one or more highlighted objects, and causing an action to be performed in response to the second user input. In a first example of the method, the selected at least one object is a place or object of interest, and wherein the action comprises at least one of: providing information about the selected at least one object to the occupant of the vehicle, and adding a location of the selected at least one object as a destination in a navigation system of the vehicle. In a second example of the method, optionally including the first example, the selected at least one object is a vehicle or person, and the action comprises transmitting a message to the selected at least one object. In a third example of the method, optionally including one or both of the first and second examples, the selected at least one object is a vehicle or person, and the action comprises at least one of: storing one or more images captured by the external camera unit in non-transitory memory, and transmitting a message to an appropriate authority. In a fourth example of the method, optionally including one or more or each of the first through third examples, annotating the one or more images to highlight the one or more identified objects in the displayed one or more images comprises at least one of framing each object of the one or more identified objects by means of a frame, marking each object of the one or more identified objects by means of a specific color, and marking each object of the one or more identified objects by means of a number, letter or symbol. In a fifth example of the method, optionally including one or more or each of the first through fourth examples, obtaining the one or more images captured by the external camera unit comprises capturing new images with one or more cameras of the external camera unit based on the first user input. In a sixth example of the method, optionally including one or more or each of the first through fifth examples, obtaining the one or more images captured by the external camera unit comprises obtaining, from transitory memory, the one or more images captured by one or more cameras of the external camera unit based on the first user input. In a seventh example of the method, optionally including one or more or each of the first through sixth examples, the transitory memory is configured to store images obtained by the external camera unit for a buffer period of time, wherein the buffer period of time is adjustable based on a speed of the vehicle. In a eighth example of the method, optionally including one or more or each of the first through seventh examples, obtaining the one or more images captured by the external camera unit comprises filtering images obtained by the external camera unit to identify a subset of images relevant to the first user input.
The disclosure also provides support for a system in a vehicle, comprising: a processing unit, and non-transitory memory storing instructions that when executed cause the processing unit to: receive a voice prompt from an occupant of the vehicle via a voice processing unit, in response to the voice prompt, obtain one or more images captured by one or more external vehicle cameras, identify one or more objects relating to the voice prompt within the one or more images, display at least one of the one or more images on a display device, receive user selection of at least one of the one or more objects, and take an action in response to the user selection based on the voice prompt and the selected at least one of the one or more objects. In a first example of the system, the action includes at least one of adjusting one or more vehicle operation states, transmitting one or more messages, and obtaining and displaying information regarding the selected at least one of the one or more objects. In a second example of the system, optionally including the first example, adjusting the one or more vehicle operation states comprises adjusting one or more settings of an advanced driver assistance system (ADAS), transmitting one or more messages comprises transmitting messages to at least one of the selected at least one of the one or more objects, other vehicles in a vicinity of the vehicle, and emergency services, and obtaining the information regarding the selected at least one of the one or more objects comprises accessing one of a database and a cloud.
The description of embodiments has been presented for purposes of illustration and description. Suitable modifications and variations to the embodiments may be performed in light of the above description or may be acquired from practicing the methods. The described arrangements are exemplary in nature, and may include additional elements and/or omit elements. As used in this application, an element recited in the singular and proceeded with the word “a” or “an” should be understood as not excluding plural of said elements, unless such exclusion is stated. Furthermore, references to “one embodiment” or “one example” of the present disclosure are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features. The terms “first,” “second,” and “third,” etc. are used merely as labels, and are not intended to impose numerical requirements or a particular positional order on their objects. The described systems are exemplary in nature, and may include additional elements and/or omit elements. The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various systems and configurations, and other features, functions, and/or properties disclosed. The following claims particularly point out subject matter from the above disclosure that is regarded as novel and non-obvious.
Number | Date | Country | Kind |
---|---|---|---|
23 218 534.8 | Dec 2023 | EP | regional |