Patent Application
20220180666
This application claims priority to German Patent Application No. DE 10 2019 204 054.3, filed on Mar. 25, 2019 with the German Patent and Trademark Office. The contents of the aforesaid patent application are incorporated herein for all purposes.
The present invention relates to a method to provide a speech dialog in sign language in a speech dialog system for a vehicle.
This background section is provided for the purpose of generally describing the context of the disclosure. Work of the presently named inventor(s), to the extent the work is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.
Speech dialog systems in a vehicle enable operation of the vehicle as well as an exchange of information with the vehicle through verbal communication. In times of the automation of driving functions, an additional driving experience may thereby be created that consists in being able to communicate with the vehicle to exchange information and/or for entertainment. Gesture control of instruments in the vehicles is also known. However, carrying out dialogs with the vehicle is still problematic for people with difficulties with the spoken language, or respectively deaf and hard-of-hearing people.
A need exists to overcome the above-described disadvantages at least in part and to provide a solution for a nonacoustic speech dialog system.
The need is addressed by a method, by a system, and by a vehicle according to the independent claims.
Embodiments of the invention are described in the dependent claims, the following description, and the drawings.
The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features will be apparent from the description, drawings, and from the claims.
In the following description of embodiments of the invention, specific details are described in order to provide a thorough understanding of the invention. However, it will be apparent to one of ordinary skill in the art that the invention may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the instant description.
Herein, features and details that are described in connection with the discussed method also apply to the discussed system as well as to the discussed vehicle, and in each case vice versa, and therefore reference is or may be made interchangeably in the disclosure thereof.
In a first exemplary aspect of the teachings herein, a method for providing a speech dialog in sign language in a speech dialog system is discussed, for example a nonacoustic speech dialog system, for a vehicle. In doing so, it is for example provided that the following is performed, e.g., sequentially in the indicated or any desired sequence, wherein individual steps may if applicable also be repeated:
In this way, a nonacoustic execution of a speech dialog using the speech dialog system may be provided in order to carry out natural language dialogs with the vehicle. The natural language dialog is for example formed by the input information (for example in the form of a “speech”, or question, or answer, or instruction), and the output relating thereto (for example in the form of a “response” or answer to the question, or question, or reply to the instruction). A particular benefit in this case is that the output is provided in sign language to also enable dialog for people for whom the use of a spoken language for dialog is associated with difficulties. It may therefore be possible for the dialog to be completely without spoken language.
It may furthermore be possible for the input information to have information on a sign language of the vehicle occupant. This allows the dialog provided by the speech dialog system to be entirely provided in sign language. In this regard, the input information comprises for example information on a gesture, and/or facial expression, and/or silently spoken words, and/or body posture of the vehicle occupant that may be analyzed through the evaluation by a processing assembly. In order to detect input information with this information, a recorded image of the vehicle occupant may for example be made by an imaging assembly. The evaluation may furthermore comprise an image analysis and/or methods of machine learning in order for example to perform a classification of the input information with respect to predefined gestures and/or signs. The situation in the vehicle's interior may be analyzed in this way by the speech dialog system and utilized in the dialog.
It is furthermore optionally provided (in some embodiments) that the performance of the evaluation comprises the following steps:
This enables a reliable evaluation of the input information to provide the dialog. The generation of the answer may furthermore for example comprise a contentual analysis of the translation information in order for example to determine contentual information as the answer whose content relates to the input information (for example as a question). The output may be in the form of the output of a graphic avatar, and/or an abstract form, and/or by signs (i.e., sign-based), and if applicable also as audio output.
It may be possible for at least one gesture by the vehicle occupant to be detectable in the evaluation of the input information. This gesture by the vehicle occupant may be detected as a gesture in the context of the sign language and correspondingly serve to translate the sign language into the translation information. Furthermore, the gesture may also be detected as an additional gesture that is specific to additional information. An additional gesture for example consists in pointing to an object that correspondingly provides the selected object as the additional information. By using the detected additional gesture, the additional information, in addition to the translation, may be used in the evaluation to generate the answer. For example, the translation information indicates a certain question relating to the object. In other words, the translation information may indicate the function, and the additional information may indicate a parameter for this function. The answer may then be generated as an output of the function.
The additional gesture and the additional information are for example determined in that the imaging assembly, for example a first imaging unit such as a camera, detects the input information from which the additional gesture may be derived (such as pointing to the object). The additional information (i.e., the specific object) may for example be determined by the querying (additional detection) of at least one additional imaging assembly, and/or at least one additional imaging unit of the vehicle by which the environment of the vehicle and therefore the object is detected. To accomplish this, for example directional information is derived using the additional gesture (what is the pointing direction?) and compared with the additional detection of the object. It is also conceivable for the object to be identified by using the directional information in that location information on a current location of the vehicle is evaluated, and for example compared with environment information and the direction information.
According to some embodiments, it may be provided that the answer comprises at least one of the following pieces of information (with the following content):
A system for a vehicle to provide a speech dialog in sign language is the subject of a second exemplary aspect of the teachings herein. The system may for example be designed as a vehicle electronic system and may therefore be suitable to be permanently integrated in the vehicle. In this regard, it is provided that the system has at least one of the following components:
Therefore, the system yields the same benefits as those described in detail with reference to aforementioned method according to the first aspect. In addition, the system may be suitable for performing the method according to the first aspect. It is furthermore conceivable that the system is designed as a speech dialog system of a method according to the first aspect in order to perform the steps of the method.
The system may for example be designed as a technical component in a vehicle that monitors the interior of the vehicle with the assistance of the imaging assembly, for example in the form of a camera system, and is capable of recognizing when a vehicle occupant (passenger) is interacting with the system, for example the speech dialog system (speech assistant) and translates the sign language into text. This allows the speech assistant to behave just as if the user were to speak using speech with the speech assistant. The imaging assembly is for example installed in the front in the instrument panel or in the rearview mirror and therefore offers precise user monitoring.
Some embodiments provide that the imaging assembly is designed as an infrared camera in order to optically detect at least one gesture of the vehicle occupant. The camera may for example be designed for detection in the near and/or far infrared range to detect the input information very reliably and with little interference.
Optionally, it may be provided that the output assembly comprises at least one display, for example an LED display, for the vehicle interior in order to output the visual output as a visualization of an answer to the input information in the sign language, wherein the output assembly is for example designed as an instrument cluster for the vehicle. This enables convenient recognition of the output in the sign language.
A vehicle with a system according to the present aspect is also a subject of the teachings herein. It may be beneficial for the vehicle to be designed as an autonomous vehicle, i.e., for example as a self-driving motor vehicle. Such a vehicle may be driven, controlled and parked without the influence of a human driver. In this case, the use of the speech dialog system is particularly recommendable since none of the vehicle occupants is occupied with controlling the vehicle.
It is also beneficial if the vehicle is designed as a motor vehicle, for example a trackless land motor vehicle, for example a hybrid vehicle that comprises an internal combustion engine and an electric machine for the traction, or as an electric vehicle, for example having a high-voltage on-board power supply and/or an electric motor. For example, the vehicle may be designed as a fuel cell vehicle and/or passenger car. For example, in the case of embodiments of electric vehicles, no internal combustion engine is provided in the vehicle, which is then driven exclusively by means of electrical energy.
According to some embodiments, it may be provided that the output assembly is arranged in the vehicle interior, for example in the region of a center console, and the imaging assembly is arranged in the region of an instrument cluster, and/or the center console, and/or an inside mirror, and/or rearview mirror, and/or outside mirror of the vehicle. For example, the imaging assembly may be arranged in the vehicle such that the detection of the driver of the vehicle is possible. It is advisable for example to arrange the imaging assembly on the rearview mirror and/or outside mirror on the side of the vehicle on which the driver of the vehicle is also sitting. It is furthermore also conceivable to expand detection by the imaging assembly to other occupants of the vehicle. Alternative assemblies in or on the vehicle, and/or also the use of other imaging assemblies, are then correspondingly suitable.
Other advantages, features and details will become apparent from the following description, in which exemplary embodiments will be described in detail with reference to the drawings.
In the FIGS. below, the same reference signs are used for the same technical features, even in different exemplary embodiments.
Specific references to components, process steps, and other elements are not intended to be limiting. It is further noted that the FIGS. are schematic and provided for guidance to the skilled reader and are not necessarily drawn to scale. Rather, the various drawing scales, aspect ratios, and numbers of components shown in the FIGS. may be purposely distorted to make certain features or relationships easier to understand.
A processing assembly 120 is furthermore provided for evaluating the detected input information, wherein the processing assembly 120 is for example designed as a vehicle electronics system. It may also be possible for the evaluation to be partially done by external components such as an external data processing system. To do this, the processing assembly may for example communicate with the external data processing system, for example through a network such as a mobile phone network and/or the Internet. In this way, additional resources may be used for evaluation.
Moreover, an output assembly 130 for visual output in sign language may be provided depending on the evaluation. Normally, in a speech dialog system, the input by the user as well as the answer by the system 100 (and/or vice versa) is through acoustic speech. Herein, at least the answer by the system 100 is in sign language. This accordingly enables communication with the vehicle 1 even when communication via spoken language is impossible.
The imaging assembly 110 may be designed as a camera, for example an infrared camera, in order to at least optically detect a gesture of the vehicle occupant 2 via the input information. The gesture in this case is for example part of the sign language. Moreover, the input information may also at least comprise information on a facial expression, and/or silently spoken words, and/or body posture of the vehicle occupant that may also be part of the sign language. This makes it possible to digitally record the sign language by using this information.
Moreover, the gesture detected via the input information may also be another gesture that is specific to additional information. The additional information may therefore be detected in addition to the information on the sign language. The additional gesture for example consists in pointing toward an object 5 (pointing with a finger of the vehicle occupant 2 is shown, which provides the directional information). The additional information is then the information of the indicated object 5. This additional information may for example also be determined by evaluating the input information, for example if this also comprises information on the object 5, or respectively the outer region of the vehicle 1, or it may also be determined by evaluating the detection of at least one additional imaging assembly (such as the outer region of the vehicle 1). In the evaluation, this makes it possible to evaluate a command or question transmitted using the sign language, and to compare it with the additional information. Information on the object 5 that was indicated may then be determined as the answer.
The output assembly 130 may comprise at least one display 132, for example an LED display, for the vehicle interior 40 in order to output the visual output 131 as a visualization of an answer to the input information in the sign language, wherein the output assembly 130 is exemplary designed as an instrument cluster 10 for the vehicle 1.
Moreover, the output assembly 130 in the vehicle interior 40 may for example be arranged in the region of a center console 30, and the imaging assembly 110 may be arranged in the region of an instrument cluster 10, and/or the center console 30, and/or a rearview mirror 21, and/or an outside mirror 20 of the vehicle 1.
The description of the embodiments given above describes the present invention exclusively within the scope of examples. Of course, individual features of the embodiments may be combined freely with one another, to the extent that this is technically feasible, without departing from the scope of the present invention.
The invention has been described in the preceding using various exemplary embodiments. Other variations to the disclosed embodiments may be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single processor, module or other unit or device may fulfil the functions of several items recited in the claims.
The term “exemplary” used throughout the specification means “serving as an example, instance, or exemplification” and does not mean “preferred” or “having advantages” over other embodiments.
The mere fact that certain measures are recited in mutually different dependent claims or embodiments does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10 2019 204 054.3 | Mar 2019 | DE | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2020/056407 | 3/11/2020 | WO | 00 |
