Patent Application
20120268294
The present invention relates to a human machine interface unit for a communication device in a vehicle and an I/O method using said human machine interface unit.
Modern vehicles are equipped with a complicated information and entertainment system comprising multiple data input and data output communication devices. Output communication devices represent data generated by an application or function of the device or vehicle to a driver or a passenger, and input communication devices receive data from a user. Output communication devices can comprise visual output units, such as signal lights, analogue or digital displays, head-up displays; audio output units, such as audio signal means, buzzers or loudspeakers for voice output; or mechanical output units, such as mechanical vibration units; and other haptic output units. Examples for input communication devices include keyboards, knobs, switches, jog dials, speech recognition input units, touch-sensitive displays or visual gesture recognition units. Besides, I/O communication devices can also comprise a combined input and output unit, such as a touchscreen unit or the like. An internal information and entertainment system of a vehicle can comprise a multimedia system, a music entertainment device, an on-board computer or a communication terminal of a vehicle.
Mobile I/O devices, such as smart phones, mobile computers, multimedia players etc. can interact with or can be integrated into the vehicle's information and entertainment system. Examples for applications or mobile devices may include a navigation system, telecommunication devices, such as a mobile phone, a smart phone, an internet device, a multimedia device, such as a MP3 player or a DVD player, a traffic information device, a toll payment device or a vehicle state information device.
Since vehicles are capable of representing more complex information and adopting more and more applications that originate from mobile and fixed internet devices, such as online search, SMS messaging, weather services, Twitter, etc., more complex user interface functions need to be integrated into the human machine interface (HMI) of each of said communication devices. The HMI defines the way in which information is output from or input into an application, such as an Email, an Internet browser, a Playlist editor, a vehicle state check etc., of such a communication device. Preferably, the HMI of complex mobile I/O devices, such as smart phones, mobile computers, portable navigation systems or mobile multimedia devices, have to be adapted to the constraints of the vehicle's information and entertainment system.
On the other hand, the rules for safe vehicle operation require that driver distraction should be minimized and well controlled. To this end, specific HMI methods that are convenient for a passenger are not necessarily applicable to the driver. Incoming messages, navigation data, traffic news or vehicle state information being relevant for the driver have to be prioritized depending on the driving condition and the information relevance. A passenger can be free to use a communication feature, such as an Email, or can watch a video clip while such information access should be suppressed for a driver during driving. Highly relevant information concerning driving safety, such as traffic news, vehicle failure warning or navigation information, should be clearly and directly represented to the driver and should not disturb the passenger. Such information should be represented adaptively in the form of audible, visual and/or mechanical information using different levels of intensity considering the driving situation and the information relevance.
For certain types of I/O devices and vehicles it is well known to define scenarios in which I/O devices should adapt to different driving situations and to the needs of the machine, the driver and the passenger. Such approaches are usually limited to a single HMI field, such as representing navigation information, offering a phone call or displaying vehicle state information, but fail to provide an open standard for embedding various I/O devices smoothly and comfortably into various types of HMI fields and various types of I/O devices. Such approaches are based on the detailed knowledge of available I/O devices and the functionality thereof. If information is to be output to a driver or to a passenger, a central unit selects a specific I/O device for information output and precisely selects the form of the output depending on the functionality of the I/O device. For example, if an application connected to a vehicle HMI system aims to receive a selection of an item from a list, the HMI system would select a touch-sensitive display to represent a specific table displaying a list and to receive a user selection by finger-tipping an item included in that table.
Adapting the HMI unit of different embedded vehicles and mobile communication devices to the aforementioned various requirements of application, of the driver and the passenger and to various driving conditions usually requires an in-depth understanding of the software and the hardware of the communication device and of the vehicle's infrastructure and usually leads to a complicated and costly adaption of each communication device to a specific vehicle information and entertainment system. Typically, a HMI unit of a mobile or embedded I/O device needs to be adapted to a specific vehicle information architecture which increases the integration costs and efforts. As a consequence, an adapted mobile communication device cannot be easily integrated into different vehicle environments. Safe driving, easy usability and high comfort of the HMI unit of each communication device cannot be guaranteed, especially in complex information and entertainment systems with multiple attached communication devices.
Therefore, it is desirable to simplify the adaption of HMI units of communication devices to a vehicle information and entertainment system architecture and to provide smooth and easy interaction between application, communication device and human user. To enable general application developers to understand a specific vehicle HMI paradigm and usage constraints in order to provide an application to the vehicle manufacturer (or directly to the end-user), a HMI unit and an I/O method are needed to abstract HMI functions and their behavior from the application, and hence to allow direct and consistent control over the HMI behavior to be defined by the vehicle manufacturer independently of the application development process.
The object of the present invention is achieved by a human machine interface unit, a communication device and an I/O method using said human machine interface unit according to the independent claims.
A first aspect of the invention relates to a human machine interface unit (HMI unit) for a communication device in a vehicle. The HMI unit comprises a controller means, a memory means, an API interface means in communication with a communication application means APP, and at least one I/O-interface means I/O-IM in communication with at least one physical I/O unit PHEx for physically communicating with an HMI endpoint user HEx. Said I/O interface means I/O-IM is also in communication with at least one vehicle state unit VS, a driving state unit DS and/or an environmental condition state unit EC. Said API interface means provides a standardized set of abstract input and output methods AMx for inputting data into and outputting data out of the communication application means APP. The controller means selects at least one physical I/O unit PHEx for physically inputting data from or outputting data to said physical I/O unit PHEx depending on a set of criteria stored in said memory means, comprising at least one type of a requested abstract I/O method AMx, an addressed HMI endpoint user HEx, a vehicle state VSx, a driving state DSx and/or an environmental condition ECx. The HMI unit may reside inside a mobile communication device, such as a mobile phone, a mobile computer, a navigation system, a multimedia player etc., or may reside in an internal vehicle communication device, such as an entertainment system. The controller means and said memory means can represent a microcomputer or processor system of a mobile communication device or a communication processor of a vehicle. The mobile device can communicate with a vehicle information and entertainment system by a wireless connection, e.g. Bluetooth, WLAN or the like, and can use an abstract level of communication according to a specific communication protocol based on a standardized set of abstract input and output methods AMx.
Alternatively, said communication can be based on a specific vehicle communication protocol, whereby the HMI unit resides in the mobile communication device and can convert abstract input and output methods AMx into I/O methods according to said specific communication protocol. Thus, applications running on a communication mobile device, such as Email, SMS, voice communication services, navigation applications or the like, can also have access to I/O devices hosted by said vehicle internal communication device.
The inventive HMI unit and the I/O method provided by said HMI unit for example suggest to request one or multiple physical I/O units, such as a touchscreen display, a display with keyboard, an ensemble of signal lights and selection switches, to provide a selection of an item from a list, but does not specify which device should represent the list and how the list should be represented and how the selection should be made. Each communication device independently decides how the list shall be presented and how the selection shall be made. This decision may depend on external conditions, such as driving conditions, source and flow of information etc. The HMI unit converts a specific I/O request into a device-independent request, transmits the request to one or multiple communication devices, whereby each communication device interprets the request independently and performs the request individually. There are neither structural nor functional dependencies between multimedia and communication system or the connected communication devices. Each communication device has to comply with a specific set of API instructions and has to interpret communication requests in conformity with device-specific constraints. In this way; various communication devices can be integrated in an information and entertainment system of a vehicle.
According to an advantageous embodiment, said abstract I/O method AMx can comprise a set of input methods in the form of a graphical selection input, a textual input, an audio input, a visual recognition input and/or a combination thereof; and a set of output methods can comprise an audio output, a visual output, a mechanical output and/or a combination thereof. Thus, an output of data by said application running inside a communication device can be made textually or graphically in the form of a visual output, in the form of a voice or a sound output or in the form of a mechanical output; and an input can be made either in the form of a mechanical input using a keyboard, switches, buttons, a joystick, a slider, a proximity sensor etc., an audio input in the form of speech recognition; or an optical input in the form of gesture recognition or the like. A plurality of different input and output methods provide adaptability of data input and output according to various driving, vehicle and environmental situations and can advantageously be used for reducing the distraction level of the data input and the data output and for enhancing the driving safety and comfort for the user by selecting an optimized I/O unit with the highest possible comfort and the lowest possible distraction level.
According to another advantageous embodiment, said addressed HMI endpoint user HEx can be at least a driver and one passenger or passengers, preferably a front seat passenger and a back seat passenger. Furthermore, said vehicle state VSx can be at least a vehicle parked, a vehicle locked, a vehicle moving and/or a vehicle stopped state. Furthermore, said driving state DSx can be at least a slow driving, a fast driving and/or a reverse driving state and/or said environmental condition state ECx can be at least traffic density, visibility and/or noise level. For selecting an appropriate I/O unit information about addressed HMI endpoint user HEx, a transmitter or a receiver of data are important. For selecting an I/O unit as output unit data to be output can comprise information about an addressed HMI endpoint user (name of user, position of user in vehicle, output to all persons in vehicle), whereby regarding the user type it should at least be distinguished between driver and passenger. The visual output of data to a driver should be avoided during difficult driving conditions, such as a night drive or a drive in heavy traffic. Data to be output to back seat passengers can be represented via a display integrated in the head rest of a front seat, data to be output to a driver can be represented in the form of an audio speech output. The HMI endpoint user as a source of input data can be determined by a physical location of the I/O unit via which data is input, e.g. a keyboard at a passenger seat indicates a back seat passenger as a HMI endpoint user. Furthermore, the content of the input data or an ID of a mobile device can indicate a HMI endpoint user. External properties, such as vehicle state, driving state or environmental condition can be considered when selecting one or multiple I/O units for inputting or outputting data. For instance during a vehicle park condition, a driver is allowed to use a keyboard for inputting data, and output data is displayed in the form of textual data on a display. During driving, when the background noise is increased, the volume of a speech output can be raised, or data visually output at night can be represented in a dimmed mode.
According to another advantageous embodiment, said controller means can process a function F depending on at least one type of a requested abstract I/O method AMx, an addressed HMI endpoint user HEx, a vehicle state VSx, a driving state DSx and/or an environmental condition ECx for selecting at least one physical I/O unit PHEx and addressing said selected physical I/O unit PHEx via said I/O interface means I/O-IM (24) for inputting or outputting said data, whereby input data of a physical I/O unit PHEx is converted by said controller means into an abstract input method AMix for communicating said data via said API interface means API to said communication application means APP; and an abstract output method AMox is converted by said controller means into an output request of data to be output by one or more selected physical I/O units PHEx.
To facilitate the integration of non-automotive applications into a vehicle environment, this embodiment introduces an abstraction function F that maps high-level HMI functions found on any mobile computing device or computer into vehicle-specific interface functions (the HMI abstraction layer). One key element of function F serves to provide a single set of application programming interfaces (APIs) to the developer community independently of different vehicle models and allows the specific implementation of the HMI, thus enabling rapid scaling of applications across vehicle types.
Additionally, the context in which F is executed is considered, e.g. whether the interaction was triggered by the driver or by a passenger. This causes the second key element of F to provide situational awareness within this framework and frees application developers from understanding all vehicle contexts and their limitations. One example is represented by an SMS handling messaging application. When an incoming message is received, the application would execute F to provide the message content. If the vehicle is moving at this time and if the recipient is the driver, the message may be rendered via text-to-speech. If the recipient is a passenger or if the vehicle is parked, the message may be rendered in the form of a text on an appropriate display within the vehicle. However, from the application perspective, a single API framework for access to generic input/output functionality abstracts the developer from the mechanics and HMI implementation of a specific vehicle model.
According to another advantageous embodiment and in line with the aforementioned embodiment, said HMI endpoint user HEx can be selected depending on a command of said communication application means APP and/or depending on a physical location of said physical I/O unit PHEx inside said vehicle, and/or said memory means comprising user preference data and/or priority data of said abstract input or output method AMx considered by said function F for selecting a physical I/O unit PHEx or an HMI endpoint user HEx. Priority data can be used to output data to a passenger or a driver with higher intensity, for instance with increased volume of a loudspeaker or increased brightness and contrast of a display. A high priority data output can overrule output restrictions and can be visually represented to a driver, via sound output and also via mechanical vibrations even in the case of a difficult driving situation. Data of high priority for instance can be an out-of-fuel warning, an emergency call or a vehicle failure warning.
In a further aspect of the invention a communication device is proposed, comprising a human machine interface unit according to one of the foregoing embodiments and being wire-based or wirelessly connected to at least one vehicle state unit VS in the form of a vehicle lock sensor, a vehicle park sensor, a vehicle stop sensor, a vehicle move sensor, and/or a window/roof open sensor; a driving state unit DS in the form of a speed sensor and/or a driving direction sensor; an environmental state unit EC in the form of a weather sensor, a traffic information system, a noise level sensor. The proposed communication device comprises an embodiment of an aforementioned HMI unit and provides an API with a set of abstract I/O methods for an application of said communication device or for an application of an external communication device such that an external application can use the physical I/O units of said communication device.
According to another advantageous embodiment, said communication device can comprise a wire-based or wireless communication connection line connecting one or a plurality of external physical I/O units PHEex to said human machine interface unit for inputting data from or outputting data to said physical external I/O unit PHEex. A wireless communication connection line can be based on a Bluetooth connection, a WLAN connection or another near field radio-based, optical or acoustical data exchange method such that data can be input in or output to an external physical I/O unit.
According to another advantageous embodiment of said communication device, said physical I/O unit PHEx can comprise a set of visual output units in the form of a text display, a graphic display, a head-up display and/or a signal light; a set of audio output units in the form of a loudspeaker and/or a buzzer; a set of mechanical output units in the form of a shock vibrator, a vibrating seat and/or a vibrating steering wheel, and a set of mechanical input units in the form of a keyboard, a switch, a dial, a button, a joystick, a slider and/or a touchscreen; a set of audio input units in the form of a microphone; and/or a set of visual input units in the form of a camera. A camera designed as a visual input unit can use a gesture recognition method for inputting data or commands by distinct gestures of a user, e.g. movements of a user's hand or head.
A further aspect of the invention proposes a method for inputting and outputting data for a communication device in a vehicle comprising the following steps:
According to an advantageous embodiment of said method, said input methods for inputting data can comprise a graphical selection input, a textual input, an audio input, a visual recognition input and/or a combination thereof, and said output methods for outputting data can comprise an audio output, a visual output, a mechanical output and/or a combination thereof. Thus, data can be visually output by an optical signal or on a display, such as a head-up display, can be superimposed on a display of an instrument board of the car, or can be output via sound, such as a spoken text or a buzzer sound, or in the form of a mechanical output, such as a vibration of a part of a user's seat, arm rest, steering wheel, gearshift-lever or joystick. Data can be input by mechanical selection, e.g. finger-tipping on a head-up display, a keyboard input, a switch, a button or a slider, or can be input via speech recognition or else optically via gesture recognition. The method selects an optimal way of representing data or to request an input of a user depending on the vehicle and driving situation and environmental conditions such that the human machine communication is non-distractive, clear, adequate and consistent. A driver may select an input device independently or the method may prompt a specific input and/or output method, e.g. a touch screen dialog. Thus the method may restrict input devices to one or specified devices, e.g. if a user has to choose from a list, an input device can be a touch screen and/or a dial wheel input device. During normal operation the method can allow all kind of input devices for inputting data, e.g. selecting an item from a list, but in a difficult driving condition an input device may be restricted to buttons and knobs integrated in the steering wheel.
According to an advantageous embodiment of said method, said addressed HMI endpoint user HEx can be at least a driver, a passenger, preferably a front seat passenger and a back seat passenger; said vehicle state types VSx comprising at least a vehicle parked, a vehicle locked, a vehicle moving and/or a vehicle stopped state; said driving state types DSx comprising at least a slow driving, a fast driving and/or a reverse driving state; said environmental condition state types. ECx comprising at least traffic density, visibility and/or noise level.
According to an advantageous embodiment of said method, a function F can select said physical I/O unit PHEx depending on at least one type of a requested abstract I/O method AMx, an addressed HMI endpoint user HEx, a vehicle state VSx, a driving state DSx and/or an environmental condition ECx, and can address said selected physical I/O unit PHEx for inputting or outputting said data, whereby input data of a physical I/O unit PHEx is converted into an abstract input method AMix for inputting said data into said communication application means APP, and an abstract output method AMox is converted into an output request of data for outputting said data to one or more selected physical I/O units PHEx. The function F can be executed by a controller unit and can consider data, such as priority information data or user preference data, stored in a memory means of said HMI unit for deciding on a physical I/O unit.
According to an advantageous embodiment of said method, said HMI endpoint user HEx can be selected depending on a command of said communication application means APP and/or by a physical location of said physical I/O unit PHEx inside said vehicle, and/or said function F can consider user preference data and/or priority data of said abstract input or output method AMx for selecting a physical I/O unit PHEx or a HMI endpoint user HEx. A HMI endpoint user can be selected by analyzing a content or a receiver of data to be output or by a type of data (driver-related or passenger-related data). Furthermore, a location of an I/O unit can determine a HMI endpoint user for outputting requested data. User preference data can be considered for selecting a preferred way of inputting or outputting data, e.g. data output or a request input addressed to a child can be represented differently than an I/O action addressed to an adult. A user may prefer a textual or graphical representation of an I/O action or may prefer an audio I/O mode.
According to an advantageous embodiment of said method, said physical I/O unit PHEx can be selected among at least one internal physical I/O unit PHEix of said communication device and/or an external internal I/O unit PHEex of said vehicle for inputting data from or outputting data to said I/O unit PHEx. The function F can decide if data can be output or input either by an internal or an external I/O unit and can determine a HMI endpoint user by identifying the location of an active I/O unit.
In another embodiment of the invention a computer program for execution in a data processing system of said human machine interface unit is proposed, comprising software code portions for performing a method according to anyone of the above mentioned embodiments when said program is run on a computer of said human machine interface unit. The computer program can be adapted to be downloaded to a human machine interface unit control unit or one of its components when run on a computer which is connected to the internet.
Furthermore, the invention can take the form of a computer program product accessible from a computer-usable or computer readable medium providing program code for use by or in connection with a computer or any instruction execution system of a human machine interface unit of a vehicle. For the purposes of this description, a computer-usable or computer readable medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium. Examples of a computer-readable medium include a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk and an optical disk. Current examples of optical disks include compact disk-read-only memory (CD-ROM), compact disk-read/write (CD-R/W) and DVD.
Hereinafter, the invention will be described in greater detail with reference to the attached drawings. These schematic drawings are used for illustration only and do not in any way limit the scope of the invention. In the drawings:
The abstraction function F 80 provides the HMI 10 with a guideline by means of which general applications APP 22 in the vehicle 14 can have access and retrieve user input from a physical HMI I/O unit 26 in the I/O vehicle 14. The HMI 10 can communicate with the APP 22 by a set of abstract I/O methods AM1 . . . Nm comprising:
Upon being triggered by an application APP 22, F 80 can consider the following HMI endpoint users HEx 28 as originators for its internal processing:
Additionally, upon being triggered by an application APP 22, function F 80 can consider the following vehicle states VS1 . . . 4 for its internal processing:
Based on a request of the application APP 22 or the user 28, an I/O unit 26, and/or a vehicle state VS, a driving state DS or an environmental condition state EC, the function F 80 determines the appropriate interface function to access the set of physical I/O units 26 PHEx in the vehicle 14. Such physical I/O units 26 can comprise, but are not limited to:
a and 4b schematically illustrate tables of various criteria for function F 80 according to an embodiment of the invention. While the abstract I/O method AM1 . . . N is selected by the application 22 or by the user 28 interacting with the application 22 in the course of normal operation, the incorporation of the HMI endpoint user 28, the I/O unit 26 where data is input, and/or the vehicle state VS1 . . . Nv, the driving state DS1 . . . Nd, and/or the environmental condition EC1 . . . Ne to determine which PHEix shall be used, i.e. the main body of function F 80, is defined as a policy by the vehicle manufacturer or by an other relevant entity with legitimate control over the vehicle behavior definition. All elements of function F 80 are implemented within a HMI abstraction layer, which is part of the computing platform within the vehicle 14. This enables a vehicle manufacturer to have no vehicle safety logic residing within the applications 22. In one embodiment of function F 80, such policy is implemented via the use of a two-dimensional or multi-dimensional lookup table for each action of AM1 . . . N as depicted in
According to
The present invention is not limited to the above examples, but may be varied freely within the scope of the appended claims.
