Patent Application
20240134505
The present disclosure relates to a human machine interface (HMI), including an HMI for an augmented reality (AR) or virtual reality (VR) environment
In virtual and/or augmented reality applications (e.g., those implemented on AR helmets or smart glasses), allowing users to input one or more sentences is a desirable function, which enables various levels of human-computer interaction, such as sending messages or virtual-assistant dialogs. Compared to the common messaging apps and voice assistants like Alexa, in augmented reality environments, multiple modalities including text, speech, eye gaze, gesture, and environmental semantics can potentially be applied jointly in sentence inputting, as well as in text editing (e.g., correcting/editing one or more words in a previously input sentence), in order to achieve the highest input efficiency. The optimal way to integrate the modalities may vary for different usage scenarios, thus one modality may not be efficient for one input task, but may be efficient for another input task.
For the task of text inputting, various modalities have been explored, such as key-touching with finger(s) on a virtual keyboard, finger-swiping on a virtual keyboard, eye-gaze based key selection with a virtual keyboard, and speech. However, for each of those previous systems, typically only one major modality may be involved as the input method, ignoring the various needs of users in different usage scenarios (e.g., the user may not be willing to speak out in public to type in text with private or confidential content). In addition, in previous virtual/augmented reality applications, the text-editing function to allow user to correct/change certain word(s) in the inputted text sentence is often very limited or even absent, although both virtual keyboard and speech based text inputting may generate errors in the inputting result.
A first embodiment discloses, a virtual reality device that includes a display configured to output information related to a user interface of the virtual reality device, a microphone configured to receive one or more spoken word commands from a user upon activation of a voice recognition session, an eye gaze sensor including a camera, wherein the eye gaze sensor is configured to track eye movement of the user, and a processor in communication with the display and the microphone, wherein the processor is programmed to in response to a first input from an input interface of the user interface, output one or more words of a text field of the user interface, in response to an eye gaze of the user exceeding a threshold time, emphasizing a group of one or more words of the text field associated with the eye gaze, toggle through a plurality of words of only the group utilizing the input interface, in response to a second input from the user interface associated with the toggling, highlighting and editing an edited word from the group, and in response to utilizing contextual information associated with the group of one or more words and a language model, outputting one or more suggested words associated with the edited word from the group.
A second embodiment discloses a system including a user interface that includes a processor in communication with a display and an input interface including a plurality of modalities of input, the processor programmed to in response to a first input from the input interface, output one or more words of a text field of the user interface, in response to a selection exceeding a threshold time, emphasizing a group of one or more words of the text field associated with the selection, toggle through a plurality of words of the group utilizing the input interface, in response to a second input from the user interface associated with the toggling, highlighting and editing an edited word from the group, in response to utilizing contextual information associated with the group of one or more words and a language model, outputting one or more suggested words associated with the edited word from the group, and in response to a third input, selecting and outputting one of the one or more suggested words to replace the edited word.
A third embodiment discloses, a user interface that includes a text field section and a suggestion field section, wherein the suggestion field section is configured to display suggested words in response to contextual information associated with the user interface. The user interface is configured to, in response to a first input from an input interface, output one or more words of the text field of the user interface, in response to a selection exceeding a threshold time, emphasizing a group of one or more words of the text field associated with the selection, toggle through a plurality of words of the group utilizing the input interface, in response to a second input from the user interface associated with the toggling, highlighting and editing an edited word from the group, in response to utilizing contextual information associated with the group of one or more words and a language model, outputting one or more suggested words at the suggestion field section, wherein the one or more suggested words are associated with the edited word from the group, and in response to a third input, selecting and outputting one of the one or more suggested words to replace the edited word.
Embodiments of the present disclosure are described herein. It is to be understood, however, that the disclosed embodiments are merely examples and other embodiments can take various and alternative forms. The figures are not necessarily to scale; some features could be exaggerated or minimized to show details of particular components. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to variously employ the embodiments. As those of ordinary skill in the art will understand, various features illustrated and described with reference to any one of the figures can be combined with features illustrated in one or more other figures to produce embodiments that are not explicitly illustrated or described. The combinations of features illustrated provide representative embodiments for typical applications. Various combinations and modifications of the features consistent with the teachings of this disclosure, however, could be desired for particular applications or implementations.
In this disclosure, the system may propose an advanced multi-modal virtual augmented-reality text-inputting solution, which may enable the user to: (1) select one inputting method involving a certain modality/modalities to input text sentence based on the user's usage scenario and (2) to conduct text editing (e.g., correct/change one or more words in the inputted sentence when necessary) with certain convenient modality/modalities. The modality sets involved for text-sentence inputting and text editing may be different, chosen by the user to maximize the system usability and text-input efficiency. For example, the user may choose to use speech to input the text sentence, but use a virtual keyboard to correct a mis-recognized name in one embodiment. In another case, the user may prefer to use a virtual keyboard to input a confidential text sentence, but may choose speech as the modality to edit some insensitive words in the inputted sentence.
In this disclosure, the system proposed may include a multi-modal text-inputting solution for virtual/augmented reality applications, such as a smart glasses. The solution may in general composed of three steps. A first step may include inputting a text sentence(s) by certain method, which involves one or more modalities. The inputted sentence(s) includes one or more erroneous words, or the user wants to change certain word(s). For each of those words to edit, the user may conduct a second step and select the word to edit by a certain method of input modality, which involves on or more modalities. In the third step, the user may edit the selected word by certain method, which involves one or more modalities.
Display 20 is configured to be at least partially see-through, and includes right and left display regions 20A, 20B which are configured to display different images to each eye of the user. The display may be a virtual reality or augmented reality display. By controlling the images displayed on these right and left display regions 20A, 20B, a hologram 50 may be displayed in a manner so as to appear to the eyes of the user to be positioned at a distance from the user within the physical environment 9. As used herein, a hologram is an image formed by displaying left and right images on respective left and right near-eye displays that appears due to stereoscopic effects to be positioned at a distance from the user. Typically, holograms are anchored to the map of the physical environment by virtual anchors 56, which are placed within the map according to their coordinates. These anchors are world-locked, and the holograms are configured to be displayed in a location that is computed relative to the anchor. The anchors may be placed in any location, but are often placed in positions at locations where features exist that are recognizable via machine vision techniques. Typically, the holograms are positioned within a predetermined distance from the anchors, such as within 3 meters in one particular example.
In the configuration illustrated in
In addition to visible light cameras 18, a depth camera 21 may be provided that uses an active non-visible light illuminator 23 and non-visible light sensor 22 to emit light in a phased or gated manner and estimate depth using time-of-flight techniques, or to emit light in structured patterns and estimate depth using structured light techniques.
Computing device 10 also typically includes a six degree of freedom inertial motion unit 19 that includes accelerometers, gyroscopes, and possibly magnometers configured to measure the position of the computing device in six degrees of freedom, namely x, y, z, pitch, roll and yaw.
Data captured by the visible light cameras 18, the depth camera 21, and the inertial motion unit 19 can be used to perform simultaneous location and mapping (SLAM) within the physical environment 9, to thereby produce a map of the physical environment including a mesh of reconstructed surfaces, and to locate the computing device 10 within the map of the physical environment 9. The location of the computing device 10 is computed in six degrees of freedom, which is important to displaying world-locked holograms 50 on the at least partially see through display 20. Without an accurate identification of the position and orientation of the computing device 10, holograms 50 that are displayed on the display 20 may appear to slightly move or vibrate relative to the physical environment, when they should remain in place, in a world-locked position. This data is also useful in relocating the computing device 10 when it is turned on, a process which involves ascertaining its position within the map of the physical environment, and loading in appropriate data from non-volatile memory to volatile memory to display holograms 50 located within the physical environment.
The IMU 19 measures the position and orientation of the computing device 10 in six degrees of freedom, and also measures the accelerations and rotational velocities. These values can be recorded as a pose graph to aid in tracking the display device 10. Accordingly, even when there are few visual cues to enable visual tracking, in poorly lighted areas or texture-less environments for example, accelerometers and gyroscopes can still enable spatial tracking by the display device 10 in the absence of visual tracking. Other components in the display device 10 may include and are not limited to speakers, microphones, gravity sensors, Wi-Fi sensors, temperature sensors, touch sensors, biometric sensors, other image sensors, eye-gaze detection systems, energy-storage components (e.g. battery), a communication facility, etc.
In one example, the system may utilize an eye sensor, a head orientation sensor or other types of sensors and systems to focus on visual pursuit, nystagmus, vergence, eyelid closure, or focused position of the eyes. The eye sensor may include a camera that can sense vertical and horizontal movement of at least one eye. There may be a head orientation sensors that senses pitch and yaw. The system may utilize a Fourier transform to generate a vertical gain signal and a horizontal gain signal.
The system may include a brain wave sensor for detecting the state of the user's brain wave and a heart rate sensor for sensing the heart rate of the user. The brain wave sensor may be embodied as a band so as to be in contact with a head part of a user, or may be included as a separate component in a headphone or other type of device. The heart rate sensor may be implemented as a band to be attached to the body of a user so as to check the heart rate of the user, or may be implemented as a conventional electrode attached to the chest. The brain wave sensor 400 and the heartbeat sensor 500 calculate the current brain wave state and the heart rate of the user so that the controller can determine the order of the brain wave induction and the speed of the reproduced audio according to the current brain wave state or heart rate of the user. And provides the information to the control unit 200.
The system may include an eye tracking system. The head mounted display device (HMD) may collect raw eye movement data from at least one camera. The system and method may utilize the data to determine the location of the occupant's eyes. The system and method may determine eye location to determine the line of sight of the occupant.
The system thus includes a multitude of modalities to utilize as an input interface connected to the system. The input interface may allow a user to control certain visual interfaces or graphical user interfaces. For example, the input interface may include buttons, controllers, joy sticks, mouse, or user movement. In one example, a head nod left may move a cursor left, or a head nod right may move a cursor right. The IMU 19 may be utilized to gauge the various movement.
The user may enter a letter of a word by first selecting the coarse group and then the fine group the letter belongs to. For example, if a user wants to type “h,” the coarse group is selected, the fine group is right. Thus, a user may make two selections for each letter input under an embodiment of the disclosure.
Because each fine group may be associated to a coarse group, selecting a coarse group narrows the selection space for the fine group. Thus, the fine group may be a subset associated with the coarse group subset. With the example grouping, selecting each fine group individually may require nine options (e.g., such as a T9 keyboard), whereas selecting a coarse and fine group requires six options: three for selecting the coarse group and three more for selecting the fine group within the selected coarse group in one embodiment. This is may be advantageous when the degrees of interaction are limited, such as when there is limited space on a physical controller. The spacing between the coarse sections and the size of the keyboard (distance from user) can also adjusted by the user to fit their preferences. Thus, layout 211 is an embodiment of an alternative keyboard layout.
Users can use a single device to perform the letter selection in one embodiment. In another embodiment, the user may also use multiple devices such as controllers, buttons, joysticks, and trackpad to make a selection.
The final selection of the “fine” selection may be a group of three or two characters, but can be any amount of characters (e.g., four characters or five characters). In one example, the “coarse” selection may mean a selection among three regions (e.g., left, middle, and right regions). Next, once a region of the coarse selection is selected, the “fine” selection may go ahead to select a row in the selected region. There may be three rows in each region. For example, “e,d,c” is the right row of the left region. Note that in right region, the three rows may be “u,j,m”, “I,k”, and “o,l,p”, respectively.
The system will accordingly list possible words in the word list section on the screen (the possible words may be selected based on the language model). In most cases, the user may see the suggested/predicted word e.g., the word he/she intends to input) in the word list, and select it. For example, if the user wants to input “we”, the user may only need to select the row “w,s,x” and “e,d,c”, and the interface may output the word “we” in the suggestion section to be selected. Thus, the system may predict a word based on a selection of a group of characters (e.g., not a single character). This may include a group of two or three characters, for example.
In another example, in a situation that the user cannot find the wanted word in the word list, the user can switch to the three-step input method, which uses an additional step after step2 above to select one character, i.e., explicitly tells system which character to choose in a row.
The input interface may include mobile devices include but are not limited to controllers, joysticks, buttons, rings, eye-tracking sensors, motion sensors, physiological sensors, neuro sensors, and trackpads. Table 1 is the combination of multi-device interaction. Hand gesture and head gesture can also be used in Coarse-n-Fine keyboard. Table 1 is shown below:
While Table 1 is one example, any modality may be utilized for a first coarse selection and any modality may be utilized for any fine selection. For example, a remote control device may be utilized for the coarse selection and the fine selection. Furthermore, the same or different modalities may be utilized for either selection or for both selections.
One of the simplest LM may be the n-gram model. An n-gram is a sequence of n words. For example, a bigram may be a two-word sequence of words like “please turn”, “turn your”, or “your homework”, and a trigram maybe a three-word sequence of words like “please turn your”, or “turn your homework”. After trained on text corpora (or a similar model), an n-gram model can predict the probability of the next word given the previous n−1 words. More advanced language models, such as pre-trained neural-network based models, may be applied to generate better probability estimation of the next word based on longer word history (e.g., based on all the previous words).
In one disclosure, leveraging certain language models, the system can predict the next word given the existing input and the characters. As
When a word is highlighted longer than a threshold time (e.g. threshold time B), the word may be viewed as the selected word to edit. Thus, the system may allow for a further step to edit that word (e.g., either selecting a suggested or manually inputting the words) and allow for another step that allows for such editing. In one example, once the editing is done for that word, the edited word may remain highlighted, and the user may use left/right gesture/button to move to the next word to edit. If no gesture or button pressing is detected for a time period longer than a third threshold or time-out (e.g. time threshold C), the editing task is considered completed. In another implementation, the system may directly utilize eye gazing of the user to select/highlight each word to edit by simply looking at the word for a time period longer than a fourth threshold (e.g. threshold D).
In such an example, if the list of alternatives or suggested words is not provided in certain system implementation, the proposed solution proceeds to another step that allows for manual entry, and thus to provide multiple methods to user to choose in order to input one or more words as the editing result. Any method (e.g., virtual-keyboard based text inputting, speech based inputting, finger/hand motion based inputting) that allows the user to input text word(s) and replace the target word (e.g. highlighted word) to edit with the inputted word(s) can be included into the system as one supported input method for user to choose. In one example, similar to the design shown in
The disclosure also allows for an alternative embodiment to support additional learning mechanism for selecting a suggested word. In such an embodiment, the learning mechanism may attempt to avoid the repetitive happening of a same system mistake (e.g., the ASR engine mistakenly recognizes one name into another for speech based text-inputting), with user's assistance through additional HMI (i.e., human-machine interaction) design. Such learning mechanism can be implemented with various machine learning algorithms. In such an embodiment, the system may utilize a learning strategy based on the type of each edited word, (1) with available environmental knowledge (e.g., contact names in the user's address book, emails, text messages, chat history, and/or browser history, time of day, day of the week, month, etc.) considered and (2) collecting user's confirmation from an additional HMI design when necessary. When the editing is completed for an input sentence, the system may first adopt a Named Entity Recognizer (NER) to detect the different types of names in the edited region of the sentence. For example, in the input sentence “send charging a message” (as shown in
With all the given choices of input modalities in each step, the user may be allowed the freedom to choose a desired method for each step according to the usage scenario, making the maximization of system usability and text-inputting efficiency possible. Each modality (e.g., input interface) has its own advantage and disadvantages. For example, a speech-based input method is in general efficient, while it may not be able to work in highly noisy environment, it may fail to recognize unusual names/terms, and may not be suitable to input confidential message in public space. In the meanwhile, the virtual-keyboard based input method may be relatively less efficient, but it can handle the input of confidential messages as well as the input of unusual names and terms very well. With the freedom to choose various input modality, the user can thus choose the appropriate/suitable input/edit method based on the needs in each step in real application scenario. For instance, when privacy is not a concern and environment noises are low, the user may choose to use a speech input (e.g., microphone to input sentence by speech). In case that a speech recognition error (e.g., failing to recognize an unusual name like “Jiajing”) happens, the user may edit the erroneous word by typing the correct word with the virtual keyboard, or any other input modality. In another instance, when privacy is a concern, the user may choose to use the virtual keyboard to input a sentence. In case that the user wants to correct or change a word in the inputted sentence, the user may edit the word by simply saying the desired word, especially if that word is not privacy sensitive. Note that the environment scenario may change from time to time through the use of a virtual/augment reality device. The disclosure below enables the user to always choose a suitable combination of input and editing methods to fit the user's needs and maximize the text-inputting efficiency under the specific usage circumstances.
While exemplary embodiments are described above, it is not intended that these embodiments describe all possible forms encompassed by the claims. The words used in the specification are words of description rather than limitation, and it is understood that various changes can be made without departing from the spirit and scope of the disclosure. As previously described, the features of various embodiments can be combined to form further embodiments of the invention that may not be explicitly described or illustrated. While various embodiments could have been described as providing advantages or being preferred over other embodiments or prior art implementations with respect to one or more desired characteristics, those of ordinary skill in the art recognize that one or more features or characteristics can be compromised to achieve desired overall system attributes, which depend on the specific application and implementation. These attributes can include, but are not limited to cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, serviceability, weight, manufacturability, ease of assembly, etc. As such, to the extent any embodiments are described as less desirable than other embodiments or prior art implementations with respect to one or more characteristics, these embodiments are not outside the scope of the disclosure and can be desirable for particular applications.
