Patent Grant
10261674
The present disclosure is directed to an intuitive and powerful mechanism for entering and/or editing textual input on a computing device. While the concepts of this disclosure are compatible with text-entry devices that have very small displays, the concepts are not limited to any particular device form factor.
As discussed in more detail below, text-entry devices may include touch display sensors that allow a user to input text via a virtual keyboard presented on a display; touch sensors that allow a user to draw text on a display or a non-display touch area; speech recognition systems that allow a user to enter text via spoken words; eye-tracking systems that allow a user to enter text via gaze; gesture recognition systems that allow a user to trace text shapes in air; and/or other text input mechanisms.
When included, a text-entry device's virtual keyboard may take full advantage of a small display size by filling substantially all of the display. In particular, the virtual keyboard may be presented without a significant portion of the display reserved for showing the user a letter-by-letter reproduction of what is being typed or what has already been typed. In some implementations, only the last completed word, or a portion thereof, is displayed with the keyboard—letters of an uncompleted word are not displayed until the word is completed, and the penultimate word is not displayed with the last completed word. In some implementations, a last completed word, or a portion thereof, is displayed immediately adjacent the right- or left-most key in a bottom row of the virtual keyboard. The last completed word, or a portion thereof, may additionally or alternatively be displayed (e.g., superimposed) over the keyboard temporarily (e.g., for a set period of time and/or until a suitable input is received). The superimposed word or portion thereof may be partially transparent and/or the appearance of the keyboard may be altered (e.g., dimmed) during display of the superimposed text in some examples. Optionally, a placeholder (e.g., asterisks or bullet) may be displayed in these same areas responsive to each recognized key stroke.
In some implementations, two or more different virtual keyboards may cooperate to provide a user a greater number of text input options. As an example, one keyboard may include 26 lower case letter keys; another keyboard may include 10 numeric keys; another keyboard may include one or more symbols (e.g., @, #, $, %, &, and/or others) and/or emoticons; another keyboard may include 26 upper case letter keys; and another keyboard may include one or more foreign language letters or symbols (e.g., Hanzi, Kanji, and/or others). In addition to the primary keys of a given keyboard, some keyboards may also include a space key, one or more modifier keys, one or more punctuation keys (e.g., period, comma, question mark), and/or other supplemental keys. When two or more keyboards are included, a user may selectively toggle between the keyboards with a touch swipe, voice command, gesture, gaze, and/or other user command. In some implementations, keys of one keyboard may provide direct links to different keyboards.
The computing device may leverage an on-board word-level recognizer 104 or off-board word-level recognizer 106 (e.g., within a remote computing device 108) that analyzes all input between text delimiters and recommends candidate text based on the word-level analysis. In some embodiments, (e.g., during utilization of keyboards relating to character-based languages), each input character may be processed and words may be determined based on natural word boundaries for that language without inputting specific delimiters. Once a text-delimiter, other user indication, or language-based natural word boundary signals a word completion, the word-level recognizer may determine the word or words that the user most-likely intended to input. Before the user indication signals a word completion, the word-level recognizer need not present any word predictions. By avoiding such predictions, the user will not be distracted by potentially unintended word fragments that are inaccurately predicted based on less than all user input for that word. The word-level recognizer may be trained via supervised or unsupervised machine learning and/or may include personalized training data from the computing device or a cloud-based network.
When an off-board word-level recognizer is utilized, the text-entry device may communicate all user input to the off-board word-level recognizer using one or more local or wide area networks. In some implementations, the off-board word-level recognizer may be a module of a mobile telephone or other portable computing device that is physically near the text-entry computing device. In other implementations, the off-board word-level recognizer may be offered as part of a remote cloud-based service. The local text-entry device and the off-board, word-level recognizer may communicate using any suitable protocol and/or technology. Non-limiting examples of suitable protocols and technologies are described in APPENDIX A.
When an off-board, word-level recognizer is used in cooperation with a virtual keyboard, the text-entry device optionally may use very light weight communication with the off-board, word-level recognizer. As one example, the text-entry device may communicate the detected touch contact parameters to the off-board, word-level recognizer. This may be less computationally expensive than translating raw sensor measurements on board and then communicating translation of the raw input.
The text-entry device may provide text editing functionality that allows a user to verify and/or change candidate text that has been recognized using any of the device's text input mechanisms. The text editor is displayed independently and is not part of the virtual keyboard's graphical user interface. By temporally separating display of a virtual keyboard and the text editor, both the virtual keyboard and the text editor are allowed to independently take full advantage of whatever display real estate the text-entry device offers. The virtual keyboard does not have to make any compromises for the text editor or vice versa. In some implementations, a user may selectively toggle between a virtual keyboard and the text editor. Such toggling may be initiated by a touch swipe, voice command, gesture, gaze, greater-than-threshold pause, and/or other event or user command. A toggle between a virtual keyboard and a text editor may include an animation where the virtual keyboard appears to be pushed off the display by the text editor, or vice versa.
The text editor may display candidate text translated from user input (e.g., via a word-level recognizer). In some instances, the candidate text may not be the final text that the user wants. As such, the text editor is configured to allow the user to select any word for editing, delete any word, add new words, and/or make other changes. In some examples, the text editor may provide one or more alternate candidate words for each candidate word.
Turning now to
Due to the relatively small size of the touch display 102, in order to allow a user to effectively visualize and enter touch input to each key of the virtual keyboard, menu option, etc., the user interface 202 may be configured such that only a portion of the user interface is displayed at any given time (e.g., either keyboard page or text editor page). For example, the user interface may be configured as a ribbon, and only a section of the ribbon may be displayed at one time, with each section of the ribbon viewable as a user scrolls through the user interface. Additional information regarding the ribbon configuration of the user interface will be provided below with respect to
It is to be understood that the illustrated arrangement of the virtual keyboard 204 is one example of possible arrangements, and other configurations are possible. For example, rather than having the letters arranged in the QWERTY arrangement illustrated, the letters may be arranged alphabetically or in virtually any suitable arrangement. Further, additional or alternative text delimiters may be present on virtual keyboard 204, including but not limited to a question mark key, exclamation point key, etc. Further still, while only letters are illustrated in virtual keyboard 204, it is to be understood that other characters, such as numbers or symbols, may be additionally or alternatively displayed, as explained in more detail below.
In order to create a text string, a user may touch the virtual keyboard 204. As such, computing device 100 may include a plurality of touch sensors configured to detect touch input. In some examples, more than one letter, number, or symbol may be mapped to the same touch sensor (i.e., the touch sensing resolution may be less than the number of keys).
In the example illustrated in
When the user is entering touch inputs to the user interface 202, in addition to displaying how many touch inputs have been made in the preview area 210, the user interface 202 may also be configured to provide touch input feedback to the user to inform the user where the last touch input occurred. For example, as illustrated in
A plurality of example scenarios for interacting with a user interface displayed on the touch display 102 of computing device 100 will now be described with respect to
Starting with
At time T1, the user interface 304 transitions to display a preview of a menu 308. The menu 308 may allow a user to select a desired keyboard layout, such as a keyboard that displays letters, numbers, or symbols. The preview is displayed as a hint to inform a user that the menu is available to the left of the virtual keyboard 302. At time T2, the user interface 304 further transitions to display a navigation key 310. The navigation key 310 may allow a user to exit out of the virtual keyboard 302, return to a previous user interface display, or execute other operations. At times T3 and T4, the user interface 304 slides back to only display virtual keyboard 302. It is to be understood that the type of keyboard displayed upon initialization of the virtual keyboard may be context-dependent. For example, the keyboard may be initialized responsive to a user input for responding to a received message (e.g., an email, an SMS message, or another suitable message). Accordingly, the keyboard presented responsive to such selection may have a language corresponding to a detected language of the received message. A default type of keyboard may additionally or alternatively be defined by user preference settings.
The initialization animation illustrated in
When the user enters touch input to the virtual keyboard 402, a placeholder symbol 410 is displayed in a preview area 412. For each separate touch input entered between text delimiters, a placeholder symbol is displayed. Accordingly, a first placeholder symbol (in this example, a bullet) is displayed in preview area 412 at time T0, indicating that one touch input has been entered. At time T1, two placeholder symbols are displayed in preview area 412, indicating that two touch inputs have been entered. At time T2, three placeholder symbols are displayed in preview area 412, indicating three touch inputs have been entered. At time T3, four placeholder symbols are displayed in the preview area 412, indicating that four touch inputs have been entered.
At time T4, the user enters touch input to the space key, which as explained above is a text delimiter. As such, the raw sensor data corresponding to the four touch inputs are processed to determine one or more candidate words. Once at least one candidate word is determined, a candidate word is displayed in the preview area 412. In one example, the displayed candidate word may be the word that the word-level recognizer has determined is the most likely word represented by the touch inputs received. In the example touch input scenario illustrated in
While providing input to the computing device 100, the user may access different keyboard layouts in order to expand the number and types of characters that may be entered.
At time T0 in
Turning now to
As described above,
At time T2, a horizontal swiping gesture is being performed, as represented by leftward-moving gesture indicator 714. As a result of the swiping gesture, the user interface 704 is dragged to the left to transition from the keyboard layout selection menu to the selected virtual keyboard (e.g., the symbol virtual keyboard). As illustrated, the navigational key is no longer displayed, the keyboard selection keys of the keyboard layout menu 706 are displayed on a far left side of the display, and additional symbols (e.g., relative to the symbols displayed at time T1) are displayed in the virtual keyboard area. At time T3, the swiping gesture is concluded, revealing the full symbol keyboard. It is to be understood that the illustrated symbols included in the symbol virtual keyboard are exemplary in nature, and any suitable symbols may be included in any suitable arrangement. In addition to the symbols of the symbol virtual keyboard, a delimiter key (e.g., space key 716), a delete key 718, and a preview area 720 are displayed. As illustrated at time T4, selection of a symbol in the symbol virtual keyboard (e.g., as indicated by feedback indicator 722) results in the display of the selected symbol within preview area 720. In other words, a symbol may include an implied delimiter. In some implementations, a placeholder may be displayed until a dedicated delimiter key is selected.
At time T2, a horizontal swiping gesture is being performed, as represented by leftward-moving gesture indicator 812. As a result of the swiping gesture, the user interface 804 is dragged to the left to transition from the keyboard layout selection menu to the selected virtual keyboard (e.g., the numerical virtual keyboard 809). As illustrated, a navigational key is no longer displayed, the keyboard selection keys are displayed on a far left side of the display, and additional numbers (e.g., relative to the numbers displayed at time T1) are displayed in the virtual keyboard area. At time T3, the swiping gesture is concluded, revealing the full numerical virtual keyboard 809. It is to be understood that the illustrated numbers included in the numerical virtual keyboard are exemplary in nature, and any suitable numerical values or other related characters (e.g., mathematical symbols, decimal points, and/or other suitable characters) may be included in any suitable arrangement. In addition to the numbers of the numerical virtual keyboard, a delimiter key (e.g., space key 816), a delete key 818, and a preview area 820 are displayed. As illustrated at time T4, selection of a number (e.g., “9”) in the numerical virtual keyboard (e.g., as indicated by feedback indicator 822) results in the display of the selected number within preview area 820. In other words, a number may include an implied delimiter. In some implementations, a placeholder may be displayed until a dedicated delimiter key is selected.
As described above, example virtual keyboards may be configured to occupy substantially all of a display, leaving little room for viewing entered text/characters. In order to view input that has already been entered (e.g., candidate text that has been recognized, forming at least a portion of a message), a text editor may be accessed by providing user input to the device (e.g., a touch input such as a tap, swipe, slide, or other gesture, a physical button actuation, a detected eye gaze, a voice input, or any other suitable input).
At time T0 in
The preview area is configured to truncate an end portion of the candidate text (e.g., in the event that the candidate text is too large to fit in the preview area) in order to suggest and/or provide a hint to a user as to the location of the remainder of the candidate text. For example, by noting that the candidate text is truncated by a right edge of the display, the user may be encouraged to drag the user interface (e.g., the virtual keyboard) to the left in order to reveal the remaining characters of the candidate text. Accordingly, as illustrated at time T0, a show-text-editor input, such as a leftward horizontal swiping or sliding gesture may be initiated, indicated by leftward-moving gesture indicator 910. In response to the swiping gesture, the user interface 904 may be dragged to the left, as illustrated at time T1. For example, only a right portion of the virtual keyboard may be displayed in a left region of the display, as the user interface transitions from a virtual keyboard to a text editor. The text editor may be partially displayed on a right side of the display, as illustrated by a partial string of candidate text 912 (e.g., a message string for a currently-composed message) and an action button (e.g., “SEND” key 914). It is to be understood that other suitable action buttons may be displayed. In some implementations, the “SEND” key 914 may include text and/or a graphical icon. As also illustrated at time T1, the candidate text of a last-recognized word (e.g., “DEVICE”) may be fully displayed, creating a “bleed-over” effect between the virtual keyboard and the text editor.
The swiping gesture is continued at time T1, as illustrated by the leftward-moving gesture indicator 916. In response, at time T2, the user interface is fully transitioned to the text editor 918 such that the text editor is displayed without the virtual keyboard. The text editor may display a string of candidate text 912 (e.g., recognized inputs during a current text input session and/or prior to selecting an action button such as a send key). The string of candidate text 912 may include words recognized from input to a QWERTY virtual keyboard, symbols selected from a symbol virtual keyboard, and/or numbers selected from a numerical virtual keyboard. In some implementations, the text editor may include a character counter 920. For example, some messages (e.g., SMS text messages), may be limited to include a threshold number of characters (e.g., due to bandwidth parameters and/or communication protocols). Character counter 920 may enable a user to keep track of how many characters have been entered during a text input session (e.g., how many characters are included in a message that is currently being composed).
The text editor 918 may also include a slider bar 922, which illustrates the length of the string of candidate text (e.g., a length of the slider bar may be relative to a length of the string of candidate text) and a currently-viewed location in the string of candidate text. For example, at time T2, the beginning of the string may be in view, resulting in the display of the slider bar 922 in a lower far left corner of the display. As the user navigates along the string at time T3 (e.g., as illustrated by leftward horizontal swiping gesture represented by leftward-moving gesture indicator 924), the string is moved along the display such that a different portion of the string is viewable. Accordingly, at time T3, some of the text that was viewable at time T2 is no longer viewable (e.g., is dragged out of the displayable region toward the left side of the display) and additional text that was not viewable at time T2 is brought into view (e.g., is dragged from the right side of the display into the displayable region). As also illustrated at time T3, the slider bar 922 is moved to the right, indicating that a middle portion of the string of candidate text is being viewed.
As the user continues navigating along the string at time T3 (e.g., as illustrated by leftward horizontal swiping gesture represented by leftward-moving gesture indicator 924), the string is moved along the display such that a different portion (e.g., an end) of the string is viewable. Accordingly, at time T4, some of the text that was viewable at time T3 is no longer viewable (e.g., is dragged out of the displayable region toward the left side of the display) and additional text that was not viewable at time T3 is brought into view (e.g., is dragged from the right side of the display into the displayable region). As also illustrated at time T4, the slider bar 922 is moved to the right, indicating that an end portion of the string of candidate text is being viewed.
As illustrated in
Text-entry device 1000 includes a display 1004 for presenting ribbon graphical user interface 1002, other user interfaces, applications, data, media content, and/or other content. Display 1004 has a first dimension along a first axis and a second dimension along a second axis. In the illustrated implementations, the first dimension is a height dimension and the second dimension is a width dimension. In other implementations, a display may be shaped differently. For example, while display 1004 is formatted with a landscape aspect ratio (i.e., width greater than height), a display with a portrait aspect ratio (i.e., height greater than width), a square aspect ratio, or a non-rectangular aspect ratio may be used. Further, while ribbon graphical user interface 1002 is shown as being horizontally-pannable, a vertically-scrollable graphical user interface and/or a graphical interface that is both vertically-scrollable and horizontally-pannable is within the scope of this disclosure. In some implementations, the orientation of a display may automatically rotate responsive to the text-entry device being rotated.
Ribbon graphical user interface 1002 includes a collection of user interface states or input regions anchored over a contiguous background 1006. The user interface input regions may include a keyboard selection menu, a virtual keyboard, a text editor, and/or any other suitable user interface input region. As used herein, the term anchored means that the user interface input regions move with the contiguous background. In other words, the input regions do not scroll over the background while the background remains stationary. Instead, the input regions and the background move together. It should be understood, however, that some input regions may move relative to the contiguous background (e.g., growing, shrinking, rotating, changing form, and/or exhibiting other suitable relative movements). Furthermore, in some implementations, the background can optionally be scrolled at a different speed than the foreground to give the visual perception of parallax, which may help enforce a three-dimensional metaphor of the user interface.
As shown in
As further illustrated in
In the example illustrated in
As described above, alternate candidate words 1112 may be presented above and below a selected candidate word, allowing a user to scroll through the list of alternate candidate words. In some embodiments, the height and/or touch sensing capabilities of a display may hinder the performance of such vertical scrolling gestures.
In some examples, a predetermined number of alternate candidate words (e.g., the second through fourth most-likely alternate candidate words identified by a word-level recognizer) may be displayed. Depending on the modality, the source of the alternate candidate words may vary. For example, if the text was generated by speech recognition, acoustic speech recognition candidates (e.g., from a word confusion network) may be utilized in addition to language model candidates. In other embodiments, all alternate candidate words may be viewable by horizontally scrolling through the displayed list of alternate candidate words (e.g., by providing a horizontal swiping gesture along a top of the display). At time T2, a user selects a desired alternate candidate word to replace the selected candidate word (e.g., as indicated by feedback indicator 1214). In response, at time T3, the selected alternate candidate word replaces the selected candidate word.
Text/character entry may be input via mechanisms other than touch input on virtual keyboards. For example, the text-entry device may include a microphone to capture voice input, which may be processed to generate recognized candidate words locally or remotely (e.g., via an on-board voice recognition/speech-to-text module within the text-entry device and/or via an off-board voice recognition/speech-to-text module within a computing device located remotely from the text-entry device and communicating with the text-entry device over a communication link such as a wireless communication link). In examples where a remote computing device is utilized to perform voice recognition, raw and/or partially processed voice data may be transmitted from the text-entry device and received by the remote computing device. After performing the voice recognition/speech-to-text processing, the remote computing device may transmit candidate word(s) to the text-entry device.
In some examples, the text-entry device may send raw and/or partially processed voice data to an external computing device (e.g., a smartphone or other mobile computing device, which may be within direct communication range of the text-entry device). The external computing device may perform some or no processing on the received voice data prior to sending the received and/or further processed voice data to a remote service via a network (e.g., a cloud computing device located remotely from the text-entry device and the external computing device). The remote service may convert the voice data received from the external computing device to text and/or perform additional processing and then return the text and/or additionally processed voice data to the external computing device. The external computing device may then forward the text and/or additionally processed voice data to the text-entry device.
The voice input mode of the user interface as described above may be entered from any suitable region of the user interface. For example, the voice input mode may be an additional option in the keyboard layout selection menu described in
Additionally or alternatively, a handwriting, drawing, or shape-writing input mode may be selected via a displayed icon or menu option. The drawing input mode may enable a user to replace and/or enter characters/words by drawing directly on the display with a finger, stylus, or other input device. In some embodiments, characters or shapes may be drawn on a display orientation that is different than the display orientation used for the text editor (e.g., text may be drawn in a portrait orientation and the text editor may be in a landscape orientation). The drawing input may include one or more touch inputs (e.g., a series of continuous touch inputs, each continuous touch input forming or representing a character or a portion of a character) to the display. The touch inputs and/or data corresponding to the touch inputs may be analyzed by the computing device that includes the display and/or may be transmitted to a remote computing device for processing to determine characters or words that correspond to the drawing input.
The determined characters or words may be added to a string of entered text and displayed in the text editor upon completing a drawing input session (e.g., responsive to exiting the drawing input mode). In some examples, the drawing input mode may utilize delimiters similarly to the virtual keyboard in order to trigger character or word recognition. For example, lifting a finger, pausing (e.g., not touching the display or maintaining contact with a single point of the display) for a time that exceeds a threshold, selecting a user interface element (e.g., an “end drawing mode” user interface element), or other delimiter input (e.g., a tap, a swipe gesture, and/or another suitable input) may trigger the character or word recognition of touch inputs received since a last delimiter input or the start of the drawing mode.
At 1602, method 1600 includes presenting a virtual keyboard. As discussed above, the virtual keyboard may be presented to occupy substantially an entire display, thus allowing each key of the virtual keyboard to be as large as the display allows.
At 1604, method 1600 includes recognizing input. Such input may include key strokes (e.g., tap input) on the virtual keyboard, voice input, gaze input, gesture input, hardware button presses, or any other type of input recognizable by the text-entry device.
At 1606, if the recognized input is a command to leave the virtual keyboard, the method ends at 1608. Such a command may be a swipe or other input to toggle to a text editor, open another application, turn the text-entry device off, or perform another operation. If the recognized input is not a command to leave the virtual keyboard, the method advances to 1610.
At 1610, if the recognized input is a delimiter key, the method proceeds to 1612. If the recognized input is not a delimiter key, the method proceeds to 1614, where the detected input is optionally sent to an on-board or off-board word-level recognizer. As discussed below with reference to 1612, instead of sending each input after each input recognition, a plurality of inputs between successive delimiters may be bundled and sent together.
At 1616, method 1600 optionally includes displaying a placeholder with the virtual keyboard. When a placeholder is displayed, it may be an asterisk, bullet, or any other generic symbol that signals that a key has been input, but without distracting the user with a potentially wrong key. In some implementations, an actual letter or other symbol corresponding to the particular input may be used instead. The placeholder may be displayed responsive to the input not selecting any of the delimiter keys.
As indicated at 1618, the placeholder may replace a displayed word that corresponds to input before the last delimiter (i.e., the previous word). As indicated at 1620, a plurality of placeholders may be displayed together, such that each new input before a delimiter will add a new placeholder. In this way, the user can see that each touch input is being recognized. In some implementations, only a threshold number of placeholders will be displayed (e.g., the number of placeholders that will fit on the display without altering the virtual keyboard size or position). In some implementations, the placeholders and/or other aspects of the virtual keyboard page can flash or otherwise visually indicate when touch input is recognized. In some implementations, haptic feedback and/or other mechanisms may be used to signal touch input recognition to the user.
At 1612, method 1600 optionally includes bundling all key inputs between successive delimiter inputs and sending the bundle to an on-board or off-board word-level recognizer. Such bundling may reduce the number of communications necessary to convey the input, for example. Alternatively, as discussed above with reference to 1614, each input may be sent individually.
At 1622, method 1600 includes receiving a suggested candidate text from the word-level recognizer. At 1624, the received suggested candidate text is displayed with the virtual keyboard. This new suggested candidate text replaces any candidate text from a previous word or any placeholder(s) for the current word (e.g., all currently displayed placeholders) responsive to the input selecting one of the delimiter keys. In this way, at most one word is displayed with the keyboard at any given time. As such, the keyboard need not be made smaller to accommodate a larger area for displaying entered text with the virtual keyboard. As indicated at 1626, the displayed candidate text may be truncated so that a portion of a displayed word bleeds off the edge of the display. Alternatively, a size of the candidate text may be reduced so that an entire word fits on the display with the virtual keyboard.
In some examples, in addition to or as an alternative to replacing placeholders with the candidate text, the new suggested candidate text may be temporarily displayed over the keyboard. For example, the new suggested candidate text may be displayed over the keyboard in a semi-transparent manner and/or the keyboard may be faded (e.g., in a similar manner to the fading described above with respect to times T0-T3 in
The methods and processes described herein are tied to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as a computer-application program or service, an application-programming interface (API), a library, and/or other computer-program product.
Computing system 1700 includes a logic machine 1702 and a storage machine 1704. Computing system 1700 may optionally include a display subsystem 1706, input subsystem 1708, communication subsystem 1710, and/or other components not shown in
Logic machine 1702 includes one or more physical devices configured to execute instructions. For example, the logic machine may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, achieve a technical effect, or otherwise arrive at a desired result.
The logic machine may include one or more processors configured to execute software instructions. Additionally or alternatively, the logic machine may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. Processors of the logic machine may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and/or distributed processing. Individual components of the logic machine optionally may be distributed among two or more separate devices, which may be remotely located and/or configured for coordinated processing. Aspects of the logic machine may be virtualized and executed by remotely accessible, networked computing devices configured in a cloud-computing configuration.
Storage machine 1704 includes one or more physical devices configured to hold instructions executable by the logic machine to implement the methods and processes described herein. When such methods and processes are implemented, the state of storage machine 1704 may be transformed—e.g., to hold different data.
Storage machine 1704 may include removable and/or built-in devices. Storage machine 1704 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.), semiconductor memory (e.g., RAM, EPROM, EEPROM, etc.), and/or magnetic memory (e.g., hard-disk drive, floppy-disk drive, tape drive, MRAM, etc.), among others. Storage machine 1704 may include volatile, nonvolatile, dynamic, static, read/write, read-only, random-access, sequential-access, location-addressable, file-addressable, and/or content-addressable devices.
It will be appreciated that storage machine 1704 includes one or more physical devices. However, aspects of the instructions described herein alternatively may be propagated by a communication medium (e.g., an electromagnetic signal, an optical signal, etc.) that is not held by a physical device for a finite duration.
Aspects of logic machine 1702 and storage machine 1704 may be integrated together into one or more hardware-logic components. Such hardware-logic components may include field-programmable gate arrays (FPGAs), program- and application-specific integrated circuits (PASIC/ASICs), program- and application-specific standard products (PSSP/ASSPs), system-on-a-chip (SOC), and complex programmable logic devices (CPLDs), for example.
The terms “module,” “program,” and “engine” may be used to describe an aspect of computing system 1700 implemented to perform a particular function. In some cases, a module, program, or engine may be instantiated via logic machine 1702 executing machine-readable instructions held by storage machine 1704. It will be understood that different modules, programs, and/or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and/or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms “module,” “program,” and “engine” may encompass individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc.
It will be appreciated that a “service”, as used herein, is an application program executable across multiple user sessions. A service may be available to one or more system components, programs, and/or other services. In some implementations, a service may run on one or more server-computing devices.
When included, display subsystem 1706 may be used to present a visual representation of data held by storage machine 1704. This visual representation may take the form of a graphical user interface (GUI). As the herein described methods and processes change the data held by the storage machine, and thus transform the state of the storage machine, the state of display subsystem 1706 may likewise be transformed to visually represent changes in the underlying data. Display subsystem 1706 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with logic machine 1702 and/or storage machine 1704 in a shared enclosure, or such display devices may be peripheral display devices.
When included, input subsystem 1708 may comprise or interface with one or more user-input devices such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input subsystem may comprise or interface with selected natural user input (NUI) componentry. Such componentry may be integrated or peripheral, and the transduction and/or processing of input actions may be handled on- or off-board. Example NUI componentry may include a microphone for speech and/or voice recognition; an infrared, color, stereoscopic, and/or depth camera for machine vision and/or gesture recognition; a head tracker, eye tracker, accelerometer, and/or gyroscope for motion detection and/or intent recognition; as well as electric-field sensing componentry for assessing brain activity.
When included, communication subsystem 1710 may be configured to communicatively couple computing system 1700 with one or more other computing devices. Communication subsystem 1710 may include wired and/or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem may be configured for communication via a wireless telephone network, or a wired or wireless local- or wide-area network. In some embodiments, the communication subsystem may allow computing system 1700 to send and/or receive messages to and/or from other devices via a network such as the Internet.
An example provides a method of providing a user interface on a computing device, the method comprising presenting a virtual keyboard on a display of the computing device, detecting input to the virtual keyboard, and, for each detected input, determining whether the input selects any of one or more delimiter keys, displaying a placeholder for the input responsive to the input not selecting any of the one or more delimiter keys, and receiving suggested candidate text from a word-level recognizer and replacing all currently displayed placeholders with the suggested candidate text responsive to the input selecting any of the one or more delimiter keys. In such an example, each detected input may additionally or alternatively be sent to the word-level recognizer responsive to detecting the input to the virtual keyboard. In such an example, the computing device may additionally or alternatively be a first computing device, and the word-level recognizer may additionally or alternatively be included in a second computing device, remote from the first computing device and communicating with the first computing device via a wireless communication link. In such an example, displaying the placeholder for the input may additionally or alternatively comprise incrementing a number of placeholders displayed if a maximum number of placeholders are not displayed and maintaining the number of placeholders displayed if the maximum number of placeholders are displayed. In such an example, detecting input to the virtual keyboard may additionally or alternatively comprise detecting tap inputs to the virtual keyboard, and the method may additionally or alternatively comprise detecting a slide input and transitioning from displaying the virtual keyboard to displaying a different portion of the user interface responsive to the slide input. In such an example, displaying the different portion of the user interface may additionally or alternatively comprise displaying a keyboard selection menu. In such an example, displaying the different portion of the user interface may additionally or alternatively comprise displaying a text editor, the text editor presenting a list of previously-received suggested candidate text. In such an example, transitioning from displaying the virtual keyboard to displaying the different portion of the user interface may additionally or alternatively comprise shifting a portion of the virtual keyboard out of a displayable region of the user interface and shifting the different portion of the user interface into the displayable region of the user interface. In such an example, replacing all currently displayed placeholders with the suggested candidate text may additionally or alternatively comprise showing only a portion of the candidate text on a displayable region of the user interface if a length of the candidate text is greater than a threshold. Any or all of the above-described examples may be combined in any suitable manner in various implementations.
Another example provides a computing device comprising a touch display device having a top edge and a bottom edge, a logic machine, and a storage machine. In such an example, the storage machine may hold instructions executable by the logic machine to present a virtual keyboard on the touch display without a previously-entered text-editing area between the virtual keyboard and the top edge and without a text-editing area between the virtual keyboard and the bottom edge, generate recognized words based on input to the virtual keyboard, and, responsive to a show-text-editor input, transition from displaying the virtual keyboard to displaying a text editor without the virtual keyboard, the text editor presenting the recognized words. In such an example, the transition may additionally or alternatively comprise a panning of the user interface, and a portion of the virtual keyboard may additionally or alternatively be displayed alongside a portion of the text editor in a displayable region of the user interface during the transition. In such an example, presenting the virtual keyboard may additionally or alternatively comprise presenting a top row of keys of the virtual keyboard in a top third of the touch display device, presenting a middle row of keys of the virtual keyboard in a middle third of the touch display device, and presenting a bottom row of keys of the virtual keyboard in a bottom third of the touch display device. In such an example, generating recognized words based on input to the virtual keyboard may additionally or alternatively comprise detecting input to the virtual keyboard, and, for each detected input, determining whether the input selects any of one or more delimiter keys, displaying a placeholder for the input responsive to the input not selecting any of the one or more delimiter keys, and receiving suggested candidate text from a word-level recognizer and replacing all currently displayed placeholders with the suggested candidate text responsive to the input selecting any of the one or more delimiter keys. In such an example, each detected input may additionally or alternatively be sent to the word-level recognizer responsive to detecting the input to the virtual keyboard. In such an example, the word-level recognizer may additionally or alternatively be included in a remote computing device, and the remote computing device may additionally or alternatively communicate with the computing device via a wireless communication link. In such an example, displaying the placeholder for the input may additionally or alternatively comprise incrementing a number of placeholders displayed if a maximum number of placeholders are not displayed and maintaining the number of placeholders displayed if the maximum number of placeholders are displayed. In such an example, the show-text-editor input may additionally or alternatively comprise a slide input. In such an example, the computing device may additionally or alternatively comprise a microphone, and recognized words may additionally or alternatively be generated responsive to voice input detected by the microphone. Any or all of the above-described examples may be combined in any suitable manner in various implementations.
Another example provides a wrist-wearable computing device comprising a touch display device having a top edge and a bottom edge, a logic machine, and a storage machine. The storage machine may hold instructions executable by the logic machine to present a virtual keyboard on the touch display without a previously-entered text-editing area between the virtual keyboard and the bottom edge, detect input to the virtual keyboard, and, for each detected input, determine whether the input selects any of one or more delimiter keys, display a placeholder for the input responsive to the input not selecting any of the one or more delimiter keys, and receive suggested candidate text from a word-level recognizer and replacing all currently displayed placeholders with the suggested candidate text responsive to the input selecting any of the one or more delimiter keys. The storage machine may additionally or alternatively hold instructions executable by the logic machine to, responsive to a show-text-editor input, transition from displaying the virtual keyboard to displaying a text editor without the virtual keyboard, the text editor presenting previously-entered text formed from previously-received suggested candidate text. In such an example, each detected input may additionally or alternatively be sent to the word-level recognizer responsive to detecting the input to the virtual keyboard. Any or all of the above-described examples may be combined in any suitable manner in various implementations.
It will be understood that the configurations and/or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be considered in a limiting sense, because numerous variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, various acts illustrated and/or described may be performed in the sequence illustrated and/or described, in other sequences, in parallel, or omitted. Likewise, the order of the above-described processes may be changed.
The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations, and other features, functions, acts, and/or properties disclosed herein, as well as any and all equivalents thereof.
This application claims priority to U.S. Provisional Patent Application No. 62/046,883, filed Sep. 5, 2014, and entitled “DISPLAY-EFFICIENT TEXT ENTRY AND EDITING”, the entirety of which is hereby incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5546538 | Cobbley | Aug 1996 | A |
| 6073146 | Chen | Jun 2000 | A |
| 8866748 | Sirpal | Oct 2014 | B2 |
| 9047268 | Ouyang | Jun 2015 | B2 |
| 9165257 | Badger | Oct 2015 | B2 |
| 9420089 | Rao | Aug 2016 | B1 |
| 9665276 | Bi | May 2017 | B2 |
| 20020072918 | White | Jun 2002 | A1 |
| 20080077393 | Gao | Mar 2008 | A1 |
| 20080220718 | Sakamoto | Sep 2008 | A1 |
| 20090276839 | Peneder | Nov 2009 | A1 |
| 20110098583 | Pandia et al. | Apr 2011 | A1 |
| 20110302532 | Missig | Dec 2011 | A1 |
| 20120131501 | Lazaridis | May 2012 | A1 |
| 20120249596 | Colley | Oct 2012 | A1 |
| 20120328187 | Vuleta | Dec 2012 | A1 |
| 20130002559 | Vardi | Jan 2013 | A1 |
| 20130045037 | Schaffer | Feb 2013 | A1 |
| 20130120267 | Pasquero | May 2013 | A1 |
| 20130169540 | Dinh | Jul 2013 | A1 |
| 20130205244 | Decker | Aug 2013 | A1 |
| 20130212515 | Eleftheriou | Aug 2013 | A1 |
| 20130263039 | Fahlgren | Oct 2013 | A1 |
| 20130298071 | Wine | Nov 2013 | A1 |
| 20130326392 | Maltesson | Dec 2013 | A1 |
| 20130346168 | Zhou et al. | Dec 2013 | A1 |
| 20140019253 | Ricasata | Jan 2014 | A1 |
| 20140053098 | Leyon | Feb 2014 | A1 |
| 20140055343 | Kim | Feb 2014 | A1 |
| 20140108925 | Zhai | Apr 2014 | A1 |
| 20140109024 | Miyazaki | Apr 2014 | A1 |
| 20140129974 | Ben-Harrush | May 2014 | A1 |
| 20140139637 | Mistry et al. | May 2014 | A1 |
| 20140157191 | Shiplacoff | Jun 2014 | A1 |
| 20140160055 | Margolis et al. | Jun 2014 | A1 |
| 20140173494 | Goldsmith | Jun 2014 | A1 |
| 20140241540 | Hodges et al. | Aug 2014 | A1 |
| 20140259154 | Finlow-Bates | Sep 2014 | A1 |
| 20150065893 | Ye | Mar 2015 | A1 |
| 20150154412 | Nakaizawa | Jun 2015 | A1 |
| 20150169975 | Kienzle | Jun 2015 | A1 |
| 20150294098 | Suwa | Oct 2015 | A1 |
| 20150323998 | Kudekar | Nov 2015 | A1 |
| 20150378599 | Lee | Dec 2015 | A1 |
| 20160011766 | Kosaka | Jan 2016 | A1 |
| 20160042165 | Nicholson | Feb 2016 | A1 |
| 20160156615 | Arai | Jun 2016 | A1 |
| 20160266891 | Phipps | Sep 2016 | A1 |
| 20160378336 | Katsuyama | Dec 2016 | A1 |
| 20170038967 | Sotnikov | Feb 2017 | A1 |
| 20170315719 | Couelier | Nov 2017 | A1 |
| 20180129897 | Hinckley | May 2018 | A1 |
| Number | Date | Country |
|---|---|---|
| 101393506 | Mar 2009 | CN |
| 103576792 | Feb 2014 | CN |
| 2011113057 | Sep 2011 | WO |
| Entry |
|---|
| Clark et al., What Do We Want from a Wearable User Interface, Citeseer 200, pp. 1-6. |
| Medjahed et al., Human Activities of Daily Living Recognition Using Fuzzy Logic for Elderly Home Monitoring, IEEE 2009, pp. 2001-2006. |
| Weir et al., Uncertain Text Entry on Mobile Devices, ACM 2014, pp. 2307-2316. |
| Xiao et al., Expanding the Input Expressivity of Smartwatches with Mechanical Pan, Twist, Tilt and Click, ACM 2014, pp. 193-196. |
| Ahmad et al., A Keystroke and Pointer Control Input Interface for Wearable Computers, IEEE 2006, pp. 1-10. |
| Fuccella et al., Gestures and Widgets: Performance in Text Editing on Multi-Touch Capable Mobile Devices, ACM 2013, pp. 2785-2794. |
| Funk et al., Using a Touch-Sensitive Wristband for Text Entry on Smart Watches, Google 2014, pp. 2306-2311. |
| Wiedenbeck et al., PassPoints: Design and Longitudinal Evaluation of a Graphical Password System, Science Direct 2005, pp. 102-127. |
| Bunke, Recognition of Cursive Roman Handwriting—Past, Present and Future, IEEE 2003, pp. 1-12. |
| Manmatha et al., Word Spotting: A New Approach to Indexing Handwriting, IEEE 1996, pp. 631-637. |
| Rick, Performance Optimizations of Virtual Keyboards for Stroke-based Text Entry on a Touch-based Tabletop, ACM 2010, pp. 77-86. (Year: 2010). |
| Olberding et al., AugmentedForearm: Exploring the Design Space of a Display-enhanced Forearm, ACM 2013, pp. 9-12. (Year: 2013). |
| Talshum's Channel, “Full Screen Keyboard-Super Keyboard,” Retrieved from https://www.youtube.com/watch?v=Cba_gDNrsGk, Uploaded Jul. 10, 2010, 1 Page. |
| Blindtype, “BlindType on Android,” Retrieved from https://www.youtube.com/watch?v=m6eKm1gUnTE, Uploaded Aug. 7, 2010, 8 Pages. |
| Opinionativereviewer, “Keyboard2 iPhone App Review,” Retrieved from https://www.youtube.com/watch?v=8MRHeUPaw3Q, Uploaded Apr. 2, 2011, 8 Pages. |
| USA European Patent Office, International Search Report and Written Opinion Issued in Application No. PCT/US2015/048445, dated Feb. 24, 2016, WIPO, 21 Pages. |
| “Will we Type with Virtual Keyboards on Smart Watches?”, Published on: May 17, 2013 Available at: http://www.atelier.net/en/trends/articles/will-we-type-virtual-keyboards-smart-watches_419585. |
| Scaramuzzino, Paolo, “Evolution or Revolution?”, Published on: Jun. 24, 2014 Available at: https://www.linkedin.com/today/post/article/20140624101302-74188167-evolution-or-revolution. |
| Smith, Dave, “Android Wear is great, but Google's Smartwatches are Sorely Lacking in the Hardware Department”, Published on: Aug. 9, 2014 Available at: http://www.businessinsider.in/Android-Wear-Is-Great-But-Googles-Smartwatches-Are-Sorely-Lacking-In-The-Hardware-Department/articleshow/39964216.cms. |
| Lomas, et al., “5-Tiles Keyboard Targets Wearables to Hunt the Post-Qwerty Holy Grail”, Published on: May 24, 2014 Available at: http://techcrunch.com/2014/05/24/5-tiles-keyboard/. |
| Shah, Agam, “Cloud-savvy Bluetooth 4.1 to reach devices by year end”, Published on: Apr. 25, 2014 Available at: http://www.computerworld.com/article/2488702/wireless-networking/cloud-savvy-bluetooth-4-1-to-reach-devices-by-year-end.html. |
| Smith, Mat, “Minuum's Android Wear Keyboard Makes Smartwatch Typing a Fidgety Reality”, Published on: Jul. 16, 2014 Available at: http://www.engadget.com/2014/07/16/minuum-android-wear-keyboard/. |
| Brian, Matt, “Fleksy Brings a Predictive Messaging App to the Gear 2 Smartwatch”, Published on: Jun. 18, 2014 Available at: http://www.engadget.com/2014/06/18/fleksy-messenger-app-gear-2-tizen/. |
| Wollman, Dana, “Samsung Gear S preview: What's it like to type emails on a 2-inch screen?” Published on: Sep. 3, 2014 Available at: http://www.engadget.com/2014/09/03/samsung-gear-s-hands-on/. |
| Fingas, Jon, “TomTom's new GPS watches track your heart rate without a chest strap (update: US pricing)”, http://www.engadget.com/2014/04/03/tomtom-cardio-gps-watches/, Apr. 3, 2014, 10 pages. |
| Goode, Lauren, “Samsung's New Gear Fit Needs to Work on the “Fit” Part”, http://recode.net/2014/04/08/samsungs-new-gear-fit-needs-to-work-on-the-fit-part/, Apr. 8, 2014, 10 pages. |
| “Samsung Gear Fit, Gear 2 and Gear 2 Neo go on sale worldwide”, NDTV Gadgets, http://gadgets.ndtv.com/others/news/samsung-gear-fit-gear-2-and-gear-2-neo-go-on-sale-worldwide-507220, Apr. 11, 2014, 3 pages. |
| Poeter, Damon, “Meet Simband, Samsung's Next-Gen Health Tracker”, http://www.pcmag.com/article2/0,2817,2458663,00.asp, May 28, 2014, 5 pages. |
| “First Office Action and Search Report Issued in Chinese Patent Application No. 201580047733.3”, dated Jul. 18, 2018, 21 Pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20160070441 A1 | Mar 2016 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62046883 | Sep 2014 | US |
