Patent Application
20120264516
Typing text on small physical or virtual keyboards is difficult and can be frustrating for users. Existing systems use touch models to correlate various contact areas or touch points with particular intended characters and to resize particular keys. For example, the user may consistently press one side of the “a” character rather than the center of the “a” character. The touch model then correlates that portion of the “a” character with an intended “a” press. To perform accurately, the touch models require sufficient input data from the user for training. Existing systems for improving touch models, however, focus on training the user to alter key press locations by advancing through a series of characters only when the user presses the correct character. Further, the existing systems generally fail to identify particular areas of improvement for the user.
Embodiments of the disclosure conduct a game to obtain data to train a touch model associated with a computing device. A key input layout is displayed to a user in a first partition of a touch screen of a computing device. An ordered plurality of characters is displayed in a second partition of the touch screen or otherwise provided to the user. Each of the ordered plurality of characters has a portion of the displayed keyboard associated therewith. A sequence of touch screen inputs is received from the user. Each of the sequence of touch screen inputs is intended by the user to correspond to one of the displayed ordered plurality of characters. A contact area associated with each of the received touch screen inputs is determined. Each determined contact area is compared to the portion of the displayed keyboard associated with the corresponding character in the ordered plurality of characters. A user-specific touch model associated with the computing device and/or the user is defined based on the comparison. In some embodiments, the key input layout is a physical keyboard (e.g., no touch screen) and the ordered plurality of characters is provided via computer-generated speech.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
Corresponding reference characters indicate corresponding parts throughout the drawings.
Referring to the figures, embodiments of the disclosure enable the collection of input data from users 104 to train touch models 116 and to improve text entry for the users 104 on touch screen keyboards. Some embodiments include a downloadable, installable application that executes as a game to present an ordered plurality of characters to the user 104 for entry. Contact areas are determined for each press by the user 104, and the game advances through each of the characters when the user 104 presses the correct character or an adjacent character. By advancing to the next character when an adjacent character is pressed, aspects of the disclosure collect natural touch points from the user 104.
Aspects of the disclosure enable collection of user input and determination of contact areas to improve touch modeling. Embodiments are also applicable to any key input layout including traditional (e.g., QWERTY) keyboards, non-traditional (e.g., non-QWERTY) keyboards, 12-key numeric layouts, reduced soft input panels (SIPs), and any other arrangement and representation of characters. Aspects of the disclosure are operable with any language.
Referring next to
The mobile computing device 102 represents any device executing instructions (e.g., as application programs, operating system functionality, or both) to implement the operations and functionality associated with the mobile computing device 102. The mobile computing device 102 may be any portable device. In some embodiments, the mobile computing device 102 includes a mobile telephone, laptop, tablet, computing pad, netbook, gaming device, and/or portable media player. Alternatively, aspects of the disclosure are operable with less portable devices such as desktop personal computers, kiosks, and tabletop devices. Additionally, the mobile computing device 102 may represent a group of processing units or other computing devices.
In the example of
The mobile computing device 102 further has one or more computer readable media such as the memory area 108. The memory area 108 includes any quantity of media associated with or accessible by the mobile computing device 102. The memory area 108 may be internal to the mobile computing device 102 (as shown in
The memory area 108 stores, among other data, one or more applications 110. The applications 110, when executed by the processor 106, operate to perform functionality on the mobile computing device 102. Exemplary applications 110 include mail application programs, web browsers, calendar application programs, address book application programs, messaging programs, media applications, location-based services, search programs, and the like. The applications 110 may communicate with counterpart applications or services such as web services accessible via a network 128. For example, the applications 110 may represent downloaded client-side applications that correspond to server-side services executing in a cloud.
Aspects of the disclosure may be embodied as one or more of the applications 110. For example, embodiments of the disclosure include a mobile application downloadable from an application marketplace. Upon download, the user 104 executes the mobile application to at least perform the functionality illustrated in
The memory area 108 further stores one or more user-specific touch models 116. In some embodiments, the user-specific touch models 116 are associated with the user 104 and with the mobile computing device 102. The touch models 116 may represent a probability distribution over likely touch points mapped to keys or characters given an intention to press a particular character.
The memory area 108 further stores one or more computer-executable components. Exemplary components include a touch screen component 118, a game component 120, a pattern component 122, and a performance component 124. Operation of the computer-executable components is described below with reference to
The touch screen 107 of the mobile computing device 102 receives input from, and provides output to, the user 104. The touch screen 107 may represent a small portion of the mobile computing device 102, or may consume a significant portion of one or more sides of the mobile computing device 102. The touch screen 107 may be any touch screen having capacitive and/or resistive touch sensors.
In some embodiments, the mobile computing device 102 communicates with a cloud service 126 via the network 128 such as the Internet. For example, the mobile computing device 102 sends data collected during execution of the game to the cloud service 126 for analysis. In another example, the mobile computing device 102 receives touch models 116 and other data from the cloud service 126, such as touch models 116 based on crowd-sourced data that are pushed to a plurality of the mobile computing devices 102 (e.g., as an update to existing touch models 116).
Referring next to
In some embodiments, the ordered plurality of characters comprises one or more of the following: a word, a phrase, a sentence, symbols, graphic elements, abbreviations, and emoticons (e.g., a smiley face). The characters may be numeric and/or alphabetic in some embodiments. In general, the ordered plurality of characters includes any symbols assigned to the keys of the keyboard. Further, the ordered plurality of characters may be provided to the user 104 by means other than a display. For example, a text-to-speech generator may speak or otherwise convey the ordered plurality of characters to the user by sound.
The mobile computing device 102 receives a sequence of touch screen inputs from the user 104. Each of the touch screen inputs is intended by the user 104 to correspond to one of the displayed, ordered plurality of characters. For example, the user 104 types in a displayed word, phrase, or sentence. For each received touch screen input, the mobile computing device 102 processes the touch screen input.
In some embodiments, the mobile computing device 102 provides entry characteristics to the user 104 for providing the sequence of touch screen inputs. The entry characteristics describe how the user 104 should hold or use the mobile computing device 102 when entering data to enable aspects of the disclosure to learn contextual touch models. Exemplary entry characteristics include, but are not limited to, one or more of the following: one finger input, two finger input, input while walking, input while sitting, and input while standing.
As an example, if the mobile computing device 102 receives a touch screen input from the user 104 at 204 (e.g., receives a finger press), the mobile computing device 102 determines the contact area associated with the received touch screen input at 206. For example, the mobile computing device 102 determines the contact area by determining one or more of the following: an X-Y coordinate, a shape of a contact surface, a pressure reading, a key down event corresponding to a finger press, and a key up event corresponding to a finger release. However, other contact area calculations are within the scope of the disclosure.
Aspects of the disclosure may also be used for shape writing training (e.g., training a shape writing language model). For example, the user 104 tries to swipe words as quickly as possible while embodiments of the disclosure build up a database of touch information that corresponds to the words.
The mobile computing device 102 compares the determined contact area to the portion of the displayed keyboard associated with the corresponding character in the ordered plurality of characters at 208. In some embodiments, the comparison determines a distance between the intended contact area and the actual determined contact area. In this manner, the mobile computing device 102 is able to calculate an accuracy.
If the determined contact area is at, or adjacent to, the corresponding keyboard portion at 210, and if the current character is not the last character displayed at 212, the mobile computing device 102 advances to the next character at 214. For example, the current character may disappear, get smaller, change colors, animate, unbold, or otherwise shift the focus of the user 104 to the next character in the ordered plurality of characters. Similarly, the mobile computing device 102 visually distinguishes the next character in the ordered plurality of characters to attract the attention and focus of the user 104. For example, the next character may get larger, change colors, animate, bold, or otherwise attract the attention of the user 104. The mobile computing device 102 then waits for another touch screen input from the user 104.
In some embodiments, the mobile computing device 102 determines whether the contact area is at, or adjacent to (e.g., the user 104 hits one of the surrounding keys), the corresponding keyboard portion by comparing a linear distance between the contact area (e.g., a center thereof) and the corresponding keyboard portion (e.g., a center or other point therein) with a threshold distance. If the distance is less than or otherwise satisfies the threshold distance, then the mobile computing device 102 determines that the contact area is at, or adjacent to, the corresponding keyboard portion. If the distance is greater than or otherwise violates the threshold distance, then the mobile computing device 102 determines that the contact area is not close enough to the corresponding keyboard portion.
The threshold distance may be set by one or more of the following: the user 104, a manufacturer of the mobile computing device 102, a touch screen provider, an operating system provider, or other entity. For example, the threshold distance may be set based on the distance between characters on the keyboard. In such an example, if the letter “d” is the current character, the game advances to the next character if the user 104 touches “d” or any character surrounding “d” (e.g., “w,” If the determined contact area is not at, or not adjacent to, the corresponding keyboard portion at 210, the mobile computing device 102 proceeds to wait for another touch screen input. That is, the game does not advance to the next character in the ordered plurality of characters.
If the current character is the last character in the ordered plurality of characters at 212, the user 104 has completed the game at 218. The mobile computing device 102 may apply knowledge of the ordered plurality of characters, touch screen inputs, and determined contact areas to create, define, improve, customize, or otherwise alter the user-specific touch model 116 at 216. For example, the mobile computing device 102 may update the user-specific touch model 116 based on the determined contact areas of the touch screen inputs (e.g., adjust a center of the intended characters based on the corresponding determined contact areas). The touch model 116 is altered based on data of the form <key, touch point> where <key> represents the correct, intended character and <touch point> represents the contact area pressed by the user 104 when attempting to press <key>.
In some embodiments, the mobile computing device 102 may modify a generic touch model 116 associated with the particular mobile computing device 102 model number. In this example, the generic touch model 116 is specific to the particular mobile computing device 102 model and applicable to all users 104 of the mobile computing device 102. After the user 104 plays the game, however, the generic touch model 116 becomes customized to the user 104 as described in the operations illustrated in
Alternatively or in addition, the mobile computing device 102 may calculate and display statistics about the user's performance in the game. For example, the mobile computing device 102 may calculate an accuracy of the received touch screen inputs. The accuracy is determined based on the comparison between the determined contact area and the corresponding keyboard portion associated with the intended or current character. In another example, the mobile computing device 102 may calculate a speed based on receipt times of the touch screen inputs. In some embodiments, the speed calculation may further be based on, or compared to, the calculated accuracy.
In some embodiments, the mobile computing device 102 shares the game completion statistics and other data with the cloud service 126 for analysis. For example, the mobile computing device 102 sends or identifies game completion statistics such as the ordered plurality of characters, the corresponding determined contact areas, the calculated speed, and/or the calculated accuracy to the cloud service 126. Additional data sent may include configuration information such as hardware information (e.g., mobile computing device make/model and touch-controller information), screen size and resolution, color themes (e.g., light vs. dark), contrast, font size, key size, orientation (e.g., portrait vs. landscape), sensor input (e.g., accelerometer readings), and/or whether the user 104 is using one or two fingers, two thumbs, two fingers, etc. The cloud service 126 aggregates the game completion statistics from a plurality of the mobile computing devices 102. The aggregated data represents crowd-sourced data that spans multiple devices and multiple users. The cloud service 126 analyzes the aggregated crowd-sourced data to provide each user 104 with feedback on the typing speed and accuracy compared to other users 104.
The cloud service 126 may further analyze the aggregated crowd-sourced data to determine, for example, an aggregate speed and an aggregate accuracy per make and model of the plurality of mobile computing devices 102. Alternatively or in addition, the cloud service 126 analyzes the aggregated crowd-sourced data to identify problems per make and model of the plurality of mobile computing devices 102. The cloud service 126 may also identify characteristics of the users 104 associated with particular errors. For example, users 104 with large hands may make similar errors. Aspects of the disclosure may then adjust the touch model 116 for users 104 with those characteristics.
Some embodiments of the disclosure operate to optimize, improve, or otherwise adjust the key input layout based on performance of the user 104 during the game. For example, the key input layout represents a first key input layout and the user-specific touch model represents a first touch model corresponding thereto. The first key input layout may be altered or replaced with a second key input layout (e.g., an ergonomic or curved layout), and a second touch model may be learned based on operation of the game operations described herein for the second key input layout. Performance of the first key input layout and the second key input layout may be compared (e.g., compare the speed and accuracy of the user 104 with each key input layout). The key input layout with the better performance (e.g., faster speed and/or higher accuracy) may be selected, or a third key input layout may be created using characteristics of the key input layout having the better performance. In this manner, an optimal key input layout may be defined.
When the user 104 decides to play another round of the game, the mobile computing device 102 accesses the game completion statistics from one or more of the previous rounds to select the next ordered plurality of characters to display. In some embodiments, the cloud service 126, rather than the mobile computing device 102, selects the next ordered plurality of characters to display. For example, the next ordered plurality of characters may be selected based on the calculated accuracy or calculated speed. In one such example, one or more of the characters may be selected based on a determination that the user 104 had a low accuracy or slow speed with these characters. Selecting the ordered plurality of characters in this way enables the user 104 to work on and improve accuracy and speed. In another example, the ordered plurality of characters may be selected based on the determined contact area. In one such example, if the user 104 had contact areas for a particular character that were not consistently located, the mobile computing device 102 selects an ordered plurality of characters that includes the particular character to allow the user 104 to work on consistently hitting the intended area. In still another example, the ordered plurality of characters may be selected based on a desire by the mobile computing device 102 and/or the cloud service 126 to collect data regarding particular characters or patterns of characters.
In some embodiments, the computer-executable components illustrated in
In some embodiments, the touch screen component 118 includes a graphics card for displaying data to the user 104 and receiving data from the user 104. The touch screen component 118 may also include computer-executable instructions (e.g., a driver) for operating the graphics card. Further, the touch screen component 118 may include a display (e.g., the touch screen 107) and/or computer-executable instructions (e.g., a driver) for operating the touch screen 107.
The game component 120, when executed by the processor 106 of the mobile computing device 102, causes the processor 106 to visually alter each of the ordered plurality of characters in sequence when the contact area determined by the impression component overlaps with or is adjacent to the portion of the displayed keyboard associated with the corresponding character in the ordered plurality of characters. For example, if the user 104 presses the intended, expected key or any key adjacent to the intended, expected key, the game advances to the next character. In some embodiments, the game component 120 visually alters each of the ordered plurality of characters by causing each of the ordered plurality of characters to disappear in sequence.
The pattern component 122, when executed by the processor 106 of the mobile computing device 102, causes the processor 106 to overlay, on the displayed keyboard, the contact areas determined by the touch screen component 118 to display a user-specific touch model 116. For example, the contact areas may be displayed as an indication of game performance. The performance component 124, when executed by the processor 106 of the mobile computing device 102, causes the processor 106 to calculate a speed value and an accuracy value as the touch screen component 118 receives the sequence of touch screen inputs from the user 104.
Referring next to
Referring next to
In the example of
Referring next to
In some embodiments, the contact areas are displayed to the user 104 at game completion. Alternatively or in addition, the contact areas may be displayed while the game is in progress.
While the contact areas are illustrated in grayscale in
Referring next to
Referring next to
Referring next to
While aspects of the disclosure are described herein with respect to keyboards displayed on a touch screen of the mobile computing device 102 such as a mobile telephone, aspects of the disclosure are operable with physical keyboards (e.g., no touch screen). In some embodiments, the mobile computing device 102 detects only finger down events and finger up events. In such embodiments, the contact areas include the entire character pressed. The game advances to the next character when the intended character and an adjacent character are pressed. Aspects of the disclosure may be used for training button mashing algorithms.
In other embodiments, the mobile computing device 102 includes a physical keyboard where direction and pressure for each finger press are capable of being detected. In such embodiments, the contact area is defined by the character pressed, the direction of the press, and/or the pressure applied by the user 104.
While some aspects of the disclosure have been described with reference to the cloud service 126, other embodiments are used to customize the touch model for the user 104 without interacting with the cloud service 126. For example, the touch model may be adapted to the actual typing patterns of the user 104 based on the results of the game played by the user 104 on the computing device. Such embodiments improve the experience for the user 104 on the computing device on which the game has been played.
During exemplary game play, the ordered plurality of characters includes a word randomly selected from a corpus of 10,000 words. The corpus is based on email messages and transcribed voicemail messages, in this example, to include representative character sequences encountered in such data. When the user 104 touches the correct key (e.g., the intended key or an adjacent key), the corresponding displayed character explodes out then fades away. When the user 104 touches an incorrect key, the character turns red and a beep is sounded. To move to the next word, the user 104 touches the space bar.
Exemplary data collected during game play is included as Table A-1 in Appendix A.
At least a portion of the functionality of the various elements in
In some embodiments, the operations illustrated in
While no personally identifiable information is tracked by aspects of the disclosure, embodiments have been described with reference to data monitored and/or collected from users 104. In such embodiments, notice is provided to the users 104 of the collection of the data (e.g., via a dialog box or preference setting) and users 104 are given the opportunity to give or deny consent for the monitoring and/or collection. The consent may take the form of opt-in consent or opt-out consent presented, for example, when the user 104 launches the game.
Exemplary computer readable media include flash memory drives, digital versatile discs (DVDs), compact discs (CDs), floppy disks, and tape cassettes. By way of example and not limitation, computer readable media comprise computer readable storage media and communication media. Computer readable storage media store information such as computer readable instructions, data structures, program modules or other data. Computer readable storage media exclude propagated data signals. Communication media typically embody computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and include any information delivery media.
Although described in connection with an exemplary computing system environment, embodiments of the invention are operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with aspects of the invention include, but are not limited to, mobile computing devices, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, gaming consoles, microprocessor-based systems, set top boxes, programmable consumer electronics, mobile telephones, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
Embodiments of the invention may be described in the general context of computer-executable instructions, such as program modules, executed by one or more computers or other devices. The computer-executable instructions may be organized into one or more computer-executable components or modules. Generally, program modules include, but are not limited to, routines, programs, objects, components, and data structures that perform particular tasks or implement particular abstract data types. Aspects of the invention may be implemented with any number and organization of such components or modules. For example, aspects of the invention are not limited to the specific computer-executable instructions or the specific components or modules illustrated in the figures and described herein. Other embodiments of the invention may include different computer-executable instructions or components having more or less functionality than illustrated and described herein.
Aspects of the invention transform a general-purpose computer into a special-purpose computing device when configured to execute the instructions described herein.
The embodiments illustrated and described herein as well as embodiments not specifically described herein but within the scope of aspects of the invention constitute exemplary means for conducting a game to obtain data to train the touch model 116 associated with the mobile computing device 102, and exemplary means for identifying areas for improvement based on the comparison between the determined contact areas and the portion of the displayed keyboard associated with the corresponding character in the ordered plurality of characters.
The order of execution or performance of the operations in embodiments of the invention illustrated and described herein is not essential, unless otherwise specified. That is, the operations may be performed in any order, unless otherwise specified, and embodiments of the invention may include additional or fewer operations than those disclosed herein. For example, it is contemplated that executing or performing a particular operation before, contemporaneously with, or after another operation is within the scope of aspects of the invention.
When introducing elements of aspects of the invention or the embodiments thereof, the articles “a,” “an,” “the,” and “said” are intended to mean that there are one or more of the elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements.
Having described aspects of the invention in detail, it will be apparent that modifications and variations are possible without departing from the scope of aspects of the invention as defined in the appended claims. As various changes could be made in the above constructions, products, and methods without departing from the scope of aspects of the invention, it is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense.
Exemplary data collected during game play is illustrated in Table A-1 below. In Table A-1, the fields at the top left identify the Username, Device Make, Device Model, Operating System Version, Firmware Version, Landscape/Portrait Mode, and Game Version. The subsequent data points take the following form:
<KeyUp/KeyDown, intended character, x, y, timestamp>
The intended character represents the target key, while the x, y values indicate a location of the touch point.
