Patent Application
20120173222
This specification relates generally to text input, and more specifically, to facilitating text input for a user not conversant with the input language.
For some Latin languages, such as English, words are composed of alphabetic letters that are, when typing on a computer, keyed in one letter each time. There may be some errors in the text string input by a user, such as misspellings of a word, or grammatical mistakes, such as wrong tense, or other error. For an use inputting English text and whose native language is not English, for example, it may be necessary for the user to find additional resources, such as an online dictionary or a translation service, to determine a definition of a word or meaning of a sentence in the user's native language to ensure that the proper words or phrases are used when typing in English. In some cases, the service used may provide some text input assistance features, such as a spell checker, a thesaurus for finding synonyms or antonyms, prediction of the most likely form (e.g., spelling), etc., for a particular string.
In the above two examples, however, the input assistance features are either limited to a specific application, such as the text editor in
On the other hand, for some non-Latin languages, such as the East Asian languages of Chinese, Korean or Japanese, it is not possible for a user to use a common keyboard to type in any characters in such languages directly since no characters or symbols of these languages are found on the keyboard. Various input methods are used to facilitate the input of text in these languages. This is achieved by implementing a so-called Input Method Editor (IME) in the operating system and invoking it in each application that may require text input from a user in an application. More than one IME can be installed on one computer for inputting text in different languages or the same language, and each can be used by a user according to his preference. An IME Framework (IMF) is the managing platform in the operating system that manages the different IME. The IMF exists in various modern operating systems, such as Mircrosoft Windows (IMM32 and Text Service Framework), Linux (scim and bus), Mac OSX and Android (IMF). Switching between the different IMEs in the operating system is managed by the IMF. However, aside from the function of providing indirect input of characters, most input methods provided for such Asian languages do not have many additional functions in assisting the user in his text input experience.
For a user who may often require input assistance when typing or writing in a non-native language, such as a user whose native language is Chinese and is typing in English in an input field of a particular application, it will be helpful if the user can resort to just one service for advanced runtime input assistance features without invoking multiple other services, regardless of what type of application is being used or interacted with.
The systems and methods described herein aim to solve the above problem by providing an input assistant, implemented as IME, for invocation from any applications in an operating system environment and that offers advanced input assistance features so that a user may resort to this single service for multiple input assistance services.
In general, one innovative aspect of the subject matter described in this specification relates to a method for facilitating text input. The method includes the actions of invoking an input assistant from within an application in an operating environment at a client, the input assistant being a standalone input service within the same operating environment as the application and operable to be invoked from within a plurality of applications; receiving a text string from a user in an input field of the application; selecting an input text of at least a portion of one word in the input field of the application; selecting one or more text assistance services from at least two or more different text assistance services; receiving, by the input assistant, text content related to the input text from the one or more selected text assistance services in communication with the input assistant; and resenting the received text content at the client.
In another aspect, providing the input prediction may be performed after a predetermined time of inaction by the user since the input of the last character, and the input prediction may comprise a list of words or expressions ranked according to their relevance to the input string.
In yet another aspect, the items of the received text content may be presented in a menu within the application according to the respective ranking of relevance of each item to the selected input text, and when the user confirms a selection of an item in the menu, e.g., by pressing the space bar or the “return” key, the selected item may be committed to the input field to substitute the selected input text.
In one aspect, the text assistance services in communication with the input assistant may at least include: a dictionary service that provides meanings in a first language for a word or phrase in a second language; a spell checking service that detects erroneous spellings of a word in a language and provides its correct spelling in the same language; a translation service that provides text translation between at least two languages; a thesaurus service that provides synonyms and antonyms for a particular word in a language; an input history service that provides past expressions input by a particular user that are related to a target expression a grammar checker that checks the grammar of one or more input sentences; a template filler that provides predefined document types for including body text of a particular document type; or the services may even be provided by a grader that can be used to determine the overall writing quality of an input text. In a further aspect, the text string input by the user may be in English; the bilingual dictionary service may be an English-Chinese dictionary service configured to provide definitions in Chinese for a word or phrase in English; the spell checking service may be configured to detect erroneous spellings of an English word and provide its correct spelling; the translation service may be configured to translate between English and Chinese; the thesaurus service may be configured to return synonyms and antonyms of an English word.
In yet another aspect, at least one of the text assistance services is provided by a server that is remote from the client.
Other embodiments of each of these aspects may include corresponding systems, apparatus, and computer programs recorded on computer storage devices, each configured to perform the actions of methods.
The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects and advantages will be apparent from the description and drawings, and from the claims.
Like reference symbols in the various drawings indicate like elements.
These applications, regardless of whether they have their own text input assistance capability, are configured to, under the management of the operating system, invoke an external Input Method Editors (IME) or an input assistant residing under the same operating system environment. The invocation can be caused by, for example, the pressing a specific combination of keys, such as “ctrl+space”, when the focus is in the input field of the application, such as when the cursor is flickering in the input field. The invoked IME or external input assistant can provide a set of uniform input assistant features for any invoking applications that may cover a much broader range of functionalities than those provided by the input assistant features internal to the invoking applications themselves.
In the system of
The service communication interface 226 is configured to communicate with various input assistance services that may be implemented in the same computer as the input assistant 220 and the invoking applications 210-214, or may be implemented in a remote server. The interface 226 dispatches the parsed string received from the input field of the application to the appropriate services and waits for the results from these services. Once the results are received, the interface 226 sends these results to the presentation handler 224 for presentation processing and display on the client device in the invoking application.
A plurality of different text assistance services 232-238 may be available to the input assistant 220. Based on the nature of processing complexity, some or even all of the input assistance services 232-238 may reside in the same operating system environment as the input assistant 220, and may be, in some implementations, integrated with the input assistant 220 as one product for easy deployment and management under the operating system. Some other input assistance services may reside remote from the computer in which the invoking application and the input assistant 220 are running. The input assistant 220 can, through the service communication interface 226, communicate with such services via the Internet or other dedicated networks according to standard protocols, such as a call to a web service via Simple Object Access Protocol (SOAP). To ensure privacy, such communications may be implemented using privacy protection techniques and/or encryption protocols.
In some implementations, after the input field the user is currently working on has become the object of focus, e.g., when the cursor is in the input field and flickering, showing that the input field is ready to receive input text, the user can proceed to invoke the input assistant 220 by keying in a combination of keys, such as “Ctrl+space” or “Ctrl+Shift+space”, etc. The exact combination of the keys for invoking the input assistant 220 can be custom configured in the operating system environment. For example, the input assistant 220 is a service residing in the same operating system environment on the same computer as the application, and both the input assistant 220 and the application are being managed by the same operating system on this computer.
Invoking the external input assistant 220 need not automatically suppress any internal input assistance features in the application due to their different triggering settings. For example, in a particular word processing program, checking on spelling and grammar can be automatically done on the entire text already input in the input field by a background process, and finding synonyms or antonyms can be done by selecting the target word and pressing “Shift+F7”; with the input assistant 220 invoked, as long as the mechanism for calling up the input assistant 220 to process a particular input string is different from the above triggering mechanism, the user can always have the choice of calling up either the internal or external input assistance features to improve his text input experience.
After the input assistant 220 has been invoked in the application and the input field receives text input from the user, input assistance will be available to the user from the external input assistant 220. For example, the user may input the following string into the input field, “Wikipedia is a free, multili” (step 302). Thereafter, the user may stop at this point to think about how the word the user intends to input is spelled.
In some implementations, the input assistant 220 can be automatically triggered if it determines that there is no activity from the user since the input of the last character for a pre-determined period of time, such as two seconds. In some implementations, the input assistant 220 will be triggered only if the pre-determined period of time has expired and the text already input is at least of a certain threshold length, for example, four characters. Upon triggering, the input assistant 220 will read the string of text from where the cursor is located back to the next space. In the above example, the string “multili” will be read as the input string and sent to the input assistant 220 for further processing.
Alternatively, the user may have moved the cursor from the end of the string he has input to the middle or right in front of the first letter of the uncompleted word. For example, the cursor may be located between the letter “l” and “t” in the string “muffin”, or right in front of the letter “m”. The user may further use a key combination, such as “Ctrl+‘\’”, to trigger the input assistant 220. In this case, the input assistant 220 can read the consecutive character string before and after the cursor, that is, without any space in between, and similarly take the character string “multili” as the input string to the input assistant 220. Further, if a user has inadvertently typed one or more spaces after the string “multili”, the input assistant 220 can be configured to skip these spaces and read back until the next space, thus still obtaining the string “multili” as the input string to the input assistant 220. Thus, the system can read from the input text the last input word or word fragment, either of which may be spelled correctly or incorrectly.
Once the input string is obtained, the input assistant 220 passes this input string to a dictionary service 232 via the service communication interface 226 to obtain any possible words that start with the string “multili”. In this case, the text parser 222 of the input assistant is not required to involve itself with the parsing on the input string since, for the purpose of providing input suggestion for completing the word starting with this character string, the string “muffin” is passed as it is and the target service is definite (or the target service is determined with a minimum degree of confidence), i.e., a dictionary service for a word or word fragment. Upon receiving the string “multili”, the dictionary service 232, which can be a digital dictionary product incorporated in the input assistant 220 itself or an online dictionary service accessible via the Internet, retrieves from its data store any words that start with the character string “multili”, sorts them according to its own ranking criteria, and exports the results to the presentation handler 224 of the input assistant 220 via the service communication interface 226 for presenting to the user. The presentation handler 224, upon receiving the list of predicted words, prepares a popup menu and displays the popup menu and the words adjacent to the input field of the application (step 303). For some users, the function of auto-completion may not be necessary. In this case, the step 303 as shown in
The list of predicted words can be presented with their basic meanings. In some implementations, the basic meanings are in a different language.
To further assist a user of non-English speaker in understanding the usage of an English word, the dictionary service 232 can further provide more detailed explanations on each meaning of the word and sample sentences in the source language (i.e., the language of the word to be input) for each such usage. In the above case, definitions of a word can be rendered in a separate window by the presentation handler 224. In
Besides the function of predicting the remaining part of a word, i.e., the functionality of auto-completion at the word level, the input assistant 220 can be more versatile in assisting a non-native speaker of the input language in obtaining information related to the input text, or providing input hints from the user's previous input history.
In some implementations, a spell checking service can be offered to check the spelling of the input word or words. In the above case with the auto-completion function, if the input assistant 220 determines that no predicted words are available from the dictionary service 232 for the input string, for example in the case where the input string is “multtpl”, the input assistant 220 may determine that the user may not be requesting the auto-completion service by waiting for the predetermined time, but instead may be requesting a service for spell-checking the input string (or may not be able to use the auto-complete service without the spell-checking service). The input assistant will forward the input string “multtpl” to the spell checking service 234 to obtain possible correct spellings of this word. The spell checking service, similar to the dictionary service 232, can be a service integrated with the input assistant 220 on the local computer or a remote service accessible via the Internet. Each result from the spell checking service 234 is scored and ranked according to its relevance to the input string. Upon receiving these results, the presentation handler 224 presents these results in a similar manner as with the results from the dictionary service 232 for review and selection by the user.
After the user finishes reviewing the results from the dictionary service 232 or the spell checking service 234 and made his choice, he may proceed to typing in additional content after his selection. In this case, the input assistant 220 will wait for the next call from the user, either through the cursor being inactive for another predetermined period of time at a new location, or through a voluntary call to the input assistant 220 from the user by a manual selection of an input string, and then send the text string to the input assistant 220. The step of activating the input assistant 220 for prediction of a complete word or performing spelling checking on the input string simply by waiting for a predetermined time is optional. The user may directly activate the input assistant 220 for input assistance, as described in detail below.
Apart from triggering the input assistant 220 simply by waiting a pre-determined time for auto-completion or spelling checking purposes, the input assistant 220 can be further manually or automatically triggered for more advanced services. In some implementations, the user can manually select a word or string of consecutive characters using the mouse or combination of keys, such as the key “Shift” plus the arrow keys, and trigger the input assistant 220 by pressing another combination of keys, such as “Ctrl+‘\’” (step 304). In an alternative implementation, the selection of one or more words from the input field can be automatically performed by the input assistant 220 when the user triggers the input assistant 220 by pressing “Ctrl+‘\’” without first manually selecting any input text. In this case, the input assistant 220 may check the text surrounding the and select, for example, the previous partial one or several words starting from where the cursor is located as the input text for invoking an input assistance service. In an alternative implementation, if the user has pressed “Ctrl+‘\’” twice within a short period, for example one second, the input assistant 220 may select the entire sentence surrounding the cursor as the input text.
Upon receiving an input string from the input field that was manually selected by the user or automatically selected by the input assistant 220, the text parser 222 of the input assistant 220 will first perform a preliminary parsing process on the input string. In some implementations, the text parser 222 first determines the language of the input string. For example, if the input string is a string of characters “”, the text parser may determine that it is in Chinese. In the case of a Chinese input, as it is very likely that the user who is unfamiliar with English words is requesting for the corresponding English word of this Chinese text, the text parser 222 will forward this Chinese text to the dictionary service 232 for the corresponding word or meaning in English. The dictionary service 232 will, in this case, return the term “orchid” to the input assistant 220 for appropriate presentation to the user by the presentation handler 224.
Conversely, if the input string is “wikipedia is a free, multilingual encyclopedia project”, the text parser 222 will determine that the input text is in English. Further, the text parser 222 will determine that there is more than one word in this input string from the text of the string, by simply identifying at least a space in the input string. In this case, it is assumed that the user does not simply want to check for the meaning of a word or words in the input string, but instead wanting to check for the meaning of a phrase of a sentence.
In some implementations, this input string may be forwarded to a translation service, which may be located remote from the computer where the input assistant 220 resides, and obtain a translation of the input string in Chinese to present to the user for his reference in confirming that the word the user has input is largely correct in English. The text parser 222 can also send this input string to a grammar service that checks the English grammar of the input string, and obtain the results in the form of suggested improvements or simply highlighting the problematic parts in the input string. The above translation and grammar checking processes can be carried out simultaneously (step 305), and the results from both services can be aggregated in one list for presentation to the user (step 306).
Different rules or standards can be set in the text parser 222 for determining which services should be called for a particular input string. The rules or standard can define the selection of services based on detected characteristics of various languages (e.g., spaces, no spaces, number of consecutive characters, etc.). For example, an input string comprising more than 10 non-consecutive spaces should be forwarded to the spelling checking service and translation service only, and an input string comprising less than three nonconsecutive spaces should be forwarded to the dictionary service 232, the spell checking service 234, the input history service 236, etc. Additional functions can be further implemented in the text parser 222 in its preliminary parsing stage to better determine which services would be appropriate to handle the received input string in providing the best possible results or combination of results for improving the user's text input experience. Alternatively, instead of always being determined by the text parser 222, a particular input assistance service may be manually designated and triggered by the user for a particular input string, by simply typing in a specific combination of keys, or invoking a functional GUI window in the input assistant 220 to select the appropriate service for processing the particular input string.
As a further alternative, the text parser 222 does not need to make the determination, but forwards the request to all the input assistance services connected with the input assistant 220. Each input assistance service will determine if the received input text is appropriate for it to provide a result. The determination can be made, for example, by reference to rules or conditions define when text is appropriate for processing by each particular service. For example, if a translation service finds that the received text is in the target language, it will return a null value to the requesting input assistant 220. The presentation handler 224 of the input assistant 220 will then reassemble the results from different services while skipping any null values for presenting to the user.
In some implementations, an input history service 236 can be used to provide user specific input history to assist the user in inputting large portions of content that the user may often use in specific contexts. For example, a patent attorney may use many fixed expressions in responding to office actions. In this case, the user may have a user account with the input assistant 220, in which the user may call up the input assistant 220 to configure the setting according to his preferences and store often used expressions for future usage in the account with the input history service 236. The input history service can be located in a central server, which allows the user to log into the account from different computers with the input assistant 220 and still have the benefits of enjoying his preferred configuration in the input assistant 220 and using the user's own input history in assisting the input experience. Storing custom content in a central service and retrieving them whenever required from anywhere through the use of user accounts are feasible in the field.
In some implementations, results from different services can be rendered in a single popup window for the user to choose from. The presentation handler 224 can sort the results from each service according to any known ranking criteria that at least include a weighted ranking of each result item from each service to thereby reflect the relevance of each result item to the input string. In some implements, the presentation handler joins the results from different services in one display window for presentation. In the above example in
In some implementations, the dictionary service 232 can present alternative results of particular category to the user for a review. These different results can be displayed in response to different user inputs, e.g., different keyboard invocation commands. For example, in
Services that assist the user in his text input experiences are not limited to the above. Other services that provide assistance with other languages, or with additional input assistance or text prediction functionalities, can also be integrated with or connected to the input assistant 220 to provide additional input assistance to the user. For example, a service may provide the function of filling in one or more lost words in an input string, which in whole constitutes a commonly used phrase, such as an input string “a of” may return a prediction of the phrase “a lot of”. Also, the different definitions of the key presses or combinations of key presses in this disclosure are for exemplary purposes only; many different combinations may be defined in different operating systems to achieve the same effects of triggering a service or handling the popup windows.
Embodiments of the subject matter and the functional operations described in this specification may be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions may be encoded on a propagated signal that is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.
The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus may include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus may also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
A computer program (which may also be referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code). A computer program may be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.
The processes and logic flows described in this specification may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).
Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer may be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.
Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.
To provide for interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user may provide input to the computer. Other kinds of devices may be used to provide for interaction with a user as well; for example, feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, speech, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.
While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination may in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.
Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.
Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/CN11/70044 | 1/5/2011 | WO | 00 | 6/30/2011 |
