Patent Application
20140081620
1. Field
Embodiments of the invention generally relate to the field of automated translation using electronic dictionaries and various applications in such areas as machine translation.
2. Related Art
There is a lot of electronic devices with display screens capable of displaying text, including devices with a touch screen, for example many mobile devices, such as smartphones, mobile phones, laptops, tablet computers, netbooks, personal digital assistants (PDAs), e-book readers, photo and video cameras.
These devices have enough resources for using electronic dictionaries, which may be installed locally. Machine translation system may be available over the Internet, WiFi, Bluetooth, and through these and other types of networks and protocols.
Many of the aforesaid devices involve advanced displays and associated logic capable of supporting non-keyboard type input methods. For example the devices may support pen-based input, voice-based input or touch-based input.
Also, most of above-identified electronic devices may involve the use of a camera and other components to facilitate receiving, capturing and using photographs and images with text.
In one embodiment, the invention provides a method including receiving an image with text (photos, PDF, Word files, pictures, etc.) with text to be translated on the screen of electronic device, automated translation of the text, displaying the result of translation on the screen of electronic device, providing a user an opportunity to swipe or refresh screens and compare the original text with the translated one. Some or all of the screens of the electronic device is swapped or refreshed upon the device receiving a swiping action.
Electronic dictionaries may comprise a software program, software logic which operate with dictionary data. The software program may include a shell, which provides a graphical user interface, morphology models to display inflected forms, a context search that uses an index, a teaching module, etc. The dictionaries that the user needs to use may be independently stored in different locations including the computer device or a server in a local area network (LAN) or a wide area network (WAN) such as the Internet.
An electronic device may also connect to a machine translation system. The machine translation system may be located on a remote server, access to which is available via a network connection or protocol to the server, e.g. over the Internet.
In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the invention. It will be apparent, however, to one skilled in the art that the invention can be practiced without these specific details. In other instances, structures and devices are shown only in block diagram form in order to avoid obscuring the invention.
Reference in this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. The instances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Moreover, various features are described which may be exhibited by some embodiments and not by others. Similarly, various requirements are described which may be requirements for some embodiments but not other embodiments.
Advantageously, the present invention discloses an electronic device that allows a user to perform machine or dictionary translation of an image or photograph containing text, display the translated variant of the image on another screen of an electronic or mobile device, and to switch between screens by performing a swiping gesture. The terms “screen” and “another screen” imply different views or representations on a physical screen or parts of a same physical screen and refer to a representation of an original image and a translated one. The electronic device preferably includes or accesses additional optical character recognition (OCR) software, firmware, algorithms or functionality to support translation of text. Such OCR software can identify the regions on the image where the text is located and then converts the image in the regions into a text format. OCR software may be installed at any remote server and run via the Internet, Bluetooth, etc. so that an OCR functionality need not be limited to steps or algorithms performed on the device that displays, captures or accesses an image of text.
The method(s) allow a user to correct results of an OCR function, an OCR result, and to re-initiate a translation of the original text, or portions of the original text such as based upon a correction made by a user. For example, a user may change a word recognized as “mike” to “make” to enable a correct translation of the word in the image into a second or target language.
Additionally, the disclosed methods also allow a user to compare the result of a machine and/or dictionary translation of the text (in a second or target language) with the text in a first or original (source) language. The methods may include the alternating use of a forward and backward swiping motion or action to a touch sensitive screen or in relation to a touch sensitive screen. The forward and backward swiping action may be, for example, respectively, a left swiping motion and a right swiping motion, or an upward and downward swiping motion, or by a counter-clockwise and clockwise rotation of a device etc.
The methods also allow a user to look through alternatives of translation and to optionally and manually make word-by-word correction of the text by insertion or replacement of a first translation with another variant of translation. The opportunity to manually correct the translated text (in the target language) is useful in the case when a user is not satisfied with the translation proposed by system translation or when a word is not translated due to text that was not accurately recognized.
A manual correction of the target language comprises, for example, touching a user interface element associated with a certain word or word combination in the translated text in an output language, displaying a list of alternative variants of translation for the source word or word combination (e.g., in a balloon or in a pop-up or overlay window), allowing a user to look through, find or enter other possible variants of translation of the word or word combination, receiving an input from a user that indicates a variant of translation, and performing an insertion or substitution of an appropriate variant of translation in the text instead of the variant originally or first proposed by the translation system.
In alternative embodiment, a subsequent translation from a target language back to a first (source) language may be made to allow a user to verify that the translation to the target language carries the correct meaning or sense. Such subsequent translation may be made by using a distinctive and different motion or swiping action in relation to the touch-sensitive screen. Only a portion of the translated text may be re-translated to a source language. Such portion may be indicated by a different swiping action (e.g., two-finger action, three-finger action), or by selecting a portion of text, and then subsequently making a swiping motion in a backward orientation or motion.
A system user can preliminarily select or “pre-set” a mode of translation (dictionary translation mode, machine translation mode, etc.), for example depending on the kind of text or accessibility of an Internet or cellular telephone connection. In one exemplary embodiment, two modes of translation are available: a dictionary translation and a machine translation. Dictionary translation performs a word-by-word or phrase-by-phrase translation based on translation of words and word combinations available in one or more dictionaries. A machine translation performs a sentence-by-sentence or phrase-by-phrase translation of the source text. In other embodiments, other modes of translation may be available including various services of machine and automated translation.
Word-by-word dictionary translation may be selected in a case when text is represented as a string of grammatically unconnected words, for example a menu or a set of ingredients in a food recipe. The choice of dictionary translation allows one to save memory and resources of an electronic or mobile device. An example of a text image translated under a dictionary mode is shown in
In alternative embodiment, a pulling of a portion of the visible language down (off the screen) and then raising it back upward (without lifting the finger from the surface of the device display) translates only those lines or portions of text that were temporarily moved off the screen. Thereby a portion of the text in the image displayed or rendered (by being in a state of encoded text) to be translated may be selected. For example, if three lines of text from a menu were pulled off the screen, only those three lines are translated. Similarly, in another exemplary embodiment, when two top lines are swiped upward and off the screen, these two lines are translated when the screen is brought up and down through a combination of a swiping motion upward followed directly by a swiping motion downward.
In yet another alternative implementation, with reference to
In another alternative embodiment, the translation of the source text into two target languages may be performed—either sequentially or at the same time, depending on a configuration setting. In this scenario, in the system, two target languages should be preliminary specified. A first swiping gesture in a forward direction initiates a translation into the first target language, and following after the first swiping gesture, a second swiping gesture in a forward direction initiates a second translation into the second target language. So, if user speaks or is familiar with two or more foreign languages, for example Spanish and German languages, a user can translate the source text into two languages based on this procedure. The desired target languages should be specified in the settings of the system or software application. In the example, the first swiping of the screen in a forward direction yields the result of the first translation into Spanish, and the second swiping in the forward direction shows the result of translation of the original (source) text into German. The swiping action makes translation between languages much easier than previously available. Instead of a user searching in a menu for a setting for a target language, a swiping motion allows a user to translate a text between two or more languages with a single gesture. Also, the opportunity to go back a previous screen by using a swiping gesture in a backward direction remains for a user. This process is available for translation of the source text into more than two target languages.
Advantageously, the method of translation with the help of translation and other types of dictionaries is most convenient in the circumstance where a device is without Internet access or when the Internet connection costs too much. Dictionaries may be installed locally on the electronic device.
On the contrary, machine translation is more reasonable to use in a case when a text to be translated includes sentences or words with linguistic and grammatical links between them. An example of a text image translated under the direction of a machine translator is shown in
In one embodiment, a method of translating a sentence from one source language into another output or target language includes analyzing the source sentence using linguistic descriptions of the source language, constructing a language-independent semantic structure to represent the meaning of the source sentence, and generating an output sentence to represent the meaning of the source sentence in the output language using linguistic descriptions of the output language.
The electronic device 102 may comprise a general purpose computer embodied in different configurations such as a smartphone, cell phone, digital camera, desktop personal computer (PC), laptop computer, or any other gadget and having touch screen as a display screen.
The electronic device 102 may be provisioned with software that includes a dictionary application to implement techniques disclosed herein. The application may include one or more local dictionaries. Additionally, the application may be able to access one or more remote dictionaries located on a remote server via a network connection to a server (not shown).
The electronic device 102 may be also provisioned with software that performs machine translation or a machine translation system may be located on a remote server and run via network connection to the server, for example a Model-Based MT system, such as one disclosed in U.S. Pat. Nos. 8,195,447 and 8,214,199. In a preferred implementation, the system provides syntactically coherent output. Syntactic and morphological descriptions of the input and output languages are used for this purpose. The teachings of these patents are incorporated by reference to the extent necessary to enable the instant invention.
Referring to
Referring now to
The process 201 is responsible for receiving and displaying a text or textual image on a screen of electronic device. For example, it may be a photograph, portable document format (PDF) file, or word processor file. After receiving the image with the text to be translated, the disclosed method is activated. In one embodiment the translation can start by detecting a pressing of a button of a camera or camera application to initiate and capture a photograph or image with text or by detecting a first swiping motion (108) in relation to an existing or original image.
At first, the software application identifies a text or at least one rectangle in an image that includes a text 202.
An optical character recognition (OCR) software 203 is applied to the identified region in the image with text. The result of the OCR is a word, word combination, sentence or longer passage of text represented by a string of characters. At the OCR step 203, morphology dictionaries may also be used as higher OCR accuracy and error correction is often achieved by comparing the recognition results with similar word forms in one or more morphology dictionaries. At least one of a machine translation or dictionary translation is performed 204. The output language of translation may be preliminary selected or configured through a selection of a target language in a software application.
Also, the dictionary or set of dictionaries may be preliminarily selected by the user or may be programmatically or automatically made available or selected based on one or more words that are successfully recognized or translated. For example, if an initial OCR operation encounters or recognizes a Russian word, the software application may assign the source language as Russian, and the target language as another language other than Russian. Dictionary software may use, by default, one or more dictionaries or a user may specify one or more desired dictionaries. A default dictionary on a given subject may be selected if the dictionary software determines that the text belongs to a specialized subject (e.g., medicine, law, automobiles, computers).
Additionally, the electronic dictionary may include a morphology module, so that the query word or word combination need not be in a base, or “dictionary” form—the morphology module identifies the base form of an inflected form. If more than one base form is possible, the morphology module identifies possible alternatives. Also, in some cases, the morphology module may determine a grammatical form of the source word, for example, a number for nouns or form of verbs, to select a proper form for the translation in an output language.
Also the morphology module identifies relevant word forms of variants of translation, for example, in accordance with the form of the translated word in the original sentence such as tense forms of verbs or a noun in a plural form.
The result of translation is displayed on another screen of the electronic device at step 205 or may be queued or rendered onto a virtual screen that is physically not visible until after a swiping motion is detected by the device. A screen or display with translated text (106) follows the screen with the original text (104) on the electronic device by a detecting a swiping action and vice versa (the original text may be re-displayed by a swiping motion in a reverse or opposite direction—e.g., a substantially sideways motion).
At step 206, screens may be switched by swiping to another screen as performed by a user (such as shown by element 106 in
If a user is not satisfied with the result of translation, such as proposed by a machine or a dictionary translation, a word-by-word or phrase-by-phrase correction or adjustment of translated text may be performed 207 such as with the help of dictionaries.
The output is a display or screen with a translation 208 of the original text. The translated text may be put over the same background. For this result, the characters or letters of the original text may be removed, and the place of their location may be painted or filled with the color of surrounding background to make the background appear substantially similar to the original background. After that, the translated text is put on the new background. Or, the translated text may be displayed without the original background. This variation may be set so that a background may be rendered or not by default when a translation is shown.
Referring now to
Referring to
After acquiring relevant coordinates, the method involves determining whether the corresponding point or region is in a text area. If the coordinates correspond to a text area, the word region is identified (302). A word region may contain one word or a word combination. The identified word or word combination is then passed as a query to the dictionary or relevant dictionary functionality.
At step 303, by touching on a word or word combination, several alternative variants of translation of the word or word combination may be displayed on the screen 106 of the electronic device 102. Variants of translation may be displayed in a dropdown list, in a balloon, in a pop-up window, as a subscript, or in any other suitable manner.
At step 304, the desired variant of translation may be chosen by, for example, touching it, touching near it, tapping on or near it, or other suitable mechanism. Manual input (typing) of an appropriate variant (that is not displayed in the list) may be performed by the user in case of the absence in the list of proposed variants for a variant that a user considers the appropriate one.
Next, at step 305, a desired variant of translation is inserted in the translation (target language text). In this way, an updated translation is shown on the display of the hardware device. Variants for a word or word combination in the source text may also be generated and displayed. In this situation, a user can modify the source text, and then making a swiping motion or gesture to receive or view a new result of translation based upon the change to the source text.
In the example shown in
As shown in
As shown in
In some cases, translation is carried out not properly or it is more useful to leave the word or the word combination without translation (in a first source language) because during translation the word or word combination can lose its original sense, for example, text associated with names and titles. Moreover, a word or word combination can have several equally valid meanings and therefore a translation may have several variants of translation into another language. Therefore, it is more reasonable in some circumstances to render a translation and retain some of the words in a first or original language and then to give a user a choice of variants of translation from the subject area corresponding to a context. So, to avoid at least the above listed shortcomings, the disclosed invention allows a user to make corrections in a translated text through a user interface and related mechanisms such as the ones shown in
If the user is not satisfied with the quality of a machine or dictionary translation, a user can manually make adjustments in a translated text. For example, as in
The most appropriate variant of translation in the example is arguably the noun “rasps.” So by touching on the display on or near this variant, a system user can insert a word in the translated text instead of taking the word or variant initially proposed by the translation system.
Moreover by touching on any word to be corrected, all other variants of its usage in the whole translated text are highlighted. So as shown in
Also, the following option can be specified. By manually correcting any word or word combination in a translated text, the system may propose automatic correction of other identical or similar variants of the same usage in the text. In the above disclosed example, a result of substitution is demonstrated by changing “shavings” to an arguably more appropriate variant of translation, “rasps.” These words are highlighted in the text shown in
In the disclosed invention, during correction of translated text, a user can manually input (e.g., type) the appropriate variant of translation in a case when there are no suitable variants in the list proposed by a dictionary. In one implementation, this option may be performed by touching the button “add” (406), manually typing a variant of translation and inserting it in the translated text. Also, the inputted variants may be stored, for example, in the user dictionary.
In one implementation the system displays not only a translation of a word, but wordforms of the word to be translated, e.g. tense forms of verbs, a noun in plural form, etc.
In our example the word “raspeln” is used in the plural in the text. That is why by touching on “raspeln” in the dropdown list with the following variants of translation from the source German language into the output in the English language are shown and proposed: shavings, grates, chips, rasps, and rattles. As mentioned above, a morphology module may be implemented for this purpose.
For a word being translated or any variant of translation, a reference or calling of an entry with a detailed translation of a word, examples of use, views of a translation from dictionaries of various subjects (e.g. Universal, Computer, Economics, Science, etc.) is possible. For example, a detailed translation may be called by double-touching on the interesting word, or any other manner of opening an entry with a detailed translation may be preliminary specified.
The disclosed method is especially useful for a quick translation of the text image that may be easily loaded in or received by an electronic or mobile device, for example a photo. By swiping the screen, translation of the text on image is performed. The result of this translation may be easily compared with the original text. The disclosed invention is useful in everyday life for example for tourists and students, who can easily and quickly generate a translation of a desired advertisement, newspaper story, title, and text from menus, posters, recipes, etc.
The hardware 500 also typically receives a number of inputs and outputs for communicating information externally. For interface with a user or operator, the hardware 500 usually includes one or more user input devices 506 (e.g., a keyboard, a mouse, imaging device, scanner, etc.) and a one or more output devices 508 (e.g., a Liquid Crystal Display (LCD) panel, a sound playback device (speaker). To embody the present invention, the hardware 500 must include at least one touch screen device (for example, a touch screen), an interactive whiteboard or any other device which allows the user to interact with a computer by touching areas on the screen. The keyboard is not obligatory in case of embodiment of the present invention.
For additional storage, the hardware 500 may also include one or more mass storage devices 510, e.g., a floppy or other removable disk drive, a hard disk drive, a Direct Access Storage Device (DASD), an optical drive (e.g. a Compact Disk (CD) drive, a Digital Versatile Disk (DVD) drive, etc.) and/or a tape drive, among others. Furthermore, the hardware 500 may include an interface with one or more networks 512 (e.g., a local area network (LAN), a wide area network (WAN), a wireless network, and/or the Internet among others) to permit the communication of information with other computers coupled to the networks. It should be appreciated that the hardware 500 typically includes suitable analog and/or digital interfaces between the processor 502 and each of the components 504, 506, 508, and 512 as is well known in the art.
The hardware 500 operates under the control of an operating system 514, and executes various computer software applications 516, components, programs, objects, modules, etc. to implement the techniques described above. In particular, the computer software applications will include the client dictionary application and also other installed applications for displaying text and/or text image content such a word processor, dedicated e-book reader etc. in the case of the client user device 102. Moreover, various applications, components, programs, objects, etc., collectively indicated by reference 516 in
In general, the routines executed to implement the embodiments of the invention may be implemented as part of an operating system or a specific application, component, program, object, module or sequence of instructions referred to as “computer programs.” The computer programs typically comprise one or more instructions set at various times in various memory and storage devices in a computer, and that, when read and executed by one or more processors in a computer, cause the computer to perform operations necessary to execute elements involving the various aspects of the invention. Moreover, while the invention has been described in the context of fully functioning computers and computer systems, those skilled in the art will appreciate that the various embodiments of the invention are capable of being distributed as a program product in a variety of forms, and that the invention applies equally regardless of the particular type of computer-readable media used to actually effect the distribution. Examples of computer-readable media include but are not limited to recordable type media such as volatile and non-volatile memory devices, floppy and other removable disks, hard disk drives, optical disks (e.g., Compact Disk Read-Only Memory (CD-ROMs), Digital Versatile Disks (DVDs), flash memory, etc.), among others. Another type of distribution may be implemented as Internet downloads.
While certain exemplary embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative and not restrictive of the broad invention and that this invention is not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those ordinarily skilled in the art upon studying this disclosure. In an area of technology such as this, where growth is fast and further advancements are not easily foreseen, the disclosed embodiments may be readily modifiable in arrangement and detail as facilitated by enabling technological advancements without departing from the principals of the present disclosure.
