Patent Grant
11972227
The present invention is directed generally to speech-to-speech translation systems for cross-lingual communication and, more particularly, to a method and system for field maintenance that enables users to add new vocabulary items and to improve and modify the content and usage of their system in the field, without requiring linguistic or technical knowledge or expertise. In various examples, the systems and methods disclosed also enable non-expert users to improve and modify the coverage and usage of their system in the field and to maximize the usefulness for effective communication without requiring linguistic or technical knowledge or expertise and provide a novel.
Automatic speech recognition (ASR) and machine translation (MT) technologies have matured to the point where it has become feasible to develop practical speech translation systems on laptops or mobile devices for limited and unlimited domains. Domain limited speech-to-speech systems, in particular, have been developed in the research field and in research laboratories for a variety of application domains, including tourism, medical deployment and for military applications. Such systems have been seen before in the works of A. Waibel, C. Fugen, “Spoken language translation” in Signal Processing Magazine, IEEE May 2008; 25(3):70-79; and Nguyen Bach, Matthias Eck, Paisarn Charoenpornsawat, Thilo Köhler, Sebastian Stüker, ThuyLinh Nguyen, Roger Hsiao, Alex Waibel, Stephan Vogel, Tanja Schultz and Alan W. Black, for examples. “The CMU TransTac 2007 eyes-free and hands-free two-way speech-to-speech translation system,” In Proc. of the IWSLT, Trento, Italy, October 2007. They are limited, however, in that they operate with a limited vocabulary which is defined by the developers of the system in advance, and is determined by the application domain, and the location where it is envisioned the system will be used. Thus vocabularies and language usage are determined largely based on example scenarios and by data that is collected or presumed in such scenarios.
In field situations, however, actual words and language usage deviate from the anticipated scenario of the laboratory. Even in simple domains such as tourism language usage will vary dramatically in the field as a user travels to different locations, interacts with different people and pursues different goals and needs. Thus, new words and new expressions will always arise. Such new words—in speech recognition parlance “out-of-vocabulary” (OOV) words will be misrecognized as an in-vocabulary word and then translated incorrectly. The user may attempt a paraphrase, but if a critical word or concept (such as a person or a city name) cannot be entered or communicated, the absence of the word or expression may lead to communication break-down.
Despite the need for user modifiable speech-to-speech translation systems, an actual solution has so far not been proposed. While adding a word to the system may seem to be easy, making such modifications proves to be extraordinarily difficult. Appropriate modifications must be made to many component modules throughout the entire system, and most modules would have to be retrained to restore the balance and integrated functioning of the components. Indeed, about 20 different modules would have to be modified or re-optimized to learn a new word. Such modifications require expertise and experience with the components of a speech translation system, and as a result, to the inventor's understanding, such modifications have so far been done only in the laboratory by experts, requiring human expertise, time and cost.
For example, if a system designed for users in Europe does not contain the name “Hong Kong” in the vocabulary. Once a speaker speaks the sentence “Let's go to Hong Kong”, the system will recognize the closest sounding similar word in the dictionary and produce: “Let's go to home call”. At this point it is not obvious if the error was the result of a recognition error or result of the absence of this word in the entire speech-to-speech translation system. The user therefore proceeds to correct the system. This can be done by one of several correction techniques. The simplest might be re-speaking or typing, but it can alternatively be done more effectively by cross-modal error correction techniques as described by other disclosures and prior art (Waibel, et al., U.S. Pat. No. 5,855,000). Once the correct spelling of the desired word sequence has been established (“Let's go to Hong Kong”), the system performs a translation. If “Hong Kong” is in the dictionary, the system would proceed from there normally, performing translation and synthesis. If, however, it is absent from the recognition and translation dictionary, the system would need to establish if this word is a named entity or not. Finally, and most importantly, even if a name or word can be translated properly to the output languages by user intervention, without learning it, the system would fail again when the user speaks the same word the next time around.
Unfortunately, learning a new word cannot be addressed just by simply typing in a new word in a word list, but it requires changes at about 20 different points and at all levels of a speech translation system. Presently it also involves manual tagging and editing of entries, collection of extensive databases involving the required word, retraining of language model and translation model probabilities and re-optimization of the entire system, so as to re-establish the consistency between all the components and components' dictionaries and to restore the statistical balance between the words, phrases and concepts in the system (probabilities have to add up to 1, and thus all words would be affected by a single word addition).
As a result, even small modifications of existing speech translation systems have generally required use of advanced computing tools and linguistic resources found in research labs. For actual field use, however, it is unacceptable to require every modification to be done at the lab, since it takes too much time, effort and cost. Instead, a learning and customization module is needed that hides all the complexity from the user, and performs all the critical operations and language processing steps semi-autonomously or autonomously behind the scenes, and interacts with the human user in the least disruptive manner possible by way of a simple intuitive interface, thereby eliminating the need for linguistic or technical expertise in the field altogether. In the present invention, we provide a detailed description for a learning and customization module that satisfies these needs.
Speech translation systems have achieved a level of performance that now makes effective speech translation in broad popular use a reality, and permits its use on small portable platforms such as laptop computers, PDA's and mobile telephones. As such, it is impractical to send speech translation systems back to the developer every time errors occur, vocabulary items are missing. Similarly, field situations dictate, that a user must be able to deal with errors quickly and effectively, and that effective communication is supported through the device, however, imperfect its performance. Error handling becomes a critical part of delivering fast and effective speech communication. Errors are problematic only, if they cannot be aborted quickly, corrected quickly and system performance improve (learn) from the correction. Thus, there is a need for systems and methods that use machine translation techniques to provide techniques to efficiently deal with errors during translation. There are also needs to provide users of such systems and methods with more robust capabilities to facilitate translation and enable the user to take advantage of its customized system such as through the creation of one or more added modules including favorites lists, language learning tools and notification of third party information based upon usage of the system and methods.
In various embodiments, the present invention solves the foregoing problems by providing a system and methods for updating the vocabulary of a speech translation system. A computer assisted method is provided for overriding the recognition or translation of an utterance input in a speech translation system for translating a first language into a second language. The methods include, in response to receiving an utterance including a first term associated with a field, translating the utterance into a second language. In response to receiving an indication to add the first term associated with the field to a first recognition lexicon, adding the first term associated with the field and the determined translation to a first machine translation module and to a shared database for a community associated with the field of the first term associated with the field, wherein the first term associated with a field added to the shared database is accessible by the community.
Those and other details, objects and advantages of the present invention will become better understood or apparent from the following description and drawings showing embodiments thereof.
The accompanying drawings illustrate examples of embodiments of the present invention. In such drawings:
Various embodiments of the present invention describe methods and systems for speech-to-speech translation. Embodiments may be used to adapt to the user's voice and speaking style via model adaptation. In further embodiments, the user can correct recognition errors and the system can explicitly learn from errors that the user corrected, thereby making it less likely that these errors occur again in the future. The present invention enables the user to customize the vocabulary to his or her individual needs and environment by either adding new words to the system, or selecting predefined dictionaries that are optimized for a specific location or task. When adding new words a multimodal interface allows the user to correct and verify automatically generated translations and pronunciations. This allows the user to add new words to the system when the user has no knowledge of the other language. In an embodiment, the system is further configured to transmit any new vocabulary inputted by a user to a community of users. This data is collated and dictionaries are automatically generated which can then be downloaded by any user.
The system also includes two machine translation modules 3 and 8, which translate text from La and Lb and from Lb and La, respectively (module 11). The MT module used in this example was the “PanDoRA” system developed at Mobile Technologies, LLC. Other MT modules could be used such as those developed by IBM Corporation, SRI, BBN or at Aachen University.
Two text-to-speech engines, 4 and 7 each corresponding to one of the machine translation modules 3 and 8, are configured to receive text produced from a corresponding ASR unit. The output text is transferred to the respective MT module, 3 or 8, that translate text from La and Lb and from Lb to La, respectively. The TTS module generates audio output to convert at least one text word in La to speech via an output device 5, such as a loud speaker, and at least one text word in Lb to speech via device 5 or another output device, such as a loud speaker 6, respectively. For this example a Cepstral US module was used. Any TTS modules which support Windows SAPI (speech application programming interface) conventions could also be employed.
A correction and repair module 11 allows the user to correct the system output via multiple modalities; including speech, gesture, writing, tactile, touch-sensitive and keyboard interfaces, and enables the system to learn from the user's corrections. The correction and repair module may be of the type such as that disclosed in U.S. Pat. No. 5,855,000. A user field customization module 12, provides an interface for users to add new vocabulary to the system, and can also select an appropriate system vocabulary for their current situation. For example, triggered by a change in location, as determined by the GPS coordinates indicating the current location of the device, or an explicit selection of task or location by the user.
The user can access the user field customization module 12 and interact with the system via a graphical user interface displayed on the screen (or active touch screen) of the device 13, and a pointing device 14, including a mouse or pen. An example of a graphical user interface is shown in
In an embodiment, the same microphone and loud-speaker can be used for both languages. Thus, microphones 1 and 10 can be a single physical device, and speakers 5 and 6 can be a single physical device.
In an embodiment, an information extraction module, 12a, will be present which extracts information from recent system logs 14e (for language La) and 14f (for language Lb), and presents targeted advertisement and supportive information via a graphical user interface displayed on the screen of the device 13. Relevant information will be obtained from the internet or a local knowledgebase.
In an embodiment, a language learning module, 12b, will be present which extracts information from recent system logs (14e and 14l) to subsequently be used for language learning. Drills are automatically generated based on recent system usage by the user, additionally the user can choose any sentence output from the system to add to his/her active learning drill (step 34c).
A flow chart illustrating the operation of an example of the method of the present invention is shown in
Translation is then applied via MT module 3 or 8 based on the input language (step 29). MT modules 3 and 8, apply three main models: a tagging or parsing [Collins02] model to identify word classes (model 22), a class-based translation model (model 23), and a class-based language model (model 24). The tagging model 22 may be any suitable type of tagging or parsing model such as the types described in J. Lafferty, A. McCallum, and F. Pereira, “Conditional random fields: Probabilistic models for segmenting and labeling sequence data,” In Proceedings of 18th International Conference on Machine Learning, pages 282-289, 2001 (“Lafferty01”) or Michael Collins, “Parameter estimation for statistical parsing models: Theory and practice of distribution-free methods” (2004) In Harry Bunt, John Carroll, and Giorgio Satta, editors, New Developments in Parsing Technology, Kluwer. Other models that are applied during the machine translation include distortion models, which constrain how words are re-ordered during translation, and sentence length models. A detailed description of class-based machine translation is given below. The resulting translation is displayed via the GUI on device 13 as shown in step 30.
To help the user determine if the translation output is adequate, the automatically generated translation (
Thereafter, if the user is dissatisfied with the generated translation, the user may intervene during the speech-to-speech translation process in any of steps from 27 to 33 or after process has completed. This invokes the Correction and Repair Module 11 at (step 35). The Correction and Repair Module (Module 11) records and logs any corrections the user may make, which can be later used to update ASR modules 2 and 9 and MT modules 3 and 8 as described in detail further below in this document. If the correction contains a new vocabulary item (step 36), or if the user enters the field customization mode to explicitly add a new word to the system in step 15c, or if a new word is automatically detected in the input audio using confidence measures or new word models, such as the method described in Thomas Schaaf, “Detection of OOV words using generalized word models and a semantic class language model,” in Proc. of Eurospeech, 2001 in step 15d; the User Field Customization Module (Module 12) is invoked.
In addition to the consecutive translation mode where the user holds down a push-to-talk button (step 15b) and speaks only a single utterance per-term, in an embodiment of the system a simultaneous translation mode will be present. In this mode no button push is required but the system continuously recognizes and translates all speech present on both microphone inputs (
In addition to the speech translation modes the user can exit the main system to enter either the “field customization” mode (step 15c), the “translations favorites” mode (step 15f), or the “language learning” mode (step 15g).
During use any sentence-pair outputted by the system can be added to the users “translations favorites” list (step 34b).
The User Field Customization Module (Module 12) provides a multimodal interface to enable users to add new words to the active system vocabulary. When a new word or phrase is added by a user the ASR, MT and TTS models (items 17, 21 and 33a) are updated as required.
A common set of classes (for example person names, place names, and organization names) is used in both ASR and MT for both languages. This provides a system-wide set of semantic slots that allows new words to be added to the system. The names, special terms and expressions that occur within these classes are the words that are most variable depending on different users' deployments, locations, cultures, customs and tasks, and thus they are in greatest need of user-customization.
In a preferred example, the specific classes used are dependent on the application domain of the system. The classes may include semantic classes for named-entities; person, place and organization names; or task-specific noun phrases; for example: names of foods, illnesses or medicines; and another open class for words or phrases that don't fit into any of the predefined classes. Syntactic classes or word equivalence classes such as synonyms could also be used. Examples of application domains include, but are not limited to, tourist, medical, peace keeping, and the like. In an example, classes required in the tourist application domain include names of persons, cities, foods and the like. In another example, for a medical professional application classes required include names of diseases, medications, anatomical names, and the like. In another example, classes required for a peace-keeping application include names of weapons, vehicles, and the like. To enable field customizable speech translation, the system permits error correction and later learning from these errors through the operation of the correction and repair module 11 in combination with a user field customization module 12.
Correction and Repair Module
The Correction and Repair Module (module 11) enables a user to intervene in the speech-to-speech translation process at any time. The user may either identify and log an error, or, if he/she wishes, correct an error in the speech recognition or translation output. Such user intervention is of considerable value, as it provides immediate correction in the human-human communication process, and opportunities for the system to adjust to user needs and interests and to learn from mistakes. A flow diagram illustrating this error feedback functionality is shown in
The user can also correct the speech recognition or machine translation output via a number of modalities. The user can correct the entire utterance, by re-speaking it or entering the sentence via a keyboard or handwriting interface. Alternatively a user can highlight an erroneous segment in the output hypothesis via the touch-screen, mouse or cursor keys and correct only that phrase or word, using the keyboard, handwriting, speech, or explicitly spelling out the word letter-for-letter. The user can also select an erroneous segment in the output hypothesis via the touch screen and correct it by selecting a competing hypothesis in an automatically generated drop-down list, or by reentering it by speech, or by any other complementary modality (e.g., handwriting, spelling, paraphrasing, etc.). These methods and how to suitably combine complementary repair actions build on methods proposed by Waibel, et al., in U.S. Pat. No. 5,855,000 for multimodal speech recognition correction and repair. Here they are applied to the speech recognition and translation modules of interactive speech translation systems.
If the user corrects the speech recognition output (step 43) the system first determines if the correction contains a new word (step 44). This determination is made by checking for the word in the recognition lexicon model 20 associated with each language, La and Lb. If the word is not found the system prompts the user to add the new word to the active system vocabulary if desired (
A user can also correct the machine translation output if they have sufficient language expertise. The same modalities as used in the ASR case can be used. If the machine translation output is corrected by the user (step 45) and the correction contains a new word, then the user is prompted with a dialog enabling them to add the new word to the active system vocabulary (
In the present invention, we introduce an abort action into the correction module. The abort action instantaneously aborts the speech-translation processes. It removes any hypotheses or partial hypotheses that may have already been output and resets the system to accept a new input. The abort action can be initiated by the user either depressing the push-to-talk button via the graphical user interface again (thus reinitiating recording for more input without waiting before the old one completes processing) or by shaking the phone to stop all processing. Output from internal accelerometers [14a] embedded in the hardware device or from an embedded camera [14b] that measures rapid shaking movement are used to determine if the device is being shaken by the user. The determination of the wrist shake by the user is then used to abort all processing that may be in progress and clear's the device's screen. The abort action can also be confirmed acoustically to the user with a crumbling paper noise, or other sound icon that confirms acoustically during field use that the speech translation process has been aborted. The simplicity of a shake of a wrist as well as the accompanying methods to confirm the abort by way of an acoustic sound icon, noise, beep, keyword or signal, provides simple fast, effective, unambiguous and intuitive signaling for both dialog partners that correction has taken place.
User Field Customization Module
User field customization module 12 enables the system to learn new words in cooperation with the user. Prior systems do not allow users to modify vocabularies in speech-to-speech translation systems. Unlike prior systems, user field customization model 12 enables the user to make incremental modifications in a running system that are relatively easy to perform for a non-expert, with minimal or no knowledge of computer speech and language processing technology or of linguistics. Model 12 offers such field customization by providing and accepting certain easy-to-understand feedback from the user, and based on this feedback deriving all the necessary parameters and system configurations autonomously. Field customization module 12 accomplishes this through: 1) an intuitive interface for user-customization, and 2) internal tools that automatically estimate all the internal parameters and settings needed for user customization, thereby relieving the user from this burden.
For unidirectional translation, the system processes a minimum of four pieces of information about the word or phrase to add a new word or phrase to the active system vocabulary. These include:
A flow chart illustrating the steps of operation of the user field customization model 12 is shown, for example, in
After the user indicates that he/she wishes to add a new word to the system vocabulary (step 50), the system first compares the word to entries in a large background recognition lexicon (item 50d), as listed in
To add a new word to the active system vocabulary, three steps are required (steps 59, 59a, 59b). First the word and its translation are added to the ASR recognition lexicons of modules 2 and 9 (step 59). The word is added to this recognition lexicon 20 along with the pronunciation(s) given by the dictionary. As the user has just entered this word its probability of occurrence is set to be greater than competing members of the same class within the ASR class-based language model 19. This is to make words that were specifically added by the user more likely. Next, the word and its translation are added to the MT models (
When the new word entered by the user is not found in the external dictionary, the system will automatically generate the information required to register the word into the active system vocabulary, and will verify this information with the user. First, the class of the new word is estimated via a tagging model (
Similarly, a new word (specifically, “word+translation+word class”) can be appended to the MT translation model (59a), the translation model 23 (which is can be stored as a hash-map within MT modules 3 and/or 8) is searched and an new translation-pair containing the new word its translation, and word class is appended. This enables the new word to be added to the MT modules 3 and/or 8, dynamically, and the new word will be translated correctly in proceeding utterances. The MT systems do not need to be re-initialized or re-started as in prior works.
Estimating all this information automatically is essential, so that a non-expert user in the field can perform the task of customization. In the following, we describe in detail, how this critical information about a word is estimated automatically, and then, how it can be obtained or verified intuitively from the user.
Generation of Pronunciations and Translations of New Words As users of speech-to-speech translation systems usually have limited or no knowledge of phonetics, linguistics, language technology, and often even have no knowledge of the word and its use in the other language, they cannot be expected to provide a translation and all the pertinent information (pronunciation, orthography, word use, etc.) of each new-word they wish to add to the system. Thus, when the user enters a new-word, the system estimates the word-class and generates the translation and pronunciation information of the word in both languages, automatically.
To register a new word into the active system vocabulary, the translation of the word and pronunciations for both the word and its translation are required. Generating this information can be implemented as a three-step process as shown, for example, in
Similarly, to add a new Japanese word “Wakayama” (item 70) to the system, first the Japanese pronunciation is generated via manually defined rules (step 71). Next, the transliteration of this word in Japanese is automatically generated via rule-based transliteration (step 72), the English pronunciation is then generated via manually defined rules (step 73). The rule based transliteration may be performed using methods of Mansur Arbabi, Scott M. Fischthal, Vincent C. Cheng, and Elizabeth Bar, “Algorithms for Arabic name transliteration,” IBM Journal of research and Development, 38(2):183¬193, 1994. The resulting information (item 74) is then verified by the user before registering the word into the active system vocabulary.
The user can verify the generated translation and pronunciation via audible output. Alternatively written form may be used if considered more suitable for the user, given their native language (i.e. in “Hanyu Pinyin” for Chinese, or “Romaji” for Japanese if the user is an English speaker). The user may edit the translation and/or pronunciation if required. Once approved by the user, the word and word characteristics are added to the multilingual system dictionary.
The system also eliminates the need for a translation of each new word that is added to the dictionary by automatically generating the required information with the assistance of interactive user input. An example of a user interface is shown in
Interactive User Interface
Thereafter, the system consults the user to confirm and verify the estimated linguistic information. This is done in an intuitive manner, so as not to presume any special linguistic or technical knowledge. Thus, a suitable interface is used. In the following we describe the user interaction during new word learning.
In the interface, the user may select a “new-word” mode from the menu, or the new word learning mode could be invoked after a user correction has yielded a new/unknown word. In the window pane that appears he/she can now type the desired new word, name, special term, concept, expression. Based on the orthographic input in the user's language (this can be character sets different from English, e.g., Chinese, 64 CIP/CIP Japanese, Russian, etc.). The system then generates a transliteration in Roman alphabet and the words predicted pronunciation. This is done by conversion rules that are either hand written or extracted from preexisting phonetic dictionaries or learned from transliterated speech data. The user then views the automatic conversion and can play the sound of the generated pronunciation via US. The user may iterate and modify either of these representations (script, Romanized transliteration, phonetic transcription, and its sound in either language) and the other corresponding entries will be regenerated similarly (thus a modified transcription in one language may modify the transcription in the other).
The system further automatically selects the most likely word class that the new word belongs to based on co-occurrence statistics of other words (with known class) in similar sentence contexts. The new word window pane also allows for a manual selection (and/or correction) of this class identity, however, so that the user can override any such estimated class assessment.
In summary, given a new word/phrase from user, the system will
If the user enters a word that does not match any of the pre-defined classes within the system, the user can assign it to the ‘unknown’ class. For ASR, the ‘unknown’ class is defined by words that occurred in the training data but not in the recognition lexicon. For SMT bilingual entries that do not occur in the translation lexicon are set to the unknown tag in the target language model.
Intra-Class Probability and Relevance Boosting
Neither of these input methods requires linguistic training and provides an intuitive way for the user to judge if a new word was suitably represented. The user may then accept this new word entry by adding the word to a “multilingual system-dictionary”, that is a user's individual lexicon. The overall system merges standardized lexica with customized lexica into the user's runtime dictionary.
In addition to the above five entries, an intra-class probability P(w|C) is also defined. In this fashion it is possible for the system to differentiate between words belonging to the same class. Thus words that are closer to the user's tasks, preferences and habits will be preferred and a higher intra-class probability assigned. This boosting of higher intra-class probability is determined based on relevance to the user, where relevance is assessed by observing:
Such observations and relevance statistics are collected based on the user's observed location, history or activity, and/or alternatively by observing the occurrence of the system's new word in a large background language resource such as the internet. Such statistics may be collected monolingually, in a data-rich language and applied to the translation dictionary and translation language model.
The relevance of boosted words may also decay over time, as the user's new activities and tasks render such words less likely over time and/or if new information (as the arrival at a different city) make a subclass of words less relevant.
Cross-Modal Entry
Optionally, a new word is entered by one of the following:
The methods described in the previous sections are all aimed at allowing an individual user to customize a speech translation system to his/her individual needs and tasks in the field. Many of such user customizations could, however, be useful to other users as well. In an embodiment, user customizations are uploaded to a community wide database, where names, special terms, or expressions are shared between interested parties. The vocabulary entries, translations and class tags are collected and related to similarly interested communities. Subsequent users can download these shared community resources and add as resource to their own system.
Alternatively, users may choose to only upload poorly translated sentences, to request manual translation from the community. For such incorrect or incomplete source words or sentences and their missing or incorrect translations other human users can provide online correction and translation on a volunteer (or paid fee) basis. The resulting corrections and translations are once again resubmitted into the updated shared community translation database.
Unsupervised Adaptation
After correction, repair and new word learning, finally, we obtain a corrected hypothesis, and thus a true transcript or translation of a spoken sentence. The speech-to-speech translation device or system automatically can use the fact that such ground truth has been provided to further adapt the ASR modules (
Class-Based Machine Translation
In the previous sections, we have described error repair and new word learning. In these modules, reference was made to class-based machine translation. In the following, we describe the detailed functioning of such class-based machine translation.
The Approach
State of the art machine translation systems perform translation on the word-level. This is evident from prior translation systems including those described in the following three documents; (1) P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin, and E. Herbst, ‘Moses: Open source toolkit for statistical machine translation’, In Proc. ACL, 2007 (“[Koehn07”); (2) D. Chiang, A. Lopez, N. Madnani, C. Monz, P. Resnik and M. Subotin, “The Hiero machine translation system: extensions, evaluation, and analysis,”, In Proc. Human Language Technology and Empirical Methods in Natural Language Processing, pp. 779-786, 2005 (“Chiang05”); and (3) K. Yamada and K. Knight “A decoder for syntax-based statistical MT”. In Proc. Association for Computational Linguistics, 2002 (“Yamada02”). Alignment is performed word-to-word; translation examples, or phrase-pairs are matched at the word level; and word-based language models are applied. Hierarchical translation modules such as those in Chiang05, and syntax-based translation models such as in Yamada02, extend on this by introducing intermediate structure. However, these approaches still require exact word matches. As each word is treated as a separate entity, these models do not generalize to unseen words.
One embodiment of class-based machine translation is class-based statistical machine translation, in which a foreign language sentence f1J=f1, f2, . . . , fJ is translated into another language eI1=e1, e2, . . . , eI by searching for the hypothesis {circumflex over ( )}eI1 with maximum likelihood, given:
Classes can be semantic classes, such as named-entities, syntactic classes or classes consisting of equivalent words or word phrases. As an example we describe the case when named-entity classes are incorporated into the system.
The two most informative models applied during translation are the target language model P(eI1) and the translation model P(fJ1|eI1). In a class-based statistical machine translation framework P(fJ1|eI1) is a class-based translation model (
Class-based models for a statistical machine translation framework can be trained using the procedure shown in
To translate an input sentence the method illustrated in
Given the tagged input sentence:
the train to @PLACE.city {Wheeling} leaves at @TIME {4:30} the following phrases can be matched:
@TIME
word or phrases within a class (i.e.: @PLACE.city{wheeling}, @Time{4:30}) are either passed directly through, which is the case for numbers/times, or the translation is determined from the translation model. Users can add new words to the translation model via the “User Field Customization Module” (
Search is performed to find the translation hypothesis with maximum likelihood P(fJ1|eI1)·P(eI1) given the translation model probability P(fJ1|eI1) (
Given the above input sentence and phrases the resulting translation will be:
In this example, even though the word “Wheeling” did not appear in the training corpora, after the user has entered the word via the “User Field Customization Module” (
Parallel Tagging of Multilingual Corpora
In an embodiment, a labeled parallel corpora is obtained by independently tagging each side of the training corpora with monolingual taggers and then removing inconsistent labels from each sentence-pair. In this approach, for each sentence-pair (Sa,Sb) the label-sequence-pair (Ta,Tb) is selected which has maximum conditional probabilities P(Ta,Sa) and P(Tb,Sb). If the occurrence count of any class-tag differs between P(Ta,Sa) and P(Tb,Sb), that class-tag is removed from the label-sequence-pair (Ta,Tb). One method to estimate P(Ta,Sa) and P(Tb,Sb) is by applying conditional random field-based tagging models Lafferty01. An example of a feature set used during monolingual tagging is shown in
In an embodiment, labeling consistency across sentence-pairs can be further improved by using the target word extracted from word-alignment (wb,j in
In another embodiment, both sentences in the translation-pair are jointly labeled while applying the constraint that the class-tag sets must be equivalent. Specifically, for the sentence-pair (Sa,Sb) we search for the label-sequence-pair (Ta,Tb) that maximizes the joint maximum conditional probability
In an embodiment, in the case where no manually annotated corpora is available for a specific language, labels can be generated by projecting labels from a first language where labels are known, across the sentence-pairs in the training corpora to the non-annotated language. One approach to do this is described in D. Yarowsky, G Ngai and R. Wicentowski, “Inducting Multilingual Text Analysis Tools via Robust Projection across Aligned Corpora,” In Proc. HLT, pages 161-168, 2001 (“Yarowsky01”). Example System and Evaluation of Class-based Machine Translation
Through experimental evaluation, we show that class-based machine translation, as detailed above, improves translation performance compared to previous approaches. Furthermore, we show that by using the parallel tagging approach described in paragraph [00082], translation accuracy is further improved.
A system for translation between Japanese and English developed for the tourist domain was evaluated. A description of the training and testing data is shown in Table 1.
To realize effective class-based SMT, accurate and consistent tagging across sentence-pairs is vital. We investigated two approaches to improve tagging quality; first, the introduction of bilingual features from word-alignment; and second, bilingual tagging, where both sides of a sentences-pair are jointly tagged. From the parallel training corpora 14,000 sentence-pairs were manually tagged using the 16 class labels indicated in Table 2.
From this manually labeled set, we selected 10% (1400 sentence-pairs) which contained one or more tags as held-out data to evaluate tagging accuracy.
First, the performance of the baseline, monolingual CRF-based taggers was evaluated. Each side of the held-out set was labeled independently, using language dependent models. The output was then compared to the manual reference. The tagging accuracy for various metrics are shown in Table 3.
For the bilingual tagging, a tag is considered correct if the entity is correctly labeled on both sides of the corpora. The right hand column indicates the percentage of sentence-pairs in which both sides were tagged correctly. Although the F-score is above 0.90 for the independent languages, the bilingual tagging accuracy is significantly lower at 0.84, and only 80% of the sentence-pairs were correctly tagged. Incorporating alignment features into the monolingual taggers improved precision for both languages and significantly improvement recall for the Japanese side, however, the percentage of correctly tagged sentence-pairs increased only slightly. Removing inconsistent tags across sentence-pairs improved precision, but the number of correctly tagged sentence-pairs did not improve.
Next, the effectiveness of bilingual tagging was evaluated using the approach described above. The tagging accuracy of this approach, and when word alignment features were incorporated are shown in the lower 2 rows of Table 3. Compared to the monolingual case, bilingual tagging significantly improved tagging accuracy. Not only did tagging consistency improve (the F-score for bilingual tagging increased from 0.84 to 0.95), but the tagging accuracy on both the English and Japanese-sides also improved. Incorporating word-alignment features gained a further small improvement in tagging accuracy for all measures.
The effectiveness of the system was further evaluated by comparing the performance of three class-based systems and a baseline system that did not use class models.
For the baseline system phrase-based translation models were trained using the Moses toolkit such as described in Koehn05 and GIZA++ (such as that used by Franz Josef Och, Hermann Ney. “A Systematic Comparison of Various Statistical Alignment Models”, Computational Linguistics, volume 29, number 1, pp. 19-51 March 2003). 3-gram language models were trained using the SRILM toolkit of A. Stolcke “SRILM—an extensible language modeling toolkit”, In Proc. of ICSLP, pp. 901-904, 2002. Decoding was performed using our PanDoRA decoder. The decoder is described in Ying Zhang, Stephan Vogel, “PanDoRA: A Large-scale Two-way Statistical Machine Translation System for Hand-held Devices,” In the Proceedings of MT Summit XI, Copenhagen, Denmark, Sep. 10-14, 2007. Systems were created for both translation directions J→E (Japanese to English) and E→J (English to Japanese) using the training set described in Table 1. The data used to train the target language models were limited to this corpora. The translation quality of the baseline system was evaluated on a test-set of 600 sentences. One reference was used during evaluation. The BLEU-score for the J→E and E→J systems were 0.4381 and 0.3947, respectively. BLEU-score is described in Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu “BLEU: a Method for Automatic Evaluation of Machine Translation, “In Proc. Association for Computational Linguistics, pp. 311-318, 2002. Translation quality using three different tagging schemes was evaluated:
Monolingual tagging was applied for the +num and +NE-class cases, and tags that were inconsistent across a sentence-pair were removed. In the +Bi-Tagging case, bilingual tagging incorporating word alignment features were used. For each tagging scheme, the entire training corpora was tagged with the appropriate set of class-labels. Class-based translation and language models were then trained using an equivalent procedure to that used in the baseline system. During testing the input sentence was tagged using a monolingual tagger. All named-entities in the test set were entered into the user dictionary to be used during translation.
The performance on the 600 sentence test-set for the baseline and class-based systems are shown in terms of BLEU-score for the J→E and E→J systems in Table 4.
The class-based SMT system using number and time tags (+num), obtained improved translation quality compared to the baseline system for both translation directions. For these models, BLEU-scores of 0.4441 and 0.4104 were obtained. When a class-based system using named-entity classes in addition to number and time tags was applied, translation quality improved significantly. BLEU-scores of 0.5014 for the J→E system and 0.4464 for the E→J case were obtained. When bilingual tagging was used to tag the training corpora (+Bi-Tagging) a further 0.8 point gain in BLEU was obtained for both translation directions. On the 14% of sentences in the test-set which contained one or more named-entities the (+Bi-Tagging) system outperformed the monolingually tagged system (“+NE-class”) by up to 3.5 BLEU points.
While the foregoing has been set forth in considerable detail, it is to be understood that the drawings and detailed embodiments are presented for elucidation and not limitation. Design and configuration variations may be made but are within the principles of the invention. Those skilled in the art will realize that such changes or modifications of the invention or combinations of elements, variations, equivalents, or improvements therein are still within the scope of the invention as defined in the appended claims.
Speech Translation Favorites
Frequently, users may say the same phrase or sentence repeatedly in a field situation. To eliminate the need to re-speak the same phrases and sentences over and over again, embodiments of the invention provide a speech translation favorites module, which stores frequently used phrases for rapid play-back. This favorites modules differs from a plain list or phrase book in one most important aspect: it gets filled and built by the speech translation device, and thus, does not require a bilingual speaker to be on hand to know the correct translation to a term or phrase. The favorites module is programmed to enable a user to perform the following functions: Copy bilingual sentence-pair from the speech translation interaction window to a favorites list; provide editing capability to the newly copied bilingual sentence-pair, so a user can modify both input and output string; and provide the ability to play back the target language side of the added sentence pair by applying the synthesis module. With the favorites list in place, a user can simply play back various accumulated phrases from the favorites list, without, speaking them first, invoking them by clicking on the selected favorite or by way of a voice command or key phrase. This saves time in field situations. At the same time the favorites list provides the full flexibility of a full two-way speech¬to-speech translator since it does not require a linguistically trained expert knowledgeable of both languages to build such a list. A customized list can be built by the user in the field and on the fly.
In addition to the basic speech translation system shown in
Speech Translation Modes
Even when performing speech translation limited to one language pair, there are variations in language use, vocabularies and speaking style depending on many factors, including (without limitation) social situation, dialect, relative relations between the speakers, social relationship between speakers, gender and age of speaker or listener, location or speaker and listener, activity, environment, regional expression and context that are typically not differentiated by a standardized speech translator. Yet, it is in many ways critically important to separate them to achieve socially appropriate results. Thus, the same user may speak formally in a formal context at work, in an informal colloquial manner at home with his/her family, and use slang at a party with friends. Similarly, there may be situational differences, such as authoritative or submissive speaking, depending on whether the device is used by a police/military officer on duty or as a clerk in a department store. In addition to language choices, we introduce a language “mode”. This mode operates like a switch that switches the device into the appropriate speaking mode, and modifies/conditions the modeling of several system subcomponents accordingly. Language modes then affect the operation of the system, by applying language-mode dependent ASR, MT and TTS models, or by simply filtering the output of ASR and MT components. The choice of a mode is given to the user by a mode switch or setting (“polite”, “informal”, etc.) or by inferring the appropriate mode perceptually (for example, the gender or a speaker, his/her location, his/her level of formality, etc.).
The basic speech translation system shown in
Speech Recognition Lexicon Selection Based on User Repair and Boosting
To run on small devices it is frequently not possible to carry a large dictionary of words that provides a reasonable good coverage for a language. The proposed method circumvents this problem by building initial systems with considerably smaller dictionaries for efficiency. Coverage by contrast is then generally a problem as many common words may not be available in the systems dictionaries. To recover generality/robustness without paying the price of more memory requirements, a method is disclosed that can achieve a tighter more targeted dictionary and language model through personalization and customization of the system by the user. In this manner, the system design sacrifices only some generality of vocabularies of an overall user population, but retains the generality of vocabulary use by the individual owner and user of the device. Prior research shows, for example, that discussions between human conversant around a certain topic of interest will generally only have vocabulary sizes of about 4,000 words, while general speech translation systems may have vocabulary sizes of 40,000 words or more (in English). The system would therefore be delivered in a state where vocabulary is more severely curtailed than in larger more general systems and thus be more parsimonious in memory use than a larger system. With vocabularies of 4,000-10,000 words, search trees, language models and pronunciation dictionaries can be reduced dramatically over vocabulary sizes of 40,000 or more. In this case, however, we will generally observe a larger mismatch between the vocabulary of the system and the desired vocabulary of the user, and out-of-vocabulary words will appear in the spoken utterances. Now, the proposed system will come with a large background dictionary, and large pre-trained language models. This is possible without loss of advantage, since the large dictionaries and language models can be stored in flash memories that are typically available in abundance (e.g. to store music, pictures, etc.) on modern mobile phones. When an out-of-vocabulary item occurs the system now provides an easy method to correct the consequential misrecognition by various corrective mechanisms. Once the correction has been noted, the appropriate dictionaries and models (recognition, translation, synthesis) can be retrieved and added or replace less useful ones.
The speed of the speech translation system shown in
Automatic Identification of the Input Language
In current speech translators a user has to select a record button that pertains to the language of the speaker/user. In speech translators for two-way dialogs, this means that at least two record buttons have to be provided for the two possible language inputs. This, unfortunately, wastes screen real estate and can lead to user errors when the wrong button is pushed. In the interest of simplification, we propose to eliminate this source of user confusion, but providing automatic language identification first and then produce translation in the other language, no matter which language was spoken.
The basic speech translation system shown in
Language Learning from Speech Translation
Speech translators today are aiming to provide two-way speech dialog communication for people who don't speak each other's languages. Frequently, though, a user may wish to learn another person's language themselves as well. To provide such possibilities, we expand the speech translator function by the function of providing language instruction to the human user. Such an expanded speech translator/language tutor/trainer differs from other language instruction/tutoring software products in two fundamental ways: 1.) the disclosed combined language system provides a speech translator for language support when the user's language abilities are inadequate to navigate a real situation in actual use, and 2.) the combined system can then provide much more customized, targeted language learning drills and instruction that is responding specifically to a user's language learning interest, situations, and it bases its instruction on what it can observe from a user's language usage during speech translation dialog use. Thus, a user may converse with other individuals through use of the speech translator over a period of time, and then gradually attempt to learn for him/herself the key concepts, vocabularies, language constructs that he/she often uses and wishes to learn for him/herself. Thus a language learning drill can be personalized and targeted much more concretely at the specific language needs of an individual user than static impersonal language learning books or software would and thus it presents a much more personal and alive human language learning experience. In the preferred embodiment, the language learning subsystem builds vocabulary, syntax or conversational drills that derive from the sentences found in the recent history of the speech translator.
In addition to the basic speech translation system shown in
Speech Translation of Telephone Conversation
In the previous disclosures, we have considered speech translators for portable devices such as smart phones and PDA's. In all these deployments, the speech translator acts as an interpreter between two people in a face to face dialog situation where the device acts as the translator. We expand this notion, by using a speech translator on a telephone as an interpreter between people speaking over that telephone as a transmission channel with each other. To achieve this functionality, we modify the user interface. Speech is now arriving via the microphone of the user of the telephone as well as by the signal transmitted over the telephone line and is recognized and translated. The speech translation is performed in either consecutive translation mode (a speaker speaks, and then waits for translation, before the other speaker takes his turn) as well as simultaneous translation mode (when either speaker can speak continuously while the translator performs speech translation output in parallel).
The basic speech translation mode operates using speech input from microphones (items 1 and 10) and loudspeakers (items 5 and 6) located on the device. In addition to this mode, speech translation can also be performed over a telephony network. In this case audio for language Lb is transmitted over the telephone network and the microphone (item 10) and loudspeaker (item 5) for language Lb will be physically located on a recipient's telephony handset. When operated over a telephony network speech recognition (modules 2, 9) and machine translation (modules 3, 8) can be carried out in two different manners. In consecutive translation the user holds a push-to-talk button down (step 15b) to start recognizing an utterance (step 27). The utterance is then recognized and translated using the steps shown in
Information Extraction Based on Recognized and Translated Speech
Speech Recognizers and Translators operating on a smart phone can also provide information as to a user's speech content. We propose to expand the speech recognizer and translator to extract topical information from two conversants' speech. Such information is then used to seek relevant related information on the internet. Such information is then presented to the user. There are multiple uses of such conversation enabled information extraction. It could be used to provide more targeted advertising (perhaps in return for cheaper calling rates). It could also be used to provide the user with helpful supporting information (for example, calling up flight schedules, hotel availabilities & rates, recalling a person's contact details, etc.) when a conversation mentions certain topics, people, places, or activities. This information can also be recalled bilingually from sources in either of the languages handled by the speech translator.
The information extraction module (module 12a) extracts key words and phrases from system logs (items 14e and 14f) generated by the speech recognition (modules 3, 8) and machine translation (modules 3, 8) components. First, usage statistics from recent system logs (step 123) are generated. Next, relevant information is obtained from the internet or local knowledgebase (item 125) based on keyword occurrence and language usage (step 124). This information is subsequently presented to the user (step 126) via the screen of the device (FIG. 1, item 13). Presented information will include targeted advertising and supportive information. Supportive information includes flight schedules, hotel availability and rates, and contact details of persons. Information can be presented bilingually in either language (La or Lb) by searching based on keywords in the output of machine translation (modules 3, 8).
While the foregoing has been set forth in considerable detail, it is to be understood that the drawings and detailed embodiments are presented for elucidation and not limitation. Design variations may be made but are within the principles of the invention. Those skilled in the art will realize that such changes or modifications of the invention or combinations of elements, variations, equivalents, or improvements therein are still within the scope of the invention as defined in the appended claims. Also, the examples and experimental results illustrated herein are exemplary and are not intended to limit the scope of the embodiments of the invention.
This application is a continuation of U.S. patent application Ser. No. 15/694,733, filed Sep. 1, 2017, which is a continuation of U.S. patent application Ser. No. 14/589,540, filed on Jan. 5, 2015 and issued as U.S. Pat. No. 9,753,918; which is a continuation of U.S. patent application Ser. No. 13/008,346, filed on Jan. 18, 2011 and issued as U.S. Pat. No. 8,972,268 on Mar. 3, 2015; which is a continuation-in-part of U.S. patent application Ser. No. 12/689,042, filed on Jan. 18, 2010 and issued as U.S. Pat. No. 9,070,363 on Jun. 30, 2015; which is a continuation-in-part of U.S. patent application Ser. No. 12/424,311, filed on Apr. 15, 2009 and issued as U.S. Pat. No. 8,204,739 on Jun. 19, 2012; which claims the benefit of U.S. Provisional Patent Application No. 61/045,079 filed on Apr. 15, 2008, U.S. Provisional Patent Application No. 61/092,581 filed on Aug. 28, 2008, and U.S. Provisional Patent Application No. 61/093,898 filed on Sep. 3, 2008. U.S. patent application Ser. No. 12/689,042 is also a continuation-in-part of U.S. patent application Ser. No. 11/925,048, filed on Oct. 26, 2007 and issued as U.S. Pat. No. 8,090,570 on Jan. 3, 2012; which claims the benefit of U.S. Provisional Patent Application No. 60/854,586, filed on Oct. 26, 2006. All of the abovementioned applications are hereby incorporated by reference in their entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5268839 | Kaji | Dec 1993 | A |
| 5855000 | Waibel et al. | Dec 1998 | A |
| 5987402 | Murata et al. | Nov 1999 | A |
| 5995935 | Hagiwara et al. | Nov 1999 | A |
| 6006221 | Liddy et al. | Dec 1999 | A |
| 6067514 | Chen | May 2000 | A |
| 6085162 | Cherny | Jul 2000 | A |
| 6104989 | Kanevsky et al. | Aug 2000 | A |
| 6144938 | Surace et al. | Nov 2000 | A |
| 6148105 | Wakisaka et al. | Nov 2000 | A |
| 6219646 | Cherny | Apr 2001 | B1 |
| 6223150 | Duan et al. | Apr 2001 | B1 |
| 6243669 | Horiguchi et al. | Jun 2001 | B1 |
| 6246976 | Mukaigawa et al. | Jun 2001 | B1 |
| 6266642 | Franz et al. | Jul 2001 | B1 |
| 6278968 | Franz et al. | Aug 2001 | B1 |
| 6282504 | Wong et al. | Aug 2001 | B1 |
| 6292769 | Flanagan et al. | Sep 2001 | B1 |
| 6339754 | Flanagan et al. | Jan 2002 | B1 |
| 6356865 | Franz et al. | Mar 2002 | B1 |
| 6374224 | Horiguchi et al. | Apr 2002 | B1 |
| 6438524 | Shi | Aug 2002 | B1 |
| 6442524 | Ecker et al. | Aug 2002 | B1 |
| 6584439 | Geilhufe et al. | Jun 2003 | B1 |
| 6697777 | Ho et al. | Feb 2004 | B1 |
| 6721697 | Duan et al. | Apr 2004 | B1 |
| 6742047 | Tso | May 2004 | B1 |
| 6826582 | Chatterjee et al. | Nov 2004 | B1 |
| 6901367 | Berstis et al. | May 2005 | B1 |
| 6917920 | Koizumi et al. | Jul 2005 | B1 |
| 6941269 | Cohen et al. | Sep 2005 | B1 |
| 6983248 | Tahara et al. | Jan 2006 | B1 |
| 6996520 | Levin | Feb 2006 | B2 |
| 6999932 | Zhou | Feb 2006 | B1 |
| 7016849 | Arnold et al. | Mar 2006 | B2 |
| 7069222 | Borquez et al. | Jun 2006 | B1 |
| 7539619 | Seligman et al. | May 2009 | B1 |
| 7542908 | Segond et al. | Jun 2009 | B2 |
| 7552053 | Gao et al. | Jun 2009 | B2 |
| 7584102 | Hwang et al. | Sep 2009 | B2 |
| 7593842 | Rousseau | Sep 2009 | B2 |
| 7668718 | Kahn et al. | Feb 2010 | B2 |
| 7711570 | Galanes et al. | May 2010 | B2 |
| 7730764 | Tachibana et al. | Jun 2010 | B2 |
| 7752031 | Childress et al. | Jul 2010 | B2 |
| 7921018 | Hong et al. | Apr 2011 | B2 |
| 7970598 | Flanagan et al. | Jun 2011 | B1 |
| 8090570 | Waibel et al. | Jan 2012 | B2 |
| 8204739 | Waibel et al. | Jun 2012 | B2 |
| 8504351 | Waibel et al. | Aug 2013 | B2 |
| 8515728 | Boyd et al. | Aug 2013 | B2 |
| 8612211 | Shires et al. | Dec 2013 | B1 |
| 9070363 | Waibel et al. | Jun 2015 | B2 |
| 9128926 | Waibel | Sep 2015 | B2 |
| 9524295 | Waibel | Dec 2016 | B2 |
| 20010029455 | Chin et al. | Oct 2001 | A1 |
| 20010044726 | Li et al. | Nov 2001 | A1 |
| 20020095292 | Mittal et al. | Jul 2002 | A1 |
| 20020169592 | Aityan | Nov 2002 | A1 |
| 20020198713 | Franz et al. | Dec 2002 | A1 |
| 20030009320 | Furuta | Jan 2003 | A1 |
| 20030115059 | Jayaratne | Jun 2003 | A1 |
| 20040167771 | Duan et al. | Aug 2004 | A1 |
| 20050119884 | Breuer | Jun 2005 | A1 |
| 20050119899 | Palmquist | Jun 2005 | A1 |
| 20050131673 | Koizumi et al. | Jun 2005 | A1 |
| 20050144012 | Afrashteh et al. | Jun 2005 | A1 |
| 20050171944 | Palmquist | Aug 2005 | A1 |
| 20050203738 | Hwang | Sep 2005 | A1 |
| 20050228641 | Chelba et al. | Oct 2005 | A1 |
| 20050288922 | Kooiman | Dec 2005 | A1 |
| 20060136193 | Lux-Pogodalla et al. | Jun 2006 | A1 |
| 20060293896 | Nakagawa | Dec 2006 | A1 |
| 20070016401 | Ehsani et al. | Jan 2007 | A1 |
| 20070032225 | Konicek et al. | Feb 2007 | A1 |
| 20070043567 | Gao et al. | Feb 2007 | A1 |
| 20070071206 | Gainsboro et al. | Mar 2007 | A1 |
| 20070100814 | Lee et al. | May 2007 | A1 |
| 20070124131 | Chino et al. | May 2007 | A1 |
| 20070133437 | Wengrovitz et al. | Jun 2007 | A1 |
| 20070136068 | Horvitz | Jun 2007 | A1 |
| 20070150257 | Cancedda et al. | Jun 2007 | A1 |
| 20070179779 | Kagoshima et al. | Aug 2007 | A1 |
| 20070233487 | Cohen et al. | Oct 2007 | A1 |
| 20070249406 | Andreasson | Oct 2007 | A1 |
| 20080103769 | Schultz et al. | May 2008 | A1 |
| 20080103907 | Maislos et al. | May 2008 | A1 |
| 20080105769 | Tschantz | May 2008 | A1 |
| 20080120091 | Waibel et al. | May 2008 | A1 |
| 20080126087 | Chou | May 2008 | A1 |
| 20080240379 | Maislos et al. | Oct 2008 | A1 |
| 20080243474 | Furihata et al. | Oct 2008 | A1 |
| 20080254821 | Kusuda et al. | Oct 2008 | A1 |
| 20090076792 | Lawson-Tancred | Mar 2009 | A1 |
| 20090274299 | Caskey et al. | Nov 2009 | A1 |
| 20090281789 | Waibel et al. | Nov 2009 | A1 |
| 20100023472 | Loeb | Jan 2010 | A1 |
| 20100082326 | Bangalore et al. | Apr 2010 | A1 |
| 20100217582 | Waibel et al. | Aug 2010 | A1 |
| 20100251137 | Qureshi | Sep 2010 | A1 |
| 20100280828 | Fein et al. | Nov 2010 | A1 |
| 20110046939 | Balasaygun | Feb 2011 | A1 |
| Number | Date | Country |
|---|---|---|
| 0773532 | May 1997 | EP |
| H03282776 | Dec 1991 | JP |
| H04319769 | Nov 1992 | JP |
| H05189481 | Jul 1993 | JP |
| H05197744 | Aug 1993 | JP |
| H07129594 | May 1995 | JP |
| H08212216 | Aug 1996 | JP |
| H08329088 | Dec 1996 | JP |
| H09134192 | May 1997 | JP |
| H1097286 | Apr 1998 | JP |
| H11352994 | Dec 1999 | JP |
| 2000305930 | Nov 2000 | JP |
| 2001325254 | Nov 2001 | JP |
| 2002091963 | Mar 2002 | JP |
| 2002183136 | Jun 2002 | JP |
| 2002207495 | Jul 2002 | JP |
| 2003288339 | Oct 2003 | JP |
| 2004310256 | Nov 2004 | JP |
| 2005122094 | May 2005 | JP |
| 2006201873 | Aug 2006 | JP |
| 2006318202 | Nov 2006 | JP |
| WO-0231814 | Apr 2002 | WO |
| WO-2009129315 | Oct 2009 | WO |
| Entry |
|---|
| Aguero P.D., et al., “Prosody Generation for Speech-to-Speech Translation,” International Conference on Acoustic Speech Signal Process, 2006, pp. 566-560. |
| Arbabi et al., “Algorithms for Arabic Name Transliteration,” IBM Journal of Research and Development, Mar. 1994, vol. 38 (2), pp. 183-194. |
| Asadi A., et al., “Automatic Modeling for Adding New Words to a Large-Vocabulary Continuous Speech Recognition System,” Acoustics, Speech, and Signal Processing, ICASSP-91, IEEE International Conference, 1991, pp. 305-308. |
| Bach et al., “The Cmu Transtac 2007 Eyes-Free and Hands-Free Two-Way Speech-to-Speech Translation System,” Interact, Language Technologies Institute, Carnegie Mellon University, 2007, 8 pages. |
| Benus S., et al., “The Prosody of Backchannels in American English,” ICPhS XVI, Aug. 6-10, 2007, vol. 1276, pp. 1065-1068. |
| Bertrand R., et al., “Backchannels Revisited From a Multimodal Perspective,” Proceedings of Auditory-visual Speech Processing, Hilvarenbeek, Netherlands, 2007, pp. 1-6. |
| Brown P.F., et al., “The Mathematics of Statistical Machine Translation: Parameter Estimation,” Association for Computational Linguistics, 1993, vol. 19 (2), pp. 263-311. |
| Cathcart N., et al., “A Shallow Model of Backchannel Continuers in Spoken Dialogue,” Proceedings of the Tenth Conference on European Chapter, Association for Computational Linguistics, 2003, vol. 1, pp. 51-58. |
| Chiang D., et al., “The Hiero Machine Translation System: Extensions, Evaluation, and Analysis,” Proceedings of Human Language Technology Conference on Empirical Methods in Natural Language Processing, Oct. 2005, pp. 779-786. |
| Collins M., “Parameter Estimation for Statistical Parsing Models: Theory and Practice of Distribution-Free Methods,” New Developments in Parsing Technology, Chapter 2, Kluwer Academic Publishers, Dordrecht, The Netherlands, 2004, pp. 19-55. |
| Examination Report dated Nov. 22, 2017 for Indian Application No. 6681/CHENP/2010, filed Oct. 19, 2010, 6 pages. |
| Extended European Search Report for European Application No. 14190061.3, dated Jul. 2, 2015, 10 pages. |
| Final Office Action dated Oct. 10, 2017 for U.S Appl. No. 14/522,164, filed Oct. 23, 2014, 13 Pages. |
| Final Office Action dated Nov. 16, 2016 for U.S. Appl. No. 14/589,540, filed Jan. 5, 2015, 17 Pages. |
| Final Office Action dated Sep. 18, 2013 for U.S. Appl. No. 13/008,346, filed Jan. 18, 2011, 21 pages. |
| Final Office Action dated Jun. 19, 2013 for U.S. Appl. No. 12/689,042, filed Jan. 18, 2010, 9 pages. |
| Final Office Action dated May 24, 2018 for U.S. Appl. No. 14/722,013, filed May 26, 2015, 10 Pages. |
| Final Office Action dated Oct. 28, 2014 for U.S. Appl. No. 12/689,042, filed Jan. 18, 2010, 15 Pages. |
| Fugen C., et al., “Advances in Lecture Recognition: The ISL RT-06S Evaluation System,” INTERSPEECH 2006-ICSLP, Sep. 17-21, 2006, pp. 1229-1232. |
| Fugen C., et al., “LingWear: A Mobile Tourist Information System,” HLT 01: Proceedings of the First International Conference on Human Language Technology Research, Mar. 2001, pp. 1-5. |
| Fugen C., et al., “Open Domain Speech Translation: From Seminars and Speeches to Lectures,” TC-STAR Workshop on Speech-to-Speech Translation, Barcelona, Spain, Jun. 19-21, 2006, pp. 81-86. |
| Fujie S., et al., “A Conversation Robot with Back-channel Feedback Function based on Linguistic and Nonlinguistic Information,” Second International Conference on Autonomous Robots and Agents, Palmerston North, New Zealand, Dec. 13-15, 2004, pp. 379-384. |
| Gales M.J.F., “Semi-Tied Covariance Matrices for Hidden Markov Models,” Abstract, IEEE Transactions on Speech and Audio Processing, May 1999, vol. 7 (3), pp. 272-281. |
| Gales M.J.F., “Maximum Likelihood Multiple Projection Schemes for Hidden Markov Models,” Abstract, IEEE Transactions on Speech and Audio Processing, 2000, 2 pages. |
| Gollan C., et al., “Cross Domain Automatic Transcription on the TC-STAR EPPS Corpus,” IEEE International Conference on Acoustics, Speech and Signal Processing, Mar. 23, 2005, pp. 825-828. |
| Hamaker J., et al., “Resegmentation and Transcription of Switchboard,” Proceedings of LVCSR Workshop, Maritime Institute of Technology, Linthicum Heights, Maryland, United States, Sep. 1998, pp. 1-4. |
| Hurst W., “Indexing, Searching, and Skimming of Multimedia Documents Containing Recorded Lectures and Live Presentations,” Proceedings of ACM Multimedia, Berkeley, USA, Nov. 2-8, 2003, pp. 1-2. |
| International Preliminary Report on Patentability for International Application No. PCT/US2009/040677, dated Apr. 14, 2011, 15 Pages. |
| International Preliminary Report on Patentability for International Application No. PCT/US2011/021530, dated Aug. 2, 2012, 8 Pages. |
| International Search Report and Written Opinion for International Application No. PCT/US2009/040677, dated Jul. 20, 2009, 8 pages. |
| International Search Report and Written Opinion for International Application No. PCT/US2011/021530, dated Mar. 28, 2011, 10 pages. |
| Janin, A. et al., “The ICSI Meeting Project: Resources and Research,” Proceedings of the ICASSP Meeting Recognition Workshop, Montreal, Canada, 2004, pp. 1-7. |
| Kashioka H., “Translation Unit Concerning Timing of Simultaneous Translation,” Proceedings of the Third International Conference on Language Resources and Evaluation, Las Palmas de Gran Canaria, Spain, May-Jun. 2, 2002, pp. 142-146. |
| Knight K., et al., “Machine Transliteration,” Association for Computational Linguistics, 1998, vol. 24 (4), pp. 599-612. |
| Koehn P., et al., “Moses: Open Source Toolkit for Statistical Machine Translation,” Proceedings of the ACL 2007 Demo and Poster Sessions, Jun. 2007, pp. 177-180. |
| Koehn P., et al., “Open Source Toolkit for Statistical Machine Translation: Factored Translation Models and Confusion Network Decoding,” Johns Hopkins University, Center for Speech and Language Processing, Sep. 3, 2007, 52 pages. |
| Koehn P., “Europarl: A Multilingual Corpus for Evaluation of Machine Translation,” Information Sciences Institute, University of Southern California, Dec. 28, 2002, pp. 1-18. |
| Kohler T.W., et al., “Rapid Porting of ASR-Systems to Mobile Devices,” Proceedings of the 9th European Conference on Speech Communication and Technology, Lisbon, Portugal, Sep. 2005, pp. 233-236. |
| Lafferty J., et al., “Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data,” 2001, 8 Pages. |
| Lamel L.F., “The Translanguage English Database,” in: ISCA Archive, Third International Conference on Spoken Language Processing (ICSLP 94), Sep. 18-22, 1994 [Retrieved on Nov. 5, 2007], 1 page, Retrieved from the Internet: URL: http://www.isca-speech.org/archive/icslp_1994/i94_1795.html. |
| Lane I.R., et al., “Class-Based Statistical Machine Translation for Field Maintainable Speech-to-Speech Translation,” Interspeech, Sep. 22, 2008, pp. 2362-2365, XP002535596. |
| “LDC-Linguistic Data Consortium,” Ldc.upenn.edu [Online], Oct. 30, 2007 [Retrieved on Nov. 5, 2007], 2 pages, Retrieved from the Internet: URL: http://www.ldc.upenn.edu/. |
| Leggetter C.J., et al., “Maximum Likelihood Linear Regression for Speaker Adaptation of Continuous Density Hidden Markov Models,” Proceedings of Computer Speech and Language, 1995, vol. 9, pp. 171-185. |
| Litman D., et al., “Predicting Automatic Speech Recognition Performance Using Prosodic Cues,” Proceedings of NAACL, 2000, pp. 218-225. |
| Mangu L., et al., “Finding Consensus among Words: Lattice-Based Word Error Minimization,” 6th European Conference on Speech Communication and Technology (EUROSPEECH 99), Budapest, Hungary, Sep. 5-9, 1999, pp. 1-4. |
| Matusov E., et al., “Evaluating Machine Translation Output with Automatic Sentence Segmentation,” Proceedings of IWSLT, Pittsburgh, USA, 2005, pp. 148-154. |
| Metze F., et al., “The NESPOLE! Speech-to-Speech Translation System,” Proceeding of HLT 2002, Second International Conference on Human Language Technology Research, San Francisco, 2002, pp. 378-383. |
| Nanjo H., et al., “Language Model and Speaking Rate Adaptation for Spontaneous Presentation Speech Recognition,” IEEE Transactions on Speech and Audio Processing, Jul. 2004, vol. 12 (4), pp. 391-400. |
| Narayanan S., et al., “Transonics: A Speech to Speech System for English-Persian Interactions,” Automatic Speech Recognition and Understanding, Nov. 30-Dec. 3, 2013, pp. 670-675, XP10171323.0. |
| Niemann H., et al., “Using Prosodic Cues in Spoken Dialog Systems,” Proceedings of SPECOM '98 Workshop, 1998, pp. 17-28. |
| Noguchi H., et al., “Prosody-Based Detection of the Context of Backchannel Responses,” ICSLP, 1998, pp. 1-4. |
| Non-Final Office Action dated Mar. 1, 2013 for U.S. Appl. No. 13/008,346, filed Jan. 18, 2011, 29 pages. |
| Non-Final Office Action dated Oct. 3, 2016 for U.S. Appl. No. 14/722,013, filed May 26, 2015, 13 pages. |
| Non-Final Office Action dated Mar. 9, 2016 for U.S Appl. No. 14/522,164, filed Oct. 23, 2014, 12 Pages. |
| Non-Final Office Action dated Jun. 10, 2016 for U.S. Appl. No. 14/589,540, filed Jan. 5, 2015, 14 Pages. |
| Non-Final Office Action dated Sep. 18, 2015 for U.S. Appl. No. 14/797,045, filed Jul. 10, 2015, 29 pages. |
| Non-Final Office Action dated Jun. 20, 2014 for U.S. Appl. No. 12/689,042, filed Jan. 18, 2010, 14 pages. |
| Non-Final Office Action dated Nov. 21, 2014 for U.S. Appl. No. 13/675,775, filed Nov. 13, 2012, 16 Pages. |
| Notification of Reasons for Rejection dated Oct. 31, 2017 for Japanese Application No. 2015-218066, filed Nov. 6, 2015, 4 Pages. |
| Och F.J., et al., “A Systematic Comparison of Various Statistical Alignment Models,” Association for Computational Linguistics, 2003, vol. 29 (1), pp. 19-51. |
| Och F.J., et al., “Improved Alignment Models for Statistical Machine Translation,” 1999, pp. 20-28, Retrieved from the Internet: URL: https://www.aclweb.org/anthology/W99-0604.pdf. |
| Office Action dated Jul. 7, 2015 for Japanese Patent Application No. 2011505169, filed on Apr. 15, 2009, 23 Pages. |
| Office Action dated Sep. 9, 2013 for European Application No. 09732921.3, filed Apr. 15, 2009, 5 Pages. |
| Office Action dated Aug. 12, 2016 for European Patent Application No. 14190061.3, filed Jan. 18, 2011, 7 pages. |
| Office Action dated Aug. 18, 2017 for European Patent Application No. 14190061.3, filed Jan. 18, 2011, 7 pages. |
| Office Action dated Apr. 26, 2014 for Korean Application No. 10-2010-7025487, filed Apr. 15, 2009, 8 Pages. |
| Office Action dated Sep. 28, 2015 for European Patent Application No. 11702324.2, filed Jan. 18, 2011, 4 pages. |
| Office Action dated Feb. 28, 2017 for Japanese Patent Application No. 2015218066, filed on Nov. 6, 2015, 8 pages. |
| Office Action dated Sep. 9, 2014 for Japanese Patent Application No. 2011505169, filed on Apr. 15, 2009, 14 pages. |
| Okuma H., et al., “Introducing Translation Dictionary Into Phrase-based SMT,” Proceedings of Machine Translation Summit XI, Sep. 10, 2007, pp. 361-368, XP002535595. |
| Okumura A., et al., “An Automatic Speech Translation System for Travel Conversation,” Second International Conference on Human Language Technology Research, 2002, vol. 43 (1), pp. 411-412. |
| Olszewski D., et al., “Highly Directional Multi-Beam Audio Loudspeaker,” in: ISCA Archive, Interspeech 2006-ICSLP: 9th International Conference on Spoken Language Processing, Sep. 17-21, 2006 [Retrieved on Nov. 5, 2007], 1 page, Retrieved from the Internet: URL: http://www.isca-speech.org/archive/interspeech_2006/106_1239.html. |
| Papineni K., et al., “Bleu: A Method for Automatic Evaluation of Machine Translation,” IBM Research Report, Computer Science, Sep. 17, 2001, pp. 1-9, 10 pages. |
| Papineni K., et al., “Bleu: A Method for Automatic Evaluation of Machine Translation,” Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Jul. 2002, pp. 311-318. |
| Partial European Search Report for European Application No. 14190061.3, dated Mar. 2, 2015, 6 pages. |
| Paulik M., et al., “Document Driven Machine Translation Enhanced ASR,” Proceedings of European Conference on Speech Communication and Technology, Interspeech, Sep. 2005, pp. 1-4. |
| Rao S., et al., “Improving Spoken Language Translation by Automatic Disfluency Removal: Evidence from Conversational Speech Transcripts,” Machine Translation Summit XI, Copenhagen, Denmark, Oct. 9, 2007, pp. 1-5. |
| Schaaf T., “Detection of OOV Words Using Generalized Word Models and a Semantic Class Language Model,” Proceedings of Eurospeech, 2001, 4 Pages. |
| Shriberg E., et al., “Prosody-Based Automatic Segmentation of Speech into Sentences and Topics,” Speech Communication, 2000, pp. 1-31. |
| Soltau H., et al., “A One-Pass Decoder Based on Polymorphic Linguistic Context Assignment,” ASRU 01: IEEE Workshop on Automatic Speech Recognition and Understanding, 2001, pp. 214-217, 1 page. |
| Sridhar V.K.R., et al., “Combining Lexical, Syntactic and Prosodic Cues for Improved Online Dialog Act Tagging,” Computer Speech and Language, 2009, vol. 23, pp. 407-422. |
| Stolcke A., et al., “Automatic Linguistic Segmentation of Conversational Speech,” Fourth International Conference on Spoken Language, ICSLP, Philadelphia, U.S.A, Oct. 3, 1996-Oct. 6, 1996, vol. 2, pp. 1005-1008. |
| Stolcke A., et al., “Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech,” Association of Computational Linguistics, 2000, vol. 26 (3), pp. 339-373. |
| Stolcke A., “Modeling Linguistic Segment and Turn Boundaries for N-Best Rescoring of Spontaneous Speech,” In Proceedings of Eurospeech, 1997, vol. 5, pp. 1-4. |
| Stolcke A., “SRILM—An Extensible Language Modeling Toolkit,” ICSLP 2002: 7th International Conference on Spoken Language Processing, Denver, CA, USA, Sep. 16-20, 2002, pp. 901-904. |
| Stuker S., et al., “The ISL TC-STAR Spring 2006 ASR Evaluation Systems,” TC-STAR Workshop on Speech-to-Speech Translation, Jun. 19-21, 2006, pp. 139-144. |
| Suhm B., et al., “Towards Better Language Models for Spontaneous Speech,” Carnegie Mellon University, 1994, 4 pages. |
| Summons to attend oral proceedings for European Patent Application No. 09732921.3, dated Mar. 17, 2015, 5 Pages. |
| Summons to attend oral proceedings for European Patent Application No. 11702324.2, dated Aug. 4, 2016, 7 Pages. |
| TC-STAR, “Technologies and Corpora for Speech-to-Speech Translation,” 2004, 19 Pages. |
| Vogel S., et al., “The ISL Statistical Translation System for Spoken Language Translation,” Proceedings of the International Workshop on Spoken Language Translation, Kvoto, Japan, 2004, pp. 65-72. |
| Waibel A., et al., “CHIL: Computer in the Human Interaction Loop,” 5th International Workshop on Image Analysis for Multimedia Interactive Services, 2004, pp. 1-4. |
| Waibel A., et al., “Spoken Language Translation,” IEEE Signal Processing Magazine, May 2008, vol. 25 (3), pp. 70-79. |
| Ward N., et al., “Prosodic Features Which Cue Back-Channel Responses in English and Japanese,” Journal of Pragmatics, 2000, vol. 32, pp. 1177-1207. |
| Ward N., “Using Prosodic Clues to Decide When to Produce Back-channel Utterances,” Proceedings of the Fourth International Conference on Spoken Language, ICSLP 96, Philadelphia, PA, USA, Oct. 3-6, 1996, vol. 3, pp. 1728-1731. |
| “Web Data for Language Modeling,” Washington.edu [Online], Jan. 4, 2006 [Retrieved on Nov. 5, 2007], 3 pages. |
| Yamada K., et al., “A Decoder for Syntax-Based Statistical MT,” Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Jul. 2002, pp. 303-310. |
| Yarwsky D., et al., “Inducing Multilingual Text Analysis Tools via Robust Projections Across Algined Corpora,” 2001, 8 pages, Retrieved from the Internet: URL: https://dl.acm.org/doi/pdf/10.3115/1072133.1072187. |
| Zhang Y., et al., “PanDoRA: A Large-scale Two-Way Statistical Machine Translation System for Hand-held Devices,” Proceedings of MT Summit XI, Copenhagen, Denmark, Sep. 10-14, 2007, pp. 1-8. |
| Zhao B., et al., “A Log-Linear Block Transliteration Model Based on Bi-Stream HMMs,” Proceedings of NAACL HLT, Apr. 2007, pp. 364-371. |
| Number | Date | Country | |
|---|---|---|---|
| 20220092278 A1 | Mar 2022 | US |
| Number | Date | Country | |
|---|---|---|---|
| 61093898 | Sep 2008 | US | |
| 61092581 | Aug 2008 | US | |
| 61045079 | Apr 2008 | US | |
| 60854586 | Oct 2006 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15694733 | Sep 2017 | US |
| Child | 17544869 | US | |
| Parent | 14589540 | Jan 2015 | US |
| Child | 15694733 | US | |
| Parent | 13008346 | Jan 2011 | US |
| Child | 14589540 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 12689042 | Jan 2010 | US |
| Child | 13008346 | US | |
| Parent | 12424311 | Apr 2009 | US |
| Child | 12689042 | US | |
| Parent | 11925048 | Oct 2007 | US |
| Child | 12689042 | US |
