Patent Grant
7103533
(Not Applicable)
(Not Applicable)
1. Technical Field
The present invention relates to the field of speech recognition software and more particularly to a method of augmenting a language model for a speech recognition vocabulary.
2. Description of the Related Art
Speech recognition is the process by which acoustic signals, received via a microphone, are “recognized” and converted into words by a computer. These recognized words may then be used in a variety of computer software applications. For example, speech recognition may be used to input data, prepare documents and control the operation of software applications. Speech recognition systems programmed or trained to the diction and inflection of a single person can successfully recognize the vast majority of words spoken by that person.
In operation, speech recognition systems can model and classify acoustic signals to form acoustic models, which are representations of basic linguistic units referred to as phonemes. Upon receipt of the acoustic signal, the speech recognition system can analyze the acoustic signal, identify a series of acoustic models within the acoustic signal and derive a list of potential word candidates for the given series of acoustic models. Subsequently, the speech recognition system can contextually analyze the potential word candidates using a language model as a guide.
The task of the language model is to express restrictions imposed on the manner in which words can be combined to form sentences. The language model can express the likelihood of a word appearing immediately adjacent to another word or words. Language models used within speech recognition systems typically are statistical models. Examples of well-known language models suitable for use in speech recognition systems include uniform language models, finite state language models, grammar based language models, and m-gram language models. Statistically, in an m-gram language model, all word sequences are deemed possible. As a result, in an m-gram language model, the probability of a word having been uttered by a speaker can be based only upon the (m−1) immediate predecessor words. Typical m-gram language models can include the unigram (m=1), bigram (m=2) and trigram (m=3) language models.
Trigram language models are formed by constructing all possible three word permutations for each word in a large corpus of text typically referred to as a training corpus. Subsequently, the frequency of each trigram appearing in the training corpus can be observed. Unigrams, bigrams, and trigrams appearing in the training corpus can be assigned the corresponding frequency values, appropriately discounted to leave some probability space for unseen bigrams and trigrams. The resulting collection of unigrams, bigrams and trigrams and their corresponding frequency values (language model statistics) form the trigram language model.
After a speech recognition vocabulary with its associated language model statistics has been created, there will arise a need to add new words. A language model developer might need to add new words when refining the speech recognition vocabulary or when building an extension to the vocabulary. An end-user of a speech recognition system might need to add his or her own personal words to the vocabulary. Hence, the needed language model statistics must be generated for each additional new word prior to adding the additional words to the speech recognition system vocabulary. However, in order to add a new word lacking language model statistics to a speech recognition system, a new training corpus containing therein the additional words must be analyzed to develop unigrams, bigrams, trigrams, and frequency data for the additional words.
Alternatively, a language model developer might edit a speech-dictated document to include the additional words by manually inserting each additional new word in a context-relevant location of the speech-dictated document. Although this alternative approach can produce adequate results when editing a small file or a small number of files, the process can become cumbersome when developing specialized speech recognition vocabularies for specialized topics such as medicine, law and travel. Such specialized topics implicate the modification of thousands of files. Moreover, typically those files exceed in size the maximum capacity of a conventional text editor.
It is sometimes possible to obtain language model statistics for a new word from contextually-related words or classes of words in the existing speech recognition vocabulary. For example, if the word “Midway”, a reference the airport located in Chicago, Ill., is to be added to the speech recognition vocabulary, language model statistics must be developed for this additional new word. However, rather than developing completely new statistical information for the additional word, the language model statistics for “Midway” can be based upon existing language model statistics for the existing word “Heathrow” in reference to the airport located in London, England.
Present methods of adding new words to a speech recognition system by an end-user include (1) correction in a speech-dictated document or (2) analysis of user-supplied sample documents. The language model statistics generated in these two methods are limited. Adding a new word during correction will only yield one sample context for the new word. The contextual coverage attained by adding new words from sample documents depends on the amount of text present in the user-supplied documents. The number of documents typically supplied for analysis tends to be small and, therefore, leads to very few sample contexts for the new words. Finally, users might well want to simply add new words to the vocabulary in isolation without any accompanying context, especially if that user is a specialist in a field for which there are no specific language models (or topics) to purchase to extend the vocabulary.
Present methods of adding additional words to speech recognition systems based upon existing language model statistics utilize class files. Class files allow a language model developer to generate a file containing words having similar contextual properties. An example of a class file includes a list of airport names. Once created, the class file itself can be referred to in the language model in lieu of each component word contained in the class file. For example, if the class file “airport.cls” contained as constituent components, “O'Hare”, “Heathrow”, and “Midway”, all instances of those specific airport names in the language model can be substituted with a generic reference to the class file “airport.cls”. As such, the trigram “Heathrow in England” would be modified to “[airport.cls] in England”.
Developers of speech recognition vocabularies, developers of speech vocabulary extensions (e.g., specialized topics) and end-users can benefit from methods that use class files to generate statistics for new words. However, new words cannot be blindly added to classes because this will often lead to contextual inaccuracies. For example, if ‘Midway’ were added to the airport class, from the perspective of the language model, ‘Midway’ in combination with ‘in Chicago’ can remain as likely a word sequence as ‘Midway’ in combination with ‘in England’—an absurdity. Thus, there has arisen a need for a better way to ensure contextual accuracy when adding additional new vocabulary words to a speech recognition system.
The present invention is a method of generating language model statistics for a new word added to a language model incorporating at least one class file containing contextually related words. The method can include the following steps: First, language model statistics can be computed based on references to at least one incorporated class file. Second, a new word can be substituted for each reference to a selected class file. Additionally, the language model statistics can be re-computed based on the new word having been substituted for the reference. Third, the re-computed language model statistics can be displayed in a user interface and modifications can be accepted to the re-computed language model statistics through the user interface. Fourth, the language model statistics can be further re-computed based on the modifications. In consequence, the language model statistics are re-computed for the new word without introducing contextual inaccuracies in the language model.
Importantly, if no modifications are accepted in the third step, the new word can be added to the selected class file. As such, the new word can inherit existing context-based language model statistics associated with the selected class file. Also, each of steps one through four can be repeated for each class file associated with the new word. Notably, in one aspect of the invention, the displaying step can include combining the computed language model statistics into a single line of source material; and, displaying the single line of source material in the user interface.
In yet another aspect of the invention, the language model statistics can include m-grams. The m-grams can be unigrams, bigrams and trigrams. Additionally, the language model statistics further can include a frequency corresponding to each m-gram. As such, the method can further include associating a relative frequency with each contextually related word in the selected class file. Moreover, the language model statistics can be computed and re-computed according to the user-specified relative frequencies. Finally, the displaying step can include for each contextually related word in the selected class file, combining all re-computed m-grams and associated relative frequencies into a line of source material; and, displaying the line of source of material in a user interface.
In one particular aspect of the present invention, a method of generating language model statistics for a new word based on existing class files and without introducing contextual inaccuracies can include the following steps. First, for each relevant class, m-grams are generated in which the class reference is replaced by the new word. Second, the generated m-grams can be displayed in a user interface. Third, modifications to the m-grams can be accepted. If modifications are made, the user-edited m-grams are added to the language model. If no modifications are made, the new word is added to the class file so the new word inherits the context of all m-grams contextually associated in the language model with the class. Thus, with or without modifications of the generated m-grams, the new word receives a richer context in the language model than it would have received using current methods.
There are presently shown in the drawings embodiments of which are presently preferred, it being understood, however, that the invention is not so limited to the precise arrangements and instrumentalities shown, wherein:
The method of the invention, as described in the foregoing specification, obtains language model statistics for a new word from the existing speech vocabulary classes while preserving contextual accuracy. In particular, a class file containing therein contextually related words can be established for extending the language model. Preferably, a relative frequency can be associated with each of the contextually related words. A user is presented the set of trigrams and bigrams containing a reference to the class file in which those references are substituted with the new word. User modifications and deletions to these m-grams are accepted to ensure contextual accuracy. Finally, the language model statistics are updated to include the modified bigrams and trigrams specified by the user. If the new word is found to be contextually identical to the other class words, the new word is added to the class file and the user is given the opportunity to update the relative frequencies of the contextually related words contained in the class file.
As shown in
In a preferred embodiment which shall be discussed herein, the operating system 9 is one of the Windows family of operating systems, such as Windows2000, Windows NT, Windows 98 which are available from Microsoft Corporation of Redmond, Wash. However, the system is not limited in this regard, and the invention can also be used with any other type of computer operating system. The system as disclosed herein can be implemented by a programmer, using commercially available development tools for the operating systems described above.
In operation, audio signals representative of sound received in microphone 7 are processed within computer system 1 using conventional computer audio circuitry so as to be made available to the operating system 9 in digitized form. The audio signals received by the computer system 1 are conventionally provided to the speech recognition engine 10 via the computer operating system 9 in order to perform speech recognition functions. As in conventional speech recognition systems, the audio signals are processed by the speech recognition engine 10 to identify words spoken by a user into microphone 7.
To assist in the search process, the process in search block 16 can enlist the assistance of acoustic models 19, lexical models 18, language models 17 and vocabulary 22. One skilled in the art will recognize that, while the present invention employs trigrams in its preferred language model, other language model forms, for instance bigrams, can be substituted for the preferred language model. Subsequently, the speech recognizer 11 returns word matches and confidence values for each match which can be used to determine recognition accuracy.
In step 40, the list of classes in the speech recognition vocabulary appears in a user interface. In step 42, a user input can be accepted which specifies which classes are contextually relevant to the word that is being added. For example, a vocabulary for ‘Travel’ might contain an airport class, an airlines class, a country class, a city class, and so on. If the user is adding ‘Midway’ to the vocabulary, he or she would select both the airport and the airline classes as being relevant.
In step 44, if all classes identified by the user have been processed, the method can branch to termination. Otherwise, in step 46, the next user-identified class and its language model statistics are retrieved from computer memory. For example, for the exemplary class file “airport.cls” containing the names of airports, each unigram, bigram, and trigram containing a reference to the class file “airport.cls” can be retrieved. Exemplary trigrams each containing a reference to the class file “airport.cls” are shown in
In step 48, the m-grams identified in step 46 are modified to include the new word in place of the class name. For example, each unigram, bigram and trigram containing a reference to the class file ‘airport.cls’ is modified so each reference to ‘airport.cls’ is replaced with the new word that is being added to the vocabulary. Exemplary modified trigrams for the example in
In step 52, if modifications to the m-grams are made by the user, the process branches to step 56 in which the edited m-grams and their relative frequencies are used to update the language model. Specifically, the user can edit or delete any of the bigrams, trigrams, or source materials that appear to be incorrect or nonsensical. For example, as shown in
By comparison, if no modifications are apparent in step 52, then the new word is contextually identical to the words in the class, which makes it reasonable to add the new word to the class file in step 54. In step 58 of
In step 60 a user input can be accepted which specifies a frequency value for each word in the class file. The frequency value can be a relative frequency value with regard to the other words in the class file. For example, if the “airport.cls” class file contains the words “O'Hare”, “LaGuardia”, “Kennedy”, “Dallas”, and “Atlanta”, then the user can specify frequency values such as “high”, “medium”, or “low”. Consequently, a word associated with a frequency value of “high” has a higher frequency value in relation to another word with a frequency value of either “medium” or “low”. Similarly, a word associated with a frequency of “medium” has a higher frequency than a word associated with the frequency of “low”. Alternatively, a numbering system with 1 representing the lowest relative frequency and 10 representing the highest relative frequency can be used.
Also, the user specified relative frequency of each word in the class file can be translated into an actual frequency value that can be used by the speech recognition system. For example, a relative frequency value of “high” can be translated into an actual frequency number. It should be appreciated that the translation can be performed through any suitable means including various algorithms or through the use of a reference table. Following step 60 or step 56, in step 62 the class is deselected to show that it has been processed. The process returns to step 44 where the user determines whether there are any more classes that he or she needs to process for the new word.
While the foregoing specification illustrates and describes the preferred embodiments of this invention, it is to be understood that the invention is not limited to the precise construction herein disclosed. The invention can be embodied in other specific forms without departing from the spirit or essential attributes. Accordingly, reference should be made to the following claims, rather than to the foregoing specification, as indicating the scope of the invention.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5027406 | Roberts et al. | Jun 1991 | A |
| 5680511 | Baker et al. | Oct 1997 | A |
| 5835893 | Ushioda | Nov 1998 | A |
| 5970460 | Bunce et al. | Oct 1999 | A |
| 6188976 | Ramaswamy et al. | Feb 2001 | B1 |
| 6314399 | Deligne et al. | Nov 2001 | B1 |
| 6501833 | Phillips et al. | Dec 2002 | B1 |
| 6532444 | Weber | Mar 2003 | B1 |
| 6606597 | Ringger et al. | Aug 2003 | B1 |
| 6691088 | Blasig | Feb 2004 | B1 |
| 6697769 | Goodman et al. | Feb 2004 | B1 |
| 6801893 | Backfried et al. | Oct 2004 | B1 |
| Number | Date | Country | |
|---|---|---|---|
| 20020116194 A1 | Aug 2002 | US |
