Patent Application
20080071533
For a more complete understanding of the present invention, reference is now made to the following descriptions taken in conjunction with the accompanying drawing, in which:
Input 401 is operative to receive the desired semantic task categories along with the brief category descriptions and category task labels from an application designer, and could also be used for communicating with thesaurus 102 (
Returning to
Note that in a true NLSAs system, the concept of “dialog state” or “prompt state” can be confusing since all available tasks are typically available for selection at all times. A user can ask for the account balance, even if the prompt is asking for a check number.
Table 1, which can be generated by manual process 101, presents an example of the input requirements to generate the SLM for the “Account Payment” prompt.
The semantic category description (as shown in Table 1) is used to extract certain pre-filler Part-of-Speech (POS) patterns which are extracted by process 105 and stored as pattern pools.
Table 2 presents some POS patterns extracted to represent the pre-filler words that can be uttered by the user for that particular semantic category. In Table 2, prp stands for preposition, nn (or NN) stands for noun, vb is verb, prn is pronoun, etc.
After the manual extraction of POS patterns (for example) from a small number (for example 20) of semantic category descriptions, it is possible to observe that these selected manual pre-filler POS patterns and their generalizations cover most of the POS patterns present in the remaining non-analyzed semantic category descriptions. Hence, the system can use the identified pool of patterns for all the remaining semantic category descriptions to keep the manual labor to a minimum. The filtering processes, such as lexico-semantic filtering 104 and WWW filtering 109, will then handle non-compatibility of the generalized POS patterns with certain semantic categories.
The POS patterns from the pool (process 105) are then searched for in a large number, say 1126, POS tagged conversations determined by process 106 using, for example, the SwitchBoard-1 conversations from the TreeBank-3 corpus obtained from the Linguistic Data Consortium (LDC) at the University of Pennsylvania to extract spontaneous/conversational speech style pre-filler phrases.
Three different pre-filler word sequence extraction methods are used by process 106. First, there are the “pure POS pattern pre-filler words” which are identified pre-filler words that adhere to the POS patterns e.g., “I want credit” for the pattern “PRP (pronoun) VB (verb) NN (noun)”. Second, there are “POS pre-filler words with gaps” which are identified pre-filler words that comply with the POS patterns but with some gaps between POS tags in the pattern e.g., “I want to get another brand” for the pattern “PRP VB VB NN”. Third, there are “POS pattern pre-filler words with additional peripheral words” which can be identified pre-filler words for “pure POS pattern pre-filler words” or “POS pattern pre-filler words with gaps” but with some additional peripheral words in the beginning and end of the POS pattern e.g., “Could I have something” for the pattern “PRP VB NN”. The “NN” words are removed from all the identified pre-filler words and the “PRP” words are replaced with appropriate personal or possessive pronouns depending on the POS pattern e.g., “PRP” words for the pattern “PRP VB NN” are replaced by “I” and “we”.
In parallel with the above pre-filler word sequence generation mechanism (processes 105, 106 generate the pre-filler phrases stored in process 107), the semantic category descriptions are also used to generate a set of semantic category synonyms. For each semantic category, the system uses its description (from Table 1) to extract a skeletal set of content words. A thesaurus is then used (process 102) to find a set of alternatives closely related to these sets of content words.
By way of example, Table 3 presents the word alternatives extracted for the category “cellular_phone”.
The output from the thesaurus contains good alternatives for the content words, however the output also contains irrelevant words e.g., for the category “arrange_a_payment”, the alternatives are found by combining the closely related words for “arrange” and “payment” and this leads to some noisy alternatives like “adapt deposit” or “organize fee”.
WordNet, such as described by C. Fellbaum, in the MIT Press, 1998, of which is incorporated herein by reference, is a lexico-semantic database containing open class words like nouns, verbs, adverbs and adjectives grouped into synonym sets. Each synonym set or WordNet synset represents a particular lexical concept. WordNet also defines various relationships that exists between lexical concepts using WordNet semantic relations. D. Moldovan and A. Novischi in their paper entitled, “Lexical Chains For Question Answering,” published in Proceedings of Coling, 2002 (hereinafter “Moldovan”), of which is incorporated herein by reference, presents a methodology for finding topically related words by increasing the connectivity between WordNet synsets (synonym set for each concept present in WordNet) using the information from WordNet glosses (definition present in WordNet for each synset).
Thus, the system determines, via process 103, if a pair of words are closely related by not only looking at the WordNet synsets but also by finding lexical paths between the word pair using the WordNet synsets and glosses. To remove the noisy alternatives, process 103 determines (for example, using the procedure outlined in the above-identified Moldovan paper) a connection between the words present and alternatives therefore. For example, the lexical chain between the words “adapt” and “deposit” has a low confidence score, while the word pair “prepare” and “amount” has a relatively higher confidence score. Hence, an alternative is considered to be valid and is added to the list if the lexical chain confidence score for its content words is greater than a threshold value. In summary, after the completion of these steps, a set of possible pre-filler and content words representing each IVR prompt is collected at process 107 and 103, respectively.
The system then attempts, at process 108, to combine each pre-filler phrase with every content word phrase to form a set of utterance alternatives. This involves combining each identified pre-filler word sequence collected at process 107 with all the content word sequences collected at process 103. For example, if “n” pre-filler word sequences are collected at process 107 and “m” content word sequences are collected at process 103 then a total of “n*m” utterance alternatives are formed. These word alternatives are then filtered using process 104 to remove those (pre-filler word sequence+content word sequence) combinations that are incompatible. For example, for the pre-filler word sequence “Check my” and the content word sequence “account balance”, their combination makes sense. However, the pre-filler word sequence “Check my” combined with the content word sequence “operator” would be filtered. Hence, a particular pre-filler phrase combination with a content word phrase is allowed only if a lexical chain is determined between the pre-filler phrase verb and the noun/verb in the content word phrase (if it is a noun phrase/verb phrase) with a confidence score greater than a defined threshold.
The lexical chain confidence score for a word pair is usually determined by the presence of one word in the WordNet gloss of the other word and vice-versa (procedure outlined in the above-identified Moldovan paper). The lengthier the chain, i.e., extending to the glosses and reverse-glosses of the hyponyms or hypernyms for the word pair, the smaller is the lexical chain confidence score (procedure outlined in the above-identified Moldovan paper). The complete set of sentences thus formed are then filtered using a statistical validation mechanism, such as WWW filtering process 109, which can, for example, use a search engine (such as Google) to search for the new sentences as one cohesive unit on the web. News groups can be used in this context since they are close to conversational style text. If the count (number of web page links) returned by the search engine exceeds a defined threshold then the sentence is added via process 110 to the data set later used to build the SLM. The count provided by the web for a particular alternative is also used to represent its probability distribution in the SLM data set which will be used later to build the SLM.
One method of evaluating the SLMs, is to use them as language models for an ASR and compare the WER/SemER produced by such an ASR for live user utterances against an ASR using the manually generated CFG grammars. To evaluate the SemER for the utterances transcribed by an ASR loaded with SLMs, a WordNet lexical chain based semantic categorizer is used to convert the ASR transcriptions into valid semantic categories. These extracted semantic categories are then compared with the actual user utterance semantic categories to obtain a semantic error rate.
Process 204 determines if all transcriptions have been valued. If so, process 205 creates a baseline result in order to test the proposed SLM generated transcriptions.
In one example, 20804 utterances were collected for 55 prompts. A total of 23 CFGs/SLMs are needed to cover all of the 55 prompts and on average, each prompt elicits responses with 10.09 different semantic categories. The baseline WER and SemER results for the 20804 utterance set in the example, are produced by, for example, a Nuance 8.5 v commercial recognizer and a SONIC system such as described by Bryan Pellom, in the published paper entitled SONIC: The University of Colorado Continuous Speech Recognizer, tech report #TR-CSLR-2001-01, which is incorporated herein by reference. SONIC was trained for the telephone transcription task using 160 CallHome and 4826 Switchboard-1 conversation sides.
Table 4 presents the transcription WER results obtained for the various tests performed on our 20804 utterance test set.
Table 5 presents the Semantic Error Rate (SemER) results obtained for the transcriptions in Table 4.
Each utterance transcription generated by the various systems presented in Table 4 is classified into one or more semantic categories using the semantic categorization technique discussed above. An absolute LCS threshold value of 65.0 was used with an LCS difference threshold value of 5.0, and a LCS difference decaying factor of 0.1. These values were derived by using the manual transcriptions of 20804 utterances as a development set. This resulted in a best SemER of 4.6%. A “NO-MATCH” category is used when an utterance does not map to any other category.
Process 301 accepts information from process 300 and establishes POS patterns for each prompt. Process 302 then expands the POS patterns into POS pre-filler phrases for spontaneous speech conversations, as obtained from process 303. Process 304 gathers the pre-filler phrases.
Process 310 also accepts information from process 300 and adds a skeletal set of content words so that process 311, working in conjunction with a thesaurus (process 312) can expand (filter) the skeletal set of content words so that process 313, working in conjunction with lexico-semantic resources (process 314) (another filter) can form content work phrases.
Process 320 then combines the pre-filler phrases from process 304 with the content word phrases from process 313 to form alternative possible utterances. Process 306 then eliminates from the alternative utterances these utterances not achieving a high enough score. Process 322 then filters the remaining utterances against normally used sentences, if desired.
In Table 4, MisCat errors are due to mismatches between the semantic category proposed by the transcription and the actual utterance semantic category. InCFG errors are due to the transcription proposing a semantic category while the utterance's actual semantic category is a NO-MATCH. OutCFG errors are due to the transcription proposing a NO-MATCH while the utterance actually has a valid semantic category. Ins errors are due to the insertion of a semantic category by the transcription while the utterance's actual semantic category list does not contain such a semantic category. Del errors are due to the deletion of a semantic category present in the utterance's actual semantic category list while the semantic category is missing in transcription's semantic category list, Total Error (%) is the sum of all the five (5) different error counts divided by the total number of reference semantic categories. Total Correct(%) is 100−MisCat(%)−InCFG(%)−OutCFG(%)−Del(%).
Table 6 presents the various errors possible due to the variations in the number of categories proposed by the transcription and the number of categories present in the reference list.
For SLM evaluation, the WordNet lexical chain based semantic categorizer is used to classify transcriptions from the SLM-loaded ASR into valid semantic categories. The SLM-loaded ASR response semantic categories are then compared against the manually labeled utterance semantic categories. By using the automatically generated SLMs, the manual labor involved in the IVR application development is reduced while the semantic error rate is comparable with the ASR loaded with manually generated grammars/SLMs.
Although the present invention and its advantages have been described in detail, it should be understood that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the invention as defined by the appended claims. Moreover, the scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification. As one of ordinary skill in the art will readily appreciate from the disclosure of the present invention, processes, machines, manufacture, compositions of matter, means, methods, or steps, presently existing or later to be developed that perform substantially the same function or achieve substantially the same result as the corresponding embodiments described herein may be utilized according to the present invention. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or steps.
