Patent Grant
9183204
The present invention relates to a system to determine index terms to generate grammar rules for speech recognition.
Document retrieval systems, such as search engines, have been the subject of considerable research and development. The sophistication of speech recognition systems has also significantly advanced. One of the difficulties facing document retrieval systems is providing a process which limits or obviates the retrieval of irrelevant documents in response to a user's query. This problem however proves even more difficult if it is desired to provide a speech recognition based interface for retrieval of documents. Speech recognition systems have previously been used for limited retrieval applications involving a very structured and limited data set for information retrieval. It is desired to provide a useful alternative to existing information retrieval systems and the steps they execute.
In accordance with the present invention there is provided a method of generating index terms for documents, including:
The present invention also provides an information retrieval method, including:
The present invention also provides an information retrieval system, including:
Advantageously, said index terms may be processed to provide grammar rules for a speech recognition engine.
A preferred embodiment of the present invention is hereinafter described, by way of example only, with reference to the accompanying drawing, wherein:
An information retrieval system, as shown in
The information retrieval system includes a number of software modules 2 to 20, shown in
In order for the system to support information retrieval, a series of tasks related to data preparation are initially performed. The first task is to generate index terms or domain concepts from the stored documents of the document database 1 which allow the documents to be matched to the voice input query terms. This process begins by processing the text of the documents by a natural language parser 3. However, if the documents are in some form other than plain text, such as a rich text or word processing format (e.g., HTML, XML, SGML, RTF, Microsoft Word™), they are first processed by a text filter 2 to remove all the formatting or other extraneous content prior to processing by the natural language parser 3.
Given a document, the natural language parser 3 uses the structure and linguistic patterns of English text to extract linguistically important words and phrases from the sentences in the document. These words and phrases are referred to as key linguistic terms of the document. The parser 3 first identifies those “chunks” that represent the local linguistic structures in a sentence. It then selects from each chunk the key linguistic terms that are likely to carry the key information in that sentence. For example, given the sentence “NASA wants to send an orbiting surveyor to Mars”, the parser 3 identifies the following chunks: “NASA”, “want to send”, “an orbiting surveyor”, “to Mars”. From them, the parser 3 would extract the following words and phrases: “NASA”, “orbiting surveyor”, and “Mars”.
To recognise sentence chunks, the parser utilises a data structure called a key-centred phrase structure frame, such as: NP→det adj*noun, where NP refers to a noun phrase having a determiner (det), adjective (adj) and noun.
The category preceded by an asterisk—“noun” in this example—is the key category that will match the content word in a chunk. Once a word with the category “noun” is identified in a sentence, this frame is attached to that word (the word is called an anchor word in the following discussion) and the key category in the frame is aligned with the word. Next, the frame is instantiated by using a tolerant bidirection pattern matching process. During the pattern matching process, the parser collects the words towards the two ends of the sentence whose categories match those in the frame. When no contiguous words of the same category remain, the matching process begins again with the category of the adjacent words, if any. A matched word is stored in a slot associated with the category in a frame. The matching process also creates a four-element tuple <S, E, CL, P> for each instantiated frame, where:
The frames are classified into different classes according to when they will be applied. For example, any frames whose right hand side (RHS) contains only terminal categories such as NP above is used in a bottom-up matching process and will be applied first. Any frames whose RHS consists of merely non-terminals will be used in a top-down analysis process and will be applied in a later stage. This frame invocation scheme allows the frame instantiation process described above to proceed in an orderly manner.
When all the frames have been instantiated, a frame relation graph is generated. An example of such a graph is given in
In a frame relation graph, a start node, one or more intermediate nodes, and an end node form a path that represents one particular way of linking some of the frames. Two rules govern the formation of these paths: (1) only two nodes representing non-overlapped frames can appear in the same path; (2) only two nodes representing two adjacent instantiated frames can be linked by an arrow. The parser then parses again each of the paths in the graph. The parsing method used here is similar to the frame instantiation process described earlier, and combines both bottom-up and top-down methods, with the difference that the lowest units in the parsing process are now the nodes in the path (i.e. the chunks recognised so far).
After the parallel parsing process, if one path gives a complete parse, this is the final path result produced by the parser 3. If more than one path gives a complete parse, the final result is selected using the following rules. The totals of the CL values of the nodes are calculated. If the total for one path is greater than any other, then that path is chosen. If not, then the same rule is applied to the totals of the P values of the nodes in the path. If no path parses completely, the above two rules are used and the path with the lowest number of nodes is selected. If there is more than one path selected (very rare), a path will be picked up arbitrarily when no other knowledge (e.g. contextual or domain knowledge) is available to perform further comparison.
The nodes in the final path correspond to the local structures or sentence chunks that contain potentially useful key words and phrases. Two more steps are required to extract these key linguistic terms: (1) all determiners such as “a”, “this”, “all”, “some”, etc are removed, and (2) for a phrase containing more than two words, some subphrases contained in that phrase are generated. For example, from the phrase “U.S. President Bill Clinton”, the parser would generate two extra phrases “President Bill Clinton” and “Bill Clinton”. These key linguistic terms are then stored with the source document for further processing.
The feature extractor 4 performs two tasks: feature extraction for individual documents and the determination of word importance within a document set. The extraction system 200 uses the output produced by the parser 3 with the output generated by feature extractor 4 to build index terms 5.
The feature extraction process extracts words and phrases from a document that is most descriptive of that document. These words and phrases form the initial feature set of the document. The process is similar to that described in J. D. Cohen, “Highlights: Language and Domain Independent Automatic Indexing terms for Abstracting”, Journal of the American Society for Information Science, 46(3): 162: 174, 1995 for generating highlights or abstracts of documents retrieved by an information retrieval system.
The feature extraction is based on n-grams. N-grams are sequences of characters of length n. Every document is represented as a sequence of characters or a vector of characters (referred to as document-sequence vector). Each document-sequence vector is processed to extract n-grams and their frequencies (number of occurrences) for that document. For example, the sequence of characters “to build” will give rise to the following 5-grams “to bu”, “o bui”, “buil”, “build”.
In order to determine the words and phrase that describe a document or a group of documents, the following is executed. First, the distribution of the n-grams over the document space is computed by counting the occurrence of n-grams in the documents. Each n-gram is assigned a score per document that indicates how novel or unique it is for the document. This novelty score is based on the probability of the occurrence of the n-gram in the document and the probability of occurrence elsewhere and is calculated using the following formula:
where Ψij is the novelty score of the n-gram i in document j, pij is the probability of the occurrence of n-gram i in document j, qij is the probability of occurrence of n-gram i elsewhere in the document space, tij is the probability of occurrence of n-gram i in the whole document space, Sj is the total count of n-grams in document j, and S is ΣSj.
Next, the novelty score of each n-gram is apportioned across the characters in the n-gram. For example, the apportioning could be so that the entire score is allocated to the middle character, and the other characters are assigned a score of zero. This apportioning allows each character in the sequence (hence each entry in the document-sequence vector) to be assigned a weight. Finally, these weights are used to compute a score for each word or phrase based on their component characters and their scores. These scores combined, if necessary, with language-dependent analysis, such as stemming, may be used to filter out non-essential features, if desired.
Thus, the output of the feature extractor is a set of terms (words or phrases) from the document, and a score indicating how well it describes the document, i.e., how close it is to the topic of the document. This also means that the same words, e.g., vector, may have different scores: a higher score in a document about vector analysis and a lower score in a document that uses vector analysis for search engines. This score is then used during information retrieval so that the query “vector” would yield both these documents, but the document about vector analysis would be ranked higher.
Since the feature extraction process is based on n-grams, there is no necessity for language-dependent pre-processing such as stemming and removal of function words. Hence, this extraction process is language-independent. It is also tolerant of spelling and typing errors since a single error in the spelling of a long word would still yield some n-grams that are same as those from the correctly spelt word. In addition, the same feature extraction technique can be used to extract words or phrases or sentences or even paragraphs (for large documents) since the fundamental unit for determining novelty is not words but character sequences. Further, the character sequences need not even be text. Hence, with modifications, the same technique may be used to pick out novel regions within an image or audio track.
After generating a set of terms with their weights indicating how well they describe the document, a feature set of the document is created as follows:
The feature set of a document is used in the document retrieval process.
The feature extractor 4 determines a score for each term in a particular document. In order to build grammars and to retrieve documents meaningfully, however, it is often necessary to know the overall importance of a word within the whole document set. For instance, if all the documents within a set are about Telstra, then the word “Telstra” is less important in that document set than, say, another word, such as “Mobilenet”. The word importance module assigns a score to each word in the index set (determined by the index generation module) based on its importance, i.e., its ability to discriminate. This ability of a word to discriminate depends on how many documents a word appears in (referred to as DF), and the frequency of that word (referred to as TF) in each of those documents. Those words that appear frequently within a few documents are more discriminating than those that appear in most of the documents infrequently. Traditionally, in information retrieval, this reasoning is captured using the TFs and DF of each word to arrive at an importance value, as described in Salton, The SMART Retrieval System—Experiments in Automatic Document Processing, Prentice-Hall, New Jersey, 1971 (“Salton”).
In the extraction system 200, the discrimination ability is determined based on the notion that when a word is discriminating and it is removed from the feature space, then the average similarity between documents in the repository increases. Thus, by determining average similarity between documents with and without a word, it is possible to determine its discriminating ability. The average similarity of a document set is determined by summing the similarities between each document and the centroid (where the centroid is the average of the word frequency vectors of all the documents in the set). The similarity is computed by using a cosine coefficient, as discussed in Salton. The input to the word importance analysis process is the word frequency vector for each document, and the output is a score for each word indicating its importance. This score is referred to as the importance score of the word.
The natural language parser 3 generates key linguistic terms that represent good candidates for index terms based on sentence structures. However, due to the nature of the syntactic analytic method used, the parser 3 will also select words and phrases that may be linguistically important but contribute little to search purpose, such as the term “hour” in the sentence “The cabinet meeting lasted about an hour.” On the other hand, the feature extractor 4 is able to identify terms that are most descriptive of a document but these terms are in general limited to one or two words. The extraction system 200 uses the features of a document to remove those key linguistic terms of the same document that have little use for search purposes. This is achieved by removing those key linguistic terms of a document determined by the parser 3 that do not contain any terms from the set of features generated by the feature extractor 4 for that document. The remaining key linguistic terms form the set of index terms or domain concepts 5 of the document that will be used in the information retrieval process. By adding the importance scores of all the words in an index term together, the importance score of the index term can also be determined.
The index terms or domain concepts 5 thus identified are used in two ways. In addition to providing index terms for locating documents matching search terms, they are also used to generate domain-specific grammar rules 8 for the speech recognition engine 12. This is performed by a bi-gram language module 6 to produce word pair grammar rules. The rules 8 constrain the speech recognition grammar and thereby enhance the document matching performance of the system. The domain-specific grammar rules 8 augment the manually coded sentence structure grammar rules which remain relatively constant across different application domains. Live voice data transcripts 11 received during use of the interface are also processed in order to supplement the initial grammar rules and enhance their coverage.
The bi-gram language module 6 takes as its inputs a list of index terms 5 plus a list of transcripts 11 that have been manually entered. The index terms 5 represent the phrases that are likely to be spoken by the user based upon the documents that can be returned. The list of transcripts represents spoken phrases that have been input in the recent past, usually the last two or three days.
As shown earlier, each of the index terms 5 has associated with it a weight that is an estimation of the overall importance of the term within a document set. Each transcript in the list of transcripts 11 also has a weight assigned to it. The weight given to a transcript in the list of transcripts is equal to or greater than the greatest weight generated by the feature extractor, multiplied by the number of times the phrase has been observed. This will bias the probabilities in favour of those word pairs that have been observed on earlier days, over those that have not been observed. Alternatively, a phrase may occur more than once in the transcript list 11 if it has been recorded more than once, and each entry is given the same weight.
The index terms 5 and the list of transcripts 11 are added together to create a single list of phrases. This list is compiled as bi-gram module input data 7. This list is considered to be a list of noun phrases, even though it contains transcriptions that may be more than noun phrases. A bi-gram language model is then generated from this combined list by the bi-gram language module 6. A bi-gram language model is a language model that states that one word can follow another word with a specific probability. A word pair language model is a bi-gram language model where the probability of one word following another can be either 0 or 1.
Each entry in the bi-gram module input data 7 can be considered to be a sequence of words as follows:
For the purposes of the bi-gram language model, each observation is considered to also include a start symbol (α) and an end symbol (Ω):
The length of each observation may differ between observations.
A two dimensional associative array is created to count the transitions from one word to another. Advantageously this may be a sparse array such that only transitions that have been observed are stored in the array.
The entry for each symbol transition Xa Xb is then incremented by the weight attached to the index term. For instance if the first phrase in the phrase list (XX) was
It would create the following entries in the associative array.
If the first phrase in the combined list was
If both entries occurred in the combined list the entries in the associative array would be
A user, however, may speak noun phrases that are shorter than those in the index terms 5. For instance, although the index terms 5 may contain an index term such as “the prime minister john howard”, users may simply say “john howard” or “the prime minister.”
Additional index terms are created that represent each word in the index term, spoken in isolation. Each one of these index terms will have the same weight as the index term it is generated from. For instance for the phrase
The following additional counts are added to the associative array.
If the combined list contained only the two previous entries in it, the associative array would be as shown below.
It can be shown that this also enables sub strings of any length from the original two index terms, for instance “community of umagico” is also a valid phrase according to the bigram model.
This bi-gram language model is then created into a set of context free rules that can be combined with other predetermined context free rules. A bi-gram transition of the form Xa Xb can be converted to a context free grammar rule of the form
A context free grammar is a set of rules, consisting of symbols. These symbols can be either words such as “community” or nonterminals that can be expanded into other symbols.
In this notation, upper case symbols represent non-terminals that need to be expanded, and lower case symbols represent words that can be spoken. A context free grammar rule thus has on the left side a nonterminal to be expanded, and a right hand side which contains a set of symbols the left hand side can be replaced with. In the notation used above, Xa represents the Non-Terminal on the left hand side. The right hand side of the rule is “xb Xb”. In addition, the rule has a probability p. The sum of the probabilities of all rules with the same left hand side must sum to 1.
When a bigram transition is created using the notation above, a nonterminal is assigned to each symbol. In the example above, the nonterminal Xb represents all of the symbols that can occur after the symbol xb is generated. In the general case, each nonterminal in the bigram or word pair grammar would have a unique prefix to ensure that the nonterminal symbol is unique. The non-terminals can also be considered to be states of a state machine. For instance the rule above defines that while in the state Xa, if a xb symbol is encountered then the state machine represented by the grammar transistions to the state Xb.
The probability of each bi-gram transition is estimated by dividing the counts associated with the bi-gram transition by the sum of all counts attached to the same non-terminal. For instance, in the example above, there are two possible words that can follow the word “community”. These words are “of”, or the termination symbol. Therefore, assuming the terminal symbol could be represented as a space (“ ”) this part of the grammar expressed in Nuance™ format would be
It can be seen that the sum of these probabilities is equal to one. The non-terminals here are prefixed with the string “NT1” to provide a unique non-terminal name.
In a number of grammar formats, empty grammar expressions are prohibited and thus the context free grammar generated at this point should have those rules containing empty right hand sides removed, without altering the phrases that can be generated, or their probabilities. This is done by considering a non-terminal with rules with empty sides on it as optional. For instance the context free grammar generated by the bi-gram associative array above would be
Where a rule doesn't contain a right hand side, e.g. NT1Community→3.54959, two copies are made of every rule that references this non-terminal, such that the non-terminal is either missing or in existence. For instance
Rule counts are modified so that the sum of the two counts remains the same, but the rule with the missing non-terminal has its count set to the original rule, multiplied by the probability of the empty rule, and the rule with non-terminal remaining has its count set to the original rule, multiplied by one minus the probability of the empty rule. For instance,
The empty rule (EG “NT1Community→3.54959”) is then removed. The remaining rules attached to the non terminal remain unaffected (e.g., “NT1Community→of NT1Of 1.371258”). This process continues until there are no more rules with empty right hand sides remaining. The resulting grammar is then converted into a format that can be loaded in by the speech recognition engine 12. Probabilities are calculated, and the probabilities that might otherwise be rounded down to zero are rounded up to a minimum probability. The given example in Nuance™ format would then be
This context free grammar can then be used by the generic grammar 8 that uses this grammar fragment as a noun phrase in a more comprehensive context free grammar. The exact structure of this grammar depends upon the question being asked, and should be modified based upon transcripts 11 either manually or automatically. Where the user is being asked to state a news topic they are interested, eg in a news retrieval service, a suitable grammar might be (in Nuance™ format) as shown in Appendix A.
The probability of a word pair is obtained by using the phrases in the transcripts and/or the generated index terms. A similar technique can be implemented if no transcripts 11 are available. In this scenario, the bi-gram grammar is built from the index terms 5 alone. It may be advantageous in this scenario not to use the calculated rule probabilities, but instead to set them to be either 0 or 1. The reason for this is that the rule probabilities are calculated using the output texts, rather than examples of phrases that are actually being spoken by users. There is, however, likely to be some correlation between the distribution of word pairs in the output text and the input phrases due to the fact that both represent examples of some small subset of spoken language related to the topic contained in the described text. When the bigram probabilities are generated from both the generated index terms and the transcripts of voice input, this biases the probabilities in favour of those word pairs already observed. In addition, the probabilities also bias the recognised phrases in favour of the more commonly occurring terms in the index terms 5 or in the transcripts 11. The decision of whether to use probabilities or not in the grammar presented to the speech recognition engine depends on the particular application, as is the weighting of counts of transcripts versus generated index terms.
The above tasks, including parsing, feature extraction, word importance determination, and bi-gram language module generation, are all executed by the ‘offline’ extraction system 200 which is applied to stored documents of the database 1 prior to retrieval. To retrieve stored documents, a user issues a spoken voice command to a microphone of the retrieval system user interface 10 in order to locate documents based on some search criteria. The user interface 10 may be a standard telephone handset that is connected by a telecommunications network to an IVR that includes the speech recognition engine 12. Alternatively, the user interface 10 may include pages of a web site served to a user that includes code able to capture sound data generated using a microphone connected to the sound card of the user's computer system, which is connected to the Internet and has received the code of the site. The speech recognition engine 12 interprets the incoming sound data as a series of words and generates a set of n-best interpretations of the query, each having a confidence score. This may be performed using a commercially available speech engine such as Nuance™ 7.0 by Nuance Communications, Inc. (http://www.nuance.com).
Many speech recognition engines such as Nuance™ allow the output of n-best interpretations of the spoken user query with some confidence scores. For example, given the voice input “is there any water in Mars”, the speech recognition engine 12 might return several top interpretations with their confidence scores as follows:
To derive useful search terms from the set of interpretations, the following steps are executed: (1) each of the interpretations is parsed by a user term selector 14, using an instance of the natural language parser 3, to determine potentially useful search terms; (2) these search terms are ranked by combining the parsing results and the confidence scores; (3) the final set of search terms is selected according to rank.
For the above example, the parser 3 would return the following search terms that are potentially useful for each of the four interpretations:
The user search term selector 14 integrates the speech recognition confidence scores and natural language analysis results to select the final set of search terms to be used for document retrieval. The following steps are executed:
The search terms identified can now be used to select the appropriate index terms. This is the task of Query Clarification module 16. The process involves the use of Longest Common Substring (LCS), a method for measuring the similarity between two strings, as discussed in Hunt & Szymanski, “A fast algorithm for computing longest common substring”, Communication of ACM, 20, 5, 350-353, 1977. The process also generates various weights for document retrieval.
The LCS can be defined as follows. Let both U=u1, u2, . . . , un, V=v1, v2, . . . , vm, be strings of text. If U′=ui1, ui2, . . . , uin′, where 1≦i1≦i2, . . . , ≦in′≦n, then U′ is called a substring of U. If U′ is also a substring of V, then U′ is a common substring of U and V. The LCS of U and V is a common substring of U and V with the greatest length among all common substrings of U and V.
As an example, given two strings A=abcbdda, B=badbabad, the LCS of A and B is abbd.
Given a search term, the following process is executed by the clarification module 16 to select the index terms relevant to that search term:
With the above method executed by the clarification module 16, for each of the search terms, a list of relevant index terms is obtained, and a list of the total fitness values between the search term and each of its relevant index terms is also obtained. Combining the list of index terms related to each search term creates a list of index terms for all the search terms. If the size of this list is less than a manageability threshold, then the index terms in this list are used for retrieval. If, on the other hand, the number of index terms in the list exceeds the threshold, the following method is further applied to reduce them:
With the set of index terms selected, they are used to retrieve relevant documents. This is performed by the Document Retrieval module 18, which executes the following steps:
Once the most relevant documents have been identified as search results 20, they are presented to the user in an appropriate form using the interface 10. For example, the user may be presented with a list of documents to choose from, and individual documents may be displayed or played with a text-to-speech engine.
The information retrieval system as herein described has several advantages over conventional speech recognition systems, particularly its capability of automatically generating domain-dependent grammar rules and index terms. The system supports adaptive voice information retrieval services capable of working on different application domains. The system also enables automatic service creation and updating, and is therefore suitable for applications with dynamically changing content or topics, for example, a daily news service and voice access to emails.
Many modifications will be apparent to those skilled in the art without departing from the scope of the present invention as herein described with reference to the accompanying drawings.
| Number | Date | Country | Kind |
|---|---|---|---|
| PR 0824/00 | Oct 2000 | AU | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 5418951 | Damashek | May 1995 | A |
| 5752051 | Cohen | May 1998 | A |
| 5752232 | Basore et al. | May 1998 | A |
| 5819220 | Sarukkai et al. | Oct 1998 | A |
| 6006221 | Liddy et al. | Dec 1999 | A |
| 6026388 | Liddy et al. | Feb 2000 | A |
| 6081774 | de Hita et al. | Jun 2000 | A |
| 6101473 | Scott et al. | Aug 2000 | A |
| 6185527 | Petkovic et al. | Feb 2001 | B1 |
| 6519562 | Phillips et al. | Feb 2003 | B1 |
| Entry |
|---|
| Kageura, Kyo, et al; Method of Automatic Term Recongnition, Department of Computer Science, University of Sheffield, XP-002477863, BNSDOCID<XP 2477863A—>, pp. 1-25 May 27, 1996. |
| Mayfield, James, et al; Indexing Using Both N-Grams and Words, The Johns Hopkins University Applied Physics Laboratory, XP-002477864, BNSDOCID<XP 2477864A—1>, pp. 1-5 1999. |
| Nuance Speech Recognition System, Version 8.0, Grammar Developer's Guide, Copyright © 1996-2001 Nuance Communications, Inc. 196 pages. |
| European Search Report, Application No. PCT/AU01/01308, dated May 13, 2008 , 8 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20150019205 A1 | Jan 2015 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 10399587 | US | |
| Child | 14311979 | US |
