Patent Grant
7295965
The present invention generally relates to methods and systems for classifying natural language text or sentences, and more particularly to methods and apparatus for determining a measure of similarity between natural language sentences or text.
Computers and computer technology have contributed greatly to our personal and professional lives. Computers now help perform many tasks that were previously performed by humans only a few years ago. While computer systems and computer technology have made significant penetration into our lives, some tasks still require significant human intervention.
A recurring problem in the computer field is the difficulty of developing computer systems that can perform tasks that have complex, unpredictable or undefined input data. To date, these tasks typically require human intervention, and often, intervention by scarce human workers that have particular knowledge or expertise. An essential problem is the translation of knowledge and skill of a human expert to a computer system in such a manner that the computer system, when provided with the same fact pattern, reaches the same conclusion or decision as the expert.
The first implementations of such systems used conventional, sequential computers that perform a sequence of operations on a very limited number of data elements, such as an add or compare operation of two data elements. A sequential system that works with large numbers of data elements often requires prohibitively long computation times, even for very fast computer systems.
To reduce the number of data items to be dealt with, “expert” systems have been developed. One form of “expert” system attempts to implement human “expertise” in a number of rules. In a rule based expert system, knowledge engineers attempt to elicit from experts a set of rules that implement the reasoning of the experts when given a set of facts. The rules typically attempt to codify, for example, the knowledge, methodology and reasoning process used by experts to solve a particular problem. The rules are programmed as a sequence of decision steps and, given a fact pattern, the system executes the programmed sequence of rule decisions in an attempt to reach the same conclusion as the expert.
A limitations of such rule based “expert” systems is that much of the expertise expressed in the rules is based on knowledge from a large number of individual fact patterns, which are of necessity and purpose more general than individual cases. This results in a loss of a significant amount of information. In addition, it is often difficult to determine whether the correct set of rules has been implemented, particularly since many experts do not consciously know and understand their own methodology and reasoning processes, and may unconsciously create “rules” that do not in fact reflect their methodology.
Another limitation of many rule based “expert” systems is that a substantial investment of knowledge engineer and expert time is required to determine and implement the appropriate set of rules. Furthermore, if the rules do not produce a desired result, or the input data changes in a material way, rewriting or updating the rules is often an extremely difficult and time consuming process. The rules often interact with one another, and a change in one rule may require corresponding changes in other related rules.
One application where such difficulties have arisen is in generating knowledge repositories (or data bases) from legacy documents. Many companies are currently processing legacy documents for use in automated reasoning systems. In one example, a knowledge repository may be used as a diagnostic fault model for an airplane. In another example, a knowledge repository may describe company business practices.
In order to create a knowledge repository, the legacy documents are typically parsed and the relevant information is identified, often by hand. Some of the information can be easily identified from the document context or by pattern-matching techniques. However, much of the information can be more efficiently and accurately identified by having some level of understanding of the textual meaning of the information in the document. To illustrate this, it is known that some legacy maintenance manuals may include certain fault descriptions. To identify and extract the fault descriptions from the the legacy document, some level of understanding of the meaning of the text within the document would be extremely helpful. For example, the sentence “REPLACE THE GO-AROUND SWITCH, S2 ON THE RIGHT THRUST LEVER” may describe a fault, while “DO THE IRU BITE PROCEDURE (MM 34-21-00, FIG. 107)” may describe a procedure.
It is known that English sentences are extremely complex, and are subject to stylistic variation. Using tediously hand-generated rules that rely on regular expressions to identify the fault statements would be extremely difficult. In general, the more complex the pattern, the more difficult it is to write a regular expression to recognize the pattern. Thus, not only would it be difficult to generate rules that rely on regular expressions to identify fault statements with a legacy document, but such a system would likely be dependent on the writing style of the author, making it difficult to transfer the hand-generated rules and/or regular expressions to different legacy documents or even to different chapters within the same document. Thus, such systems may be highly brittle and error-prone. Because of the foregoing, there is often a large cost associated with creating knowledge repositories from legacy documents.
The field of search engines is another application where it is often desirable to identify and categorize certain information within documents. Search engines typically accept a user specified search expression, and compare the search expression to text in selected documents, databases or web pages. Using rules or regular expressions to identify and categorize text within documents can be difficult, time consuming, and tedious.
The present invention provides improved systems and methods for classifying natural language text or sentences by first determining a measure of similarity between natural language sentences or text. The present invention preferably uses Machine Learning (ML) algorithms to categorize and/or identify the similarity between sentences. The ML algorithms are preferably trainable, so that they can be easily customized to any particular domain. By using trainable ML algorithms, the system may be incremental and robust, which allows the accuracy of the system to be increased with additional training. One embodiment of the present invention exploits the semantic information captured by Natural Language Processing (NLP) techniques along with statistical modelling to calculate the similarity between texts.
It is believed that the present invention may provide as much as a 10-fold reduction or more in the time required to accurately categorize sentences in certain legacy documents, resulting in a cost reduction of 30% to 50%. It is also believed that the present invention may be used to substantially increase the accuracy of many search engines.
In one illustrative embodiment, the present invention is applied to categorizing sentences in a legacy document to ultimately generate a knowledge repository. In this embodiment, a number of training sentences are selected, and one or more ML algorithms are used to assign each of the training sentences to one or more predefined categories. The algorithms are preferably dependent on a number of operating parameters (e.g., weights) over its constituent components. Once assigned, a correspondence is displayed between each of the training sentences and each of the one or more predefined categories. A user may then review the categorization of each of the training sentences, and change the categorization where appropriate. Based on the changes made by the user, the algorithm updates the operating parameters to reflect the change in the assigned categories, resulting in one iteration of training. Thereafter, the algorithm may be further trained by repeating the assigning, displaying, changing and updating steps until a desired accuracy level is achieved. Once adequately trained, all of the sentences in a legacy document may be processed.
The assigning step described above preferably uses a distance metric to determine which predefined category to assign a particular sentence. For example, the distance metrics can provide a measure of similarity between a particular sentence and one or more sentences that have already been assigned to a particular category. The particular sentence is preferably assigned to the category that includes the training sentences that are most similar to it. One possible assignment is to assign the sentence to the category of the single most similar sentence (smallest distance). Another possible assignment is to assign the particular sentence to the category that contains the majority of the k most similar sentences.
There are a number of illustrative algorithms for determining a measure of similarity between a first sentence and a second sentence. One illustrative method calculates the similarity between a pair of sentences based on inputs from several similarity factors. The final similarity value is preferably calculated by a weighted combination of each of the constituent similarity factors. The constituent similarity factors may be functions or methods based on sentence grammar, sentence semantics, statistics, or other specific techniques developed for the language or domain. Some illustrative methods or functions may include, for example, a string match, a keyword match, a phrase match, or a match based on the statistical significance of words, or a domain model match.
In order to utilize sentence grammar or semantics as a basis for a similarity factor, the sentences are preferably first parsed by a Natural Language Parsing engine. The NLP engine preferably assigns words to their pre-defined grammatical categories (e.g. nouns, verbs, adjectives, . . . ). The engine may also build complete sentence diagramming trees (e.g. noun phrase, prepositional phrase, . . . ). The engine may also assign words to pre-defined semantic categories (e.g. subject, object, modifier, . . . ). The NLP may also identify other natural language elements, including for example, proper names, currency, or sentence voice (passive/active).
The information extracted by the NLP engine can then be used by any number of similarity factors to assist in the calculation of the similarity between sentences. It is believed that by calculating the natural language properties of the text, and then comparing words or groups of words from similar classes, a more accurate estimation of the similarity of the sentences can be achieved. Some illustrative methods or functions include, for example, a string overlap match, a keyword match, and a phrase match.
The string overlap match function may contribute to the measure of the overall similarity of the two sentences by identifying the number of overlapping words in the sentences, regardless of word type or class. For example, the string overlap match function may return the number of words in the first sentence that match a word in the second sentence, as a percentage of the number of words in the longest of the first sentence and second sentence.
The keyword match function may provide a measure of the overall similarity of the two sentences by identifying the number of overlapping words in the sentences, where only words with a common syntactic or semantic type are compared. For example, the keyword match function may return the number of words in the first sentence that match a word in the second sentence, where both words share a common type, as a percentage of the maximum number of keywords in the first sentence and the second sentence. The phrase match function may be similar to the keyword match function, but may be applied to phrases or words. This method allows, as one possible example, noun phrases to be compared, including (or not) information from any constituent prepositional phrases.
Other techniques may be incorporated to improve the robustness of the natural language metrics. For example, to compensate for spelling errors or the like, it is contemplated that a word in the first sentence may be deemed to match a word in the second sentence when, for example, the word in the first sentence and the word in the second sentence each exceed a minimum number of characters, and the word in the first sentence shares a predetermined minimum number of characters with the word in the second sentence. Similarly, words or phrases may be considered similar if they are related according to some pre-defined knowledge (either domain dependent or independent). For example, the related words may include an original word with the prefix or suffix removed, various tenses of the original word, and/or different words with similar meanings as identified by, for example, a domain thesaurus. In addition, it is contemplated that one or more named entities may be extracted from the first sentence and the second sentence; the named entities may include a complex groups of nouns. The named entities may then be treated as a single object when performing word or phrase comparisons.
A second general category or factor that can be used for the overall similarity metric is statistical information. Certain words are more common or more indicative of a sentence's meaning than other words. For example, the general noun “thing” indicates much less meaning about a sentence than a more specific noun. Statistical information can be gathered from the language the text was written in (i.e. a general database of word usage), or built from the text under consideration (i.e. domain-specific). There are different methods of building and using this statistical information; one possible technique is to calculate the correlation factor of words in the training text to the category of the sentence, and this correlation factor can then be used to predict the category of new sentences. In the current embodiment, the statistical information is captured in a domain model that returns a similarity value, wherein the similarity value is increased by the correlation factor of a model word when both sentences include the model word, and is decreased by the correlation factor of a model word when only one of the first and second sentences include the model word. By decreasing the similarity value when only one of the two sentences includes the model word, a negative probability or correlation is applied.
Statistical information can augment the natural language metric, and vice versa. For example, extremely common words (and hence information-free words) can be identified as “stop” words by the statistical methods, and then the natural language methods can exclude them from the similarity analysis. Inversely, the natural language information can be used to augment the statistical calculation, such as when a word can be used as both a noun and a verb; each type of usage may have a different statistical correlation.
The two above-mentioned categories of similarity factors (natural language and statistical) and any combinations are independent of the domain of the text. In some domains, there may be domain-specific information that will augment the accuracy of the similarity metric. In one embodiment, for example, certain airline maintenance words are “more interesting” to the human users of the system than others. Any domain-specific factors can be added to the similarity metric in the same manner as the domain-independent factors, and moreover, factors that mix the domain-independent methods with domain-dependent methods can be created.
Each of these factors can then be combined to create a measure of the similarity of two sentences. The metric is flexible enough to incorporate any additional techniques as they are designed or developed. Each factor contributes to the final measure through a weighted function that can be tuned to the domain under consideration.
While the above illustrative embodiments may be used to categorize sentences in legacy documents, other applications are also contemplated. For example, the illustrative algorithms of the present invention may be applied to search engines. Often, search engines compare the similarity of a search expression with sentences or to text in search documents. The present invention can advantageously perform such a comparison, using some notion of the meaning of the sentences to be compared, which in many applications is highly desirable.
Other objects of the present invention and many of the attendant advantages of the present invention will be readily appreciated as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, in which like reference numerals designate like parts throughout the figures thereof and wherein:
For some documents, such as maintenance manuals, there are often many grammatical oddities. At a minimum, many maintenance manuals are not written with standard English, but rather often contain abbreviations and/or strange structure. For example, some maintenance manuals may contain parenthetical cross-references, and a large vocabulary unfamiliar to standard parsers. Moreover, sentences may be truncated by the character recognition software used to read the manuals into computer readable form. It has been found that TTT parses these sentences reasonably well, giving fairly accurate part-of-speech tags and syntactic chunks. The LGP provides a reasonable identification of prepositional phrases.
After parsing each sentence, block 22 calculates pair-wise distance metrics between sentences. Then, and as shown at block 24, a small set of sentences are presented to the user for labeling, such as described above with respect to
Using the tuned metric, the user may enter into a full interactive training mode. The ML algorithm presents the user with a set of training sentences, along with an estimated category for each sentence. The user can make any necessary corrections. After the corrections are made, the ML algorithm may update the model, and then categorize another set of sentences. This iterative process may continue until the user is satisfied with the accuracy of the model. The ML algorithm may then be allowed to categorizes all remaining sentences in the document.
The fully categorized sentences can be used to build fault models for automated reasoning systems. Even after a fault model has been built, the user can still make corrections to the sentence categorizations via block 24 to produce an updated model via block 26. This may be used to increase the accuracy of sentence categorization at any time.
The distance metric is preferably a mathematical combination of a set of relevant high-level features that are transformed from a low-level feature space, e.g., the statistical domain model and the natural language parse information. The transformation preferably enables a knowledge-based distance-metric to be built, i.e., one that uses knowledge about the structure and meaning of the sentence. Such a distant metric may enable the ML algorithm to powerfully differentiate between sentence categories.
One approach for building the distance metric is to leverage recent advances in Natural Language processing technology. Other domain information may also be leveraged wherever possible. By combining one or more NL processing techniques, along with statistical models and any other available information, a single distance metric may be constructed that sufficiently differentiates between sentence categories.
In one illustrative embodiment, several NL techniques and statistical information are used to provide similarity factors for the distance metric. Some of the NL techniques may include parts-of-speech tagging and syntactic chunks. Parts-of-speech tagging identifies nouns, adverbs, adjectives, etc. in the sentences. Syntactic chunks identify such things as noun phrases, prepositional phrases and verb phrases.
A word model may also be developed and used. A word model is a statistical analysis of word-to-category correlations. Each of these techniques help identify relevant parts of the sentences. By using one of more of these NL techniques, a single distance metric may be constructed that sufficiently differentiates between sentence categories.
A number of illustrative methods for calculating similarity factors for use in some distance metric calculations are now discussed.
To reduce the effects of spelling errors, stylistic changes, abbreviations, and other variations, a longest common subsequence (LCS) technique may be used when comparing the word in each sentence. The LCS technique may, for example, accept a pair of word as equal if each of the words has at least four (4) letters and the words share more than 80% of their characters.
A keyword match function may contribute to the similarity metric by comparing only words of the same part-of-speech “type”; hence the similarity of the meaning of the two sentences can be more accurately determined, as further shown and described below.
Referring to
A phrase match function may contribute to the similarity metric by comparing only phrases of the same type, hence the similarity of the meaning of the two sentences can be more accurately determined, as further shown and described below.
Referring to
Another illustrative method for calculating a similarity factor between two sentences uses a model match function. In this particular embodiment, the model match function uses a statistical model that is based on word-to-category statistics based on the observation that certain words or phrases tend to correlate strongly to a particular sentence category. For example, the words “repair”, “replace” and “install” may correlate strongly to “fault” type statements in a maintenance manual, while the words “test”, “procedure” and “check” may correlate strongly to no-fault type statements.
Using these correlation factors, an illustrative model match function may return a similarity value between the two sentences. In doing so, a similarity value is initialized to zero, and is increased by the correlation factor of a model word when both the first sentence and the second sentence include the model word, and is decreased by the correlation factor of a model word when only one of the first or second sentences include the model word. The similarity value may be divided by the number of model words detected in either of the sentences, if desired. By decreasing the similarity value when only one of the first and second sentences include the model word, a negative probability or correlation is applied. This method has proven to be fairly powerful in categorizing sentences.
(+1)+check (−0.95))/2=0.05/2=0.025.
The various sentence similarity values discussed above may be combined to calculate a distance metric.
As shown at 48, some of the similarity factors di may rely on some of the information provided by one or more of the other similarity factors dj. When so provided, separate weights wij may be applied to each of the techniques, if desired. In one example, the similarity factor “d3” for the phrase match function may rely on the similarity factor “d2” of the keyword match function, if desired. Experiments were conducted with several of the above ML techniques, and obtained the best results from a nearest-neighbor approach, using k=10 neighbors and a weighted distance voting strategy.
The fourth column 64 includes a second set of illustrative weights wi for each of the NL techniques, wherein the string overlap function has a weight of 0.0, the keyword match function and model match function of a weight of 1.0, and the phrase match function has a weight of 2.0. Using these weights, the distance metric Dpq 46 of
The above-described NL techniques are only meant to be illustrative and that numerous other possible techniques can be used in the present invention. For example, to improve the robustness of the system, it is contemplated that related words may be identified and added to a sentence. The related words may include, for example an original word with the prefix or suffix removed, various tenses of the original word, and/or different words with similar meanings as identified by, for example, a domain thesaurus. In addition, it is contemplated that one or more named entities may be extracted from the first sentence and the second sentence; the named entities may include a complex groups of nouns. The named entities may then be treated as a single object when performing word or phrase comparisons. In some embodiments, one or more stop words may also be excluded from the first and second sentences. The stop words may be identified in a predefined stop list, and may include common words that are not likely to help identify the similarity of the first and second sentences.
Having thus described the preferred embodiments of the present invention, those of skill in the art will readily appreciate that the teachings found herein may be applied to yet other embodiments within the scope of the claims hereto attached.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4916614 | Kaji et al. | Apr 1990 | A |
| 5056021 | Ausborn | Oct 1991 | A |
| 5251131 | Masand et al. | Oct 1993 | A |
| 5418948 | Turtle | May 1995 | A |
| 5555408 | Fujisawa et al. | Sep 1996 | A |
| 5559693 | Anick et al. | Sep 1996 | A |
| 5682539 | Conrad et al. | Oct 1997 | A |
| 5893092 | Driscoll | Apr 1999 | A |
| 5963940 | Liddy et al. | Oct 1999 | A |
| 5987404 | Della Pietra et al. | Nov 1999 | A |
| 6076058 | Chengalvarayan | Jun 2000 | A |
| 6076088 | Paik et al. | Jun 2000 | A |
| 6078914 | Redfern | Jun 2000 | A |
| 6088692 | Driscoll | Jul 2000 | A |
| Number | Date | Country |
|---|---|---|
| WO 0133414 | Oct 2000 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20030004716 A1 | Jan 2003 | US |
