Patent Grant
8122043
Embodiments of a system and method for using an exemplar document or search query to retrieve relevant documents from an inverted index of a large corpus of documents are presented herein.
With the ever-increasing amount of data stored in electronic form, it is becoming increasingly important to effectively search through a large amount of data to find relevant data. With respect to text data, it is important to quickly and accurately search a large number of text documents (a “corpus”) to find documents of relevance to the searcher.
Various methods are known that allow a searcher to enter search terms or utilize an exemplar document to search for other documents related to the given search terms or exemplar document. One such method in the prior art involves the ranking of individual search terms by a TF-IDF (Term Frequency-Inverse Document Frequency) score. In such a method, each search term in a query or exemplar document is assigned a TF-IDF score based on 1) the frequency of the term in the query (the “TF” component) and 2) the inverse frequency (rarity) of the term in the documents of the corpus (the “IDF” component).
In general, the TF component will be higher for a given term if the term appears a relatively large number of times in the search query. The reason for this is that a frequently occurring term in the search query is usually a term of high importance to the searcher. Conversely, the IDF component for a given term will be lower if the term appears in a relatively large number of documents in the corpus. The reason for this is that a term that is ubiquitous across documents is often a very common word that is of little value to the searcher. For instance, common English words such as the articles “the” or “an” will occur in nearly every document of English prose and thus will have a low IDF component. Since searchers are usually not concerned with the presence of such ubiquitous words, the low IDF component will minimize the effect of these words on the overall TF-IDF score for these common words.
After calculating an individual TF-IDF score for individual search terms in the query, the individual documents of the corpus are ranked to determine their likely relevance to the searcher. This is often performed using a vector space model along with cosine similarity to determine the similarity between the documents of the corpus and the search query/exemplar document.
Embodiments of the system and method described herein provide for improved accuracy of TF-IDF scoring and ranking systems. In embodiments, the effect of the TF component is dampened based on the number of unique words in the query. In the prior art, the TF component was often given a disproportionately large weight for queries/exemplar documents with relatively few terms. Thus, for short queries, the prior art often discounted the importance of the IDF component during the calculation of TF-IDF scores and the ranking of documents in the corpus.
Some embodiments of the invention also improve the accuracy of TF-IDF scoring systems by processing synonyms or related word forms in a special manner. Specifically, some embodiments analyze the query/exemplar document and group synonyms or related word forms into “word groups”. Thereafter, TF-IDF scores are calculated for each individual word group rather than each individual word. By empirical testing, we have determined that such a grouping of words into word groups minimizes the disproportionately high TF-IDF scores given to uncommon words. Prior art systems, by contrast, often give disproportionate weight to an uncommon word present in the search query, regardless of the semantic meaning of the word.
The user also provides the system with a corpus of documents from which to search. In some embodiments, the corpus of documents is resident in a local or distributed database. A distributed database can be one in which a plurality of discreet individual databases are logically treated as a single database. In other embodiments, the corpus of documents is located on a network, including, but not limited to, the public internet. In providing the system with the corpus of documents, the user can provide the actual documents themselves. Alternatively, the user can provide links to documents or instructions on how the system can search for documents or obtain documents from another source.
At step 102, the processor analyzes the Query and groups the terms of the Query by synonym into Word Groups. The processor utilizes an electronic thesaurus to group the terms. Thus, if the Query contained the words “rabbit” and “hare”, these words would get grouped together into a single Word Group if the electronic thesaurus of the embodiment indicated that “rabbit” and “hare” were synonyms.
In addition to synonyms, the thesaurus in some embodiments includes varying word forms such as plurals, participles, verb forms, and any other related word forms. In other embodiments, a separate electronic dictionary is consulted to determine varying word forms before grouping terms together. Thus, if the Query contained the three words “rabbit”, “rabbits”, and “hare”, (a three-word “tuple”), these words would be grouped together into a single Word Group in embodiments of the invention. Similarly, a Query containing “jump”, “jumps”, and “jumped” would group those three words into a single Word Group. Likewise, a Query containing the words “go” and “went” would group those two words into a single Word Group.
At step 103, the processor analyzes each Word Group to determine the number of instances of a term from that Word Group that is present in the Query. This count of the number of instances in the Query is defined as “FQ” (Frequency in the Query). For example, if the first Word Group in a Query contains the tuple (rabbit, rabbits, hare), then the processor will count the number of times one of those three words appears in the Query. Thus, if the Query contains five instances of the word “rabbit”, three instances of the word “rabbits”, and one instance of the word “hare”, then the FQ value for the Word Group (rabbit, rabbits, hare) would be nine—the sum of five and three and one.
Likewise, if the second Word Group in the Query consists of the tuple (go, went) and the Query contains 30 instances of the word “go” and 40 instance of the word “went”, then the FQ value for the second Word Group would be 70—the sum of 30 and 40. After calculating the FQ value for every Word Group in the Query, the processor determines the maximum FQ value (“FQmax”).
At step 104, the processor calculates a scaling factor, “K”, for later use in dampening the “TF component” (Term Frequency component) of the TF-IDF scores. As discussed above, prior art methods for ranking documents based on TF-IDF scores often gave disproportionate weight to Query terms that occurred frequently in the Query but rarely in the corpus. This is especially true when the Query is relatively short and contains a relatively few number of search terms.
The scaling factor “K” in embodiments of the invention is used to dampen the TF component to minimize this disproportionate weight. In general, the TF component is dampened to a greater degree if there are relatively few unique search terms in the Query. Conversely, the TF component is dampened to a lesser extent if there are a relatively large number of unique search terms in the Query. Thus, in embodiments, “K” can be implemented as a generally decreasing function (or a monotonically decreasing function) with a value ranging from one to zero over the domain of positive integers, where the domain represents the number of unique terms in the Query. A relatively high value for K indicates a high level of dampening, with a K of 1.0 indicating that the TF component is dampened 100%. Conversely, a relatively low value for K indicates a low level of dampening, with a K of zero indicating that the TF component is dampened 0%.
A unique term (or unique word) is, as its name implies, a word that appears at least one time in the document. Unlike the grouping of synonyms or word forms into Word Groups in step 102, the counting of unique words in step 104 does not group synonyms or word forms together. Thus, if a Query contains the words “rabbit”, “rabbits”, and “hare”, each of these words is treated as a unique word. Similarly, if a Query contains the word “go” and “went”, each of these words is treated as a unique word in step 104. Multiple instances of the same unique word are ignored, however. Thus, if a Query consists solely of the words (rabbit, rabbit, rabbits, hare, go, went, go), then the system and method described herein will register five unique words in the Query: (rabbit, rabbits, hare, go, went).
In one embodiment, the processor counts the number of unique words (“C”) in the Query. K is then calculated using the following equation.
K=1/SQRT((C+2)/3) Eq. 1
In other embodiments, K can be calculated in a variety of ways. Generally, K should have a range between one and zero and should generally decrease over the domain of positive integers. In some embodiments, the range of K can be even smaller. For instance, K could range from 0.9 to zero in an embodiment. In another embodiment, K could range for 0.85 to 0.03.
At step 105, the processor calculates a Term Frequency value (“TF value”) for each Word Group in the Query. As described above, the TF component of the TF-IDF score represents the prevalence of a term in the Query.
In embodiments of the invention, the TF value for a given Word Group is calculated by dividing the FQ value for that Word Group by FQmax (the maximum FQ value across all the Word Groups). This “intermediate value” (undampened value) for TF is then dampened using the K scaling factor calculated in step 104. In one embodiment, TF for a given Word Group is calculated according to the following formula:
TF=K+(1−K)(FQ/FQmax) Eq. 2
In Eq. 2, both K and FQmax are constant across all Word Groups.
As described above in relation to the scaling factor K, the intermediate TF value (FQ/FQmax) will be dampened depending on the value of K. If K is equal to 1.0, then the intermediate TF value will be dampened completely and Eq. 2 will simply leave a TF value of 1.0. As described in more detail below, a TF value of 1.0 indicates a complete disregard of the Term Frequency component of the TF-IDF score. Conversely, if K is equal to 0, then Eq. 2 will simply leave a TF value of (FQ/FQmax). Such an undampened TF value is known in the prior art. As described above, however, an undampened TF value produces misleading search results, especially for short queries or queries containing relatively uncommon words.
In other embodiments, the intermediate TF value (FQ/FQmax) can be dampened in other ways. In general, the intermediate TF value should be dampened to a relatively higher degree as the number of unique words in the Query decreases. Conversely, the intermediate TF value should be dampened to a relatively less degree as the number of unique words in the Query increases.
At step 106, the processor calculates, for each Word Group, the number of documents in the corpus that contain at least one term that exactly matches a term in the Word Group. This value is designated as the “FC” (Frequency in the Corpus) value. When calculating the FC value for a given Word Group, the processor searches for an exact match in the corpus documents and does not search for synonyms or other word forms that are not already present in the Word Group.
For example, if a given Word Group contains the words (go, went), the system and method only looks for exact matches to “go” or “went” in the corpus documents. The system and method does not consider other synonyms or word forms (e.g., “goes”) that were not present in the Query and thus are not present in the Word Group. In this example, then, a corpus document containing only the word form “goes” and not “go” or “went” will not be included in the FC count for the given Word Group.
At step 107, the processor calculates an Inverse Document Frequency value for each Word Group. As described earlier, the Inverse Document Frequency component of the TF-IDF score represents the rarity of a term among documents of the corpus. Thus, Word Groups containing ubiquitous words such as the English articles “the” or “an” will generally have a very low IDF value because such words are likely to be found in a very large percentage of documents in the corpus. Word Groups containing relatively rare words, by contrast, will have a relatively high IDF value because the words are likely in relatively few corpus documents.
In one embodiment, the processor begins the IDF calculation by counting the total number of documents in the corpus, “N”. The processor then calculates the IDF value for a particular Word Group using the following formula:
IDF=ln(1+N/FC) Eq. 3
Summarizing Eq. 3, the total number of documents in the corpus (“N”) is divided by the FC value for the given Word Group. This quotient (N/FC) is added to one (1) and the natural logarithm is taken of the resulting sum.
In Eq. 3, the value of N is constant across all Word Groups and FC is the value calculated in step 106 for the given Word Group. In the case of a Word Group with an FC value of zero (indicating no “hits” or matches of a Word Group term in any of the corpus documents), the processor will assign an IDF score of zero to the Word Group rather than perform the calculation of Equation 3.
As described above, the IDF value is a measure of the rarity (inverse frequency) of a Word Group term in the corpus. Thus, if one or more terms of a given Word Group appear in a relatively few number of corpus documents, the IDF value for that Word Group will be relatively high. Conversely, if one or more terms of the Word Group appear in a relatively large number of corpus documents, the IDF value for that Word Group will be relatively low.
In other embodiments, the IDF score is calculated using different functions than Eq. 3. In general, the IDF value will decrease as FC increases.
At step 108, the processor calculates the TF-IDF score for each Word Group. The TF-IDF score, as its name implies, is simply a composite score based on the Term Frequency (TF) component and the Inverse Document Frequency (IDF) component. In embodiments, the TF-IDF score is calculated simply by multiplying the dampened TF component of step 105 by the IDF component of step 107:
TF-IDF=TF*IDF Eq. 4
The composite TF-IDF score for a given Word Group indicates the importance of the terms in that Word Group for overall searching purposes. As described above, the importance of a Word Group is measured both by the frequency of the terms within the Query and the rarity of the terms among the corpus documents. Terms that appear frequently in the Query are deemed to be terms of particular importance. By contrast, terms that appear frequently in corpus documents are deemed to be terms of lesser importance because they do not distinguish one corpus document from another.
At step 109, the processor utilizes the TF-IDF scores for the Word Groups to rank the corpus documents by relevance. First, the processor orders the TF-IDF scores from highest to lowest, with the highest TF-IDF score indicating the Word Group with the highest relevance. The processor discards any TF-IDF scores of zero. In some embodiments, the processor also discards all Word Groups except the ones with the top “M” TF-IDF scores, where “M” is a user-configurable parameter.
Next, the processor discards all corpus documents which contain no terms from the Query. These documents were identified during the calculation of FC values for the plurality of Word Groups in step 106. That is, only the corpus documents identified during step 106 as containing at least one term from at least one Word Group are retained in step 109; all other corpus documents are discarded.
In embodiments where the processor discards all but the top “M” TF-IDF scores, the processor further discards corpus documents that contain no terms from Word Groups having the top “M” TF-IDF scores.
Finally, the processor ranks all the remaining corpus documents using a vector space model that utilizes cosine similarity for vector comparisons. The processor creates a vector for the Query using the TF-IDF scores and one for each remaining corpus document also using the TF-IDF scores. The processor then ranks the documents using a cosine similarity function to determine the similarity between a given corpus document and the original search Query.
After the corpus documents are ranked, the rankings can be displayed to a searcher who can then review the documents.
The system of
Document Search Server 312 comprises a Tokenizer 303, Synonym Finder 304, TF-IDF Calculator 305, Query Builder 306, and Search Results Formafter 307. Search Appliance 313 comprises a Synonym List 308, Term Frequency Calculator 309, and Search Engine 310. Network Attached Storage 314 comprises a database 311 for storing an Index and a corpus of documents.
In operation, a user 301 provides an Exemplar Document 302 (or a search query) to the Document Search Server 312. The Tokenizer 303 (word parser) parses the words in the Exemplar Document 302 and outputs the individual words to the Synonym Finder 304. The Synonym Finder 304 looks up synonyms in the Synonym List 308 and groups related words and synonyms into Word Groups, which are sent to the TF-IDF Calculator 305. This parsing of words and grouping of related words and synonyms into Word Groups can be accomplished as described above with regard to step 102 of
The TF-IDF Calculator 305 accesses the Term Frequency Calculator 309, which in turn accesses the Index in database 311. The Term Frequency Calculator 309 calculates the TF component and the IDF component of each Word Group, accessing the Index as necessary, as described above with regard to steps 103-108 of
The Query Builder 306 interacts with Search Engine 310, which in turn accesses the Index in database 311. The Search Engine 310 uses the ranked TF-IDF scores to find the relevant documents in the corpus, as described above with regard to step 109 of
Accordingly, while the invention has been described with reference to the structures and processes disclosed, it is not confined to the details set forth, but is intended to cover such modifications or changes as may fall within the scope of the following claims.
| Number | Name | Date | Kind |
|---|---|---|---|
| 20060143175 | Ukrainczyk et al. | Jun 2006 | A1 |
| 20080215574 | Lin et al. | Sep 2008 | A1 |
| 20100332511 | Stockton et al. | Dec 2010 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20100332503 A1 | Dec 2010 | US |
