Multistage intelligent string comparison method

Abstract

A multistage database search method includes a first stage that only considers textual polygraphic frequencies and a second stage that considers the location of the polygraphs and coarsely approximates the proper alignment between the query and the database strings and a third stage that provides a better approximation of the alignment. The final output is a listing of suggested database records ranked by their similarity to the query.

Description

FIELD OF THE INVENTION

The present invention relates to database searching and particularly to a highly error-tolerant yet time-efficient database search function based on bipartite weighted matching. More particularly, a multistage database search method includes a first stage that only considers textual polygraphic frequencies and a second stage that considers the locations of the polygraph and coarsely approximates the proper alignment between the query and the database string and a third stage that provides a better approximation of the alignment. The final output of the process is a listing of suggested database records ranked by their similarity to the query.

BACKGROUND OF THE INVENTION

Over the past decade "database" has transitioned from an application used by a relatively small number of users in highly structured corporate data processing environments, to one at the center of mainstream computing. This occurred in large part because of the decade's striking advances in connectivity. The mid 80's emphasis on local area networks has been replaced with the world Internet. At the same time the set of computer users accessing databases has grown from a somewhat homogeneous and geographically localized collection to a highly diverse group spanning the globe and speaking many languages.

The present invention is a software method, performed by software instructing a computer, which addresses a central problem that is emerging as a result of these changes. That is, the problem of "robust semistructured text retrieval" for small and medium size databases. The crux of the invention resides in a function that compares two text strings returning a numerical indication of their similarity. Typically one of these strings is the user's query and the other is a string from the database. Because this function is very fast it is possible to compare the query with thousands or even hundreds of thousands of database fields while still delivering acceptable response time. Also included is a high-speed heap data structure used to track the "best matches" encountered during a search.

Semistructured text lies between fully structured databases and unconstrained text streams. A fully structured database might, for example, represent a person's name as many fields corresponding to its parts. The semistructured approach represents the name in a less formal way using a single text field. Other examples of semistructured text fields are: addresses, item names or descriptions (as in an online catalog), book or paper titles, company or institution names. Several such fields might be combined into one. For example, the query might be "problmoptimldictionry" and the record is Anderson, Optimal Bounds on the Dictionary Problem LNCS, 401, 1989.

All three of the words in the query above are misspelled, occur in the wrong order and there are no spaces separating the words. Nevertheless, the desired record is identified, using a preferred embodiment of the invention, from a listing of 50,360 paper descriptions in the field of theoretical computer science found in J. Seiferas' "A large Bibliography on Theory/Foundations of Computer Science at ftp://ftp.cs.rochester.edu, 1996-7. Author(s) name, paper title, and related information are combined into a single database text field.

Considerable variation is possible in the description of an item using a semistructured field. A person's last name might be listed first or last. The middle name might be excluded or abbreviated. The ordering of a complex name's parts is not always well determined. In principle, a policy might be established to regularize representation but, in practice, such policies rapidly become complex and confusing. Instead, the problem of directly dealing with these variations is handled by increasing the sophistication of the software that is used to compare queries with semistructured fields. Similar variations occur in user queries where the problem is perhaps greater.

An important benefit of the invention is that the queries are simple free-form expressions of what the user is looking for. There is no query language, and the comparison function is rather robust with respect to typical errors, missing or extra information, and overall ordering. Also, a preferred embodiment of the invention includes no natural-language specific considerations. It operates on byte strings and as such may be used across languages and perhaps for applications that have nothing to do with language (such as DNA comparison).

Using a 200 Mhz Pentium-Pro processor and the preferred embodiment of the invention, processing one byte of database information typically requires roughly 0.5 .mu.s. So 100,000 fields of 30 characters can be processed in 0.15 seconds. It is in some sense a fourth generation implementation of this general approach.

Algorithms of the general type used in the present invention were introduced in the master's thesis of P. N. Yianilos entitled "The definition, computation and application of symbol string similarity functions," Emory University, Department of Mathematics, 1978, and were later used in the commercial spelling correctors of Proximity Technology Inc., and Franklin Electronic Publishers. The linguistic software components of these companies were ultimately used under license in word processing programs from hundreds of publishers, in typewriters, and in tens of millions of hand-held spelling devices.

The PF474 VLSI chip was a special purpose pipelined processor that implemented such an algorithm. The chip is described in an article entitled "A dedicated comparator matches symbol strings fast and intelligently," in Electronics Magazine, McGraw-Hill, December 1983 and in an article by S. Rosenthan entitled "The PF474--a coprocessor for string comparison," in Byte Magazine, 1984, and in U.S. Pat. No. 4,490,811 by Yianilos and Buss entitled "String Comparator Device Systems Circuit and Method." Today's software matches and even exceeds the performance of this devices--although the comparison is not entirely fair since the PF474 was clocked at only 4 Mhz. The same design implemented today would still result in a 1-2 order of magnitude hardware advantage.

The Friendly Finder software utility described by M. J. Miller in an article entitled "First look--friendly program doesn't need exact match to find database search objects," in Info World, 1987 and first introduced in 1987 by Proximity Technology, Inc. implemented the algorithm together with software accelerations and special treatment for bigrams. The result was that small database could be searched on early personal computers without using the PF474 chip. The computational heart of Friendly Finder was also made available under license and called "P2. "

A transition to the bipartite matching viewpoint took place with two articles by Buss and Yianilos, one entitled "Linear and o(n log n) time minimum-cost matching algorithms for quasi-convex tours," in the Proceedings of the 5th Annual ACM-SIAM Symposium on Discrete Algorithms, 1994, pp. 65-76, and another entitled "A bipartite matching approach to approximate string comparison and search," technical report no. 95-193, from NEC Research Institute, Inc., Princeton, N.J., and the algorithms were both improved and in some cases simplified. The result is entirely new algorithms that are still of the same family.

The present invention is the first implementation based on these new developments. The algorithms of Yianilos and Buss lead to linear time algorithms for a large class of graph cost functions including the simple linear costs used by LIKEIT. LIKEIT is a software system implementing the multistage method forming the present invention. Linear time matching algorithms for this particularly simple special case were first present in an article by R. M. Karp and S.-Y. R. Li entitled "Two special cases of the Assignment Problem" in Discrete Mathematics, 13 (1975), pp. 129-142.

A portion of the disclosure of this patent document (LIKEIT SOFTWARE) contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. v,1-83/2

An alternative approach to string comparison computes "edit distance", described in an article by Hall and Dowling entitled "Approximate String Matching," in Computing Surveys, 12, 1980, pp. 381-402 and in an article by Sankoff and Kruskal entitled "Macromolecules: The Theory and Practice of Sequence Comparison," Addison-Wesley, 1983, that is the minimum-cost transformation of one into the other via some elementary set of operations. The most common form of the approach uses weighted insertion, deletion, and substitution operations and the distance computation is a straightforward dynamic program. There are two problems that led to the present invention. First, the algorithm runs in O(m.multidot.n) time where m, n are the string lengths. The present method runs in O(m+n). Second, the edit distance approach is highly sensitive to global permutation, e.g. changing word order. Humans frequently are not so sensitive and the invention deals well with this issue.

A similar approach, also used in effect by Friendly Finder, is to build an optimal weighted matching of the letters and multigraphs in the query, and those in the each database record. Words receive no special treatment. In this sense it is related to the document retrieval approach of M. Damashek, in an article entitled "Gauging similarity with n-grams: Language-independent categorization of text," in Science, 267, 1995, pp. 843-848 and S. Huffman and M. Damashek entitled "Aquaintance: a novel vector-space n-gram Technique for Document Categorization," in Proc. Text Retrieval Conference (TREC-3), Washington, D.C., 1995, NIST, pp. 305-310.

The automaton based approach to fast string matching introduced in an article by Knuth, Morris and Pratt entitled "Fast pattern matching in strings," in SIAM Journal on Computing, 6, 1977, pp. 323-350, deals with exact matches only. A natural generalization relaxes the requirement of exact equality and allows a bounded (and in practice small) number of errors. Each such error is typically restricted to be either an insertion, deletion, substitution, or sometimes a transposition of adjacent symbols. Given a query string, it is then possible to build an automaton to detect it, or any match within the error bounds, within a second string. The recent work of Manber and Wu, in an article entitled "GLIMPSE: A tool to search through entire file systems," in Proceedings of the Winter 1994 USENIX Conference, 1994, pp. 23-32 and in an article entitled "Fast test searching allowing errors," in Communications of the ACM, 35, 1993, pp. 83-91, demonstrate that text can be scanned at very high speeds within this framework for comparison. The present invention's framework can satisfy queries that do not fall within the practical capabilities of the automaton approach because they are too different from the desired database record. The invention and related approaches are important and effective tools for medium-size textual databases, yet still small enough to scan in their entirety for each query.

SUMMARY OF THE INVENTION

In the prior art a related low-level comparison method was used operating on single letters and digraphs, and an entirely ad hoc approach to realignment was employed.

The present invention is an extension and combination of earlier ideas to form an overall method that is highly efficient. It is a time-tapered multi-stage process in which the earliest stages are fastest and eliminate many records from further consideration.

The invention extends comparison to longer polygraphs and employs a single automaton to detect all polygraphs in a database string that are present in the query. In the preferred embodiment of the invention, the first stage employs counts of matching polygraphs to eliminate many database records from further consideration.

The invention intelligently guides realignment based on the matching that arises. The mean of these edge lengths is used to suggest the next alignment. It is then possible to adjust the matching cost for this new alignment without rescanning the strings. This "free realignment" is an important idea used by the second stage. The second stage eliminates further records from consideration but requires more CPU time than the first stage.

The invention iterates this realignment-rematching process in the third stage. The result is a more intelligent match, but requires additional CPU time.

The invention compensates for variations in query and database string length by padding them to a fixed length, but effects this padding mathematically so that the resulting computation is trivial.

The output of each stage is maintained preferably in a binary heap.

The result is that in almost all cases, the final matches returned are the same as one would receive applying the final, most discerning but slow, process stage to the entire database.

The present invention will be more clearly understood when the following description is read in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic representation of the process comprising the present invention.

DETAILED DESCRIPTION

A highly-efficient ANSI-C facility is described for intelligently comparing a query string with a series of database strings. The bipartite weighted matching approach taken tolerates ordering violations that are problematic for simple automaton or string edit distance methods--yet common in practice. The method is character and polygraph based and does not require that words are properly formed in a query.

Database characters are processed at a rate of approximately 2.5 million per second using a 200 MHz Pentium Pro processor. A subroutine-level API is described along with an executable utility supporting both command-line and Web interface.

Processing begins by translating the null-terminated query string, character by character to an internal alphabet. Default operation collapses ASCII considerably by mapping all non-alphanumeric values to "space" and upper-case letters to their lower-case equivalents. This front end translation may be customized as described below.

The query is compared with a sequence S.sub.1. . . , S.sub.n of database strings. The result of each comparison is a numerical indication of similarity, and the software keeps track of the num.sub.-- matches most similar database strings encountered.

Experience shows that simple preprocessing of queries and database strings before they are passed to likeit () improves results. Leading and trailing white space is deleted, repeated white space is collapsed to a single space character, and finally, a single leading and space character is inserted. Combining these steps with the default translation described above results in, for example: ##EQU1## where the repeated spaces in the final result arise from the translation process. This combined processing was found to be an effective general purpose solution, and it is performed by the utilities described below. Other preprocessing schemes and translation mappings may be more suitable for particular applications.

The LIKEIT facility reads database records and outputs those that it regards as the most similar matches to the query provided. A weighted bipartite graph matching approach is taken to the problem of quantifying "similarity". The query string is imagined to be positioned above the database string and the matching problem is to correspond features using edges of minimum total weight. In the simplest case the features comprise single letters, and the weight of an edge is the distance (in units of string position) between occurrences in the top and bottom string.

A person's sense of string similarity fairly clearly depends on higher level features such as digraphs, trigraphs, and ultimately entire words. The LIKEIT facility captures this effect by efficiently constructing several matching problems--one for single letters, another for letter pairs (digraphs), etc. Its sense of similarity is then a composite of these solutions.

The particular alignment of the query above the database string clearly affects the cost of a matching. For this reason LIKEIT treats this alignment as a variable and attempts to minimize total matching cost over it. That is, the query is imagined to slide left or right until a position resulting in minimum total matching cost is found.

The result is a rather complicated process, and despite the emphasis on efficiency, a time consuming one. For this reason the LIKEIT facility is implemented as a three stage filter, as shown in FIG. 1, in which the computation above is the final stage designated filter F3. The two earlier stages filter F1 and filter F2 approximate the similarity judgment of F3 using far less time. The F2 stage coarsely approximates the optimization over query alignment as described below. The first stage F1 approximates the matching process itself by counting matching polygraphs--taking no account of their position.

Each of these stages, F1, F2 and F3, acts as a filter of the input database, outputting fewer records than are input. The number of records output from the first filter is denoted Y and the number delivered by the last is denoted X, where Y>>X. The size of the output of F2 interpolates between the outputs of F1 and F3. The LIKEIT facility sets Y to the greater of 10.multidot.X and 1000. The effect of this tapered filter approach is that the final output is (in practice) as though the full algorithm, F3 were applied to every database record. The stages compute increasing effective but CPU-intensive notions of similarity. The records output from each filter stage are maintained in a binary heap. The implementation is simple and fast. As a result, heap operations represent a negligible portion of the overall CPU time.

Each of the three filter stages operate on the query string and database strings as a series of polygraphs of various lengths (single letters are 1-polygraphs). Matching edges can exist only between polygraphs that occur in both the query string and the database string under consideration. Thus, all other data record polygraphs may be disregarded.

The present approach is to build a finite state machine (FSM) based on the query string which detects all polygraphs (up to some fixed length) in a database record that also occur in the query string. The machine changes state as each database record character is presented. The states correspond to the longest trailing polygraph that is also present in the query string.

The machine's construction is straightforward but involved and is described in the distribution file fsm.doc. Consideration was given to processing the database once using this machine and saving the result. But because far fewer records are considered by F2, F3 than by F1, and because the machine is very fast when compared with F2, F3, the records are reprocessed for each filter stage.

The matching filters F1, F2, F3 operate on polygraphs identified by the FSM. Default operation limits attention to polygraphs of lengths 3-6 for filter F1, and lengths 1-6 for F2, F3.

In all filters a form of normalization is required so that matching scores are comparable in the presence of variable length queries and database strings. This normalization may be regarded as "padding" both query and database string to some large length L that in the default is preferably 1024. In all cases the effect is easily computed and the padding is never actually performed.

Filter F1 counts matching polygraphs. Initialization identifies all polygraphs in the query string within the requested range (3-6 by default). The count of each within the query string is recorded. As the FSM processes database string characters and polygraphs are identified, F1 counts "matches" up to the limit imposed by each polygraph's multiplicity in the query string. For example, if "ing" occurs three times in the query string, then only the first three occurrences in the database string contribute to the match count. Database-string polygraphs that do not occur in the query string are unmatched by definition. Unmatched polygraphs also include those that were not counted because they exceeded the query string multiplicity limit.

Filter F1 takes no account of the relative position of matching polygraphs. It assigns a constant cost 0.5 L to matching polygraphs, and cost L to each pair that does not match. As such it is a relaxation of the later matching stage to trivial constant cost functions. Because position is irrelevant, alignment is not considered for F1. The final scores computed by F1, F2, F3 combine scores for each polygraph length and weight them linearly, i.e. length 1-6 receive weights 1-6 respectively.

The next stage, F2, begins with a left-aligned query, and decomposes the matching problem into subproblems for each "level" as defined in Buss and Yianilos, "Linear and o(nlogn) time minimum-cost matching algorithms for quasi-convex tours," supra. Each such level includes polygraph occurrences that alternate between the query string and database string. If the number of occurrences is even, the matching is uniquely defined. If the number of occurrences is odd, then LIKEIT approximates the optimal matching by omitting either the first or last occurrence. The entire process is implemented without actually recording the matching edges--only costs are propagated online as database characters are processed. Investing only a little additional time yields the optimal matching described in Buss and Yianilos, "A bipartite matching approach to approximate string comparison and search," supra--also in an online fashion--but this is not implemented in LIKEIT.

Having produced a matching, a single approximate realignment step is performed. This is accomplished by keeping track of the average edge length during processing, and mathematically repositioning the query string so that the average length is as near as possible to zero. It is important to note that the matching itself is unchanged--edges are simply expanded or contracted to account for the realignment. For this reason we refer to this as a "free realignment".

The final filter F3 begins as does filter F2 but after each realignment a new matching solution is constructed. This realignment-rematching step is performed a predetermined number of times, preferably three times, or until the score fails to improve. The mean-length approach to realignment taken by LIKEIT is easily implemented but we remark that the proper computation instead focuses on median length.

The facility's time performance is measured by applying the likeit program to the database of Seiferas, supra. This database is a listing of 50,360 papers in theoretical computer science. Each line gives the authors, title, journal or conference, and date. The fields were reordered to match this description and a `I` was added between them. The resulting file is 4,278,967 bytes.

The timings are made using an Intel Pentium Pro 200 MHz processor with a 512 K L2 cache under Linux version 2.0.27. The distribution was compiled using gcc--03 version 2.7.2.1.

Focus is made on the time required to process a single character of database text since this statistic enables application designers to quickly compute response time. This time is, however, not constant and the primary variables affecting it are the query's length and the number of best matches requested. Experiments verify that there is in practice much less sensitivity to the query's particular value, or the database's specific content.

TABLE 1______________________________________ # Matches ReturnedQuery Filter 1 5 25 100 500______________________________________Q1 F1 285 283 283 280 287 F2 40 42 42 42 168 F3 2 5 12 23 164 Total 327 330 337 345 619Q2 F1 301 299 299 301 306 F2 68 70 70 68 301 F3 12 21 44 108 285 Total 381 390 413 477 892Q3 F1 337 337 334 339 346 F2 98 98 96 98 437 F3 14 26 54 108 285 Total 449 461 484 545 1068______________________________________ Q1: Optimal Q2: Optimal Dictionary Problem Q3: Andersson, Optimal Bounds on the Dictionary Problem

Table 1 shows processing time (nanoseconds) per character of database text for three queries of increasing length and requests for between 1 and 500 best matches. The total time as well as times for each filter stage are shown. For example, 413 ns per database character are required for query Q2 applied to the test database, where 25 best matches are requested. The response time for this query is then 413 ns .times.4,278,967.apprxeq.1.77 seconds. It is also convenient to compute the processing rate 1/413 ns .apprxeq.2.4 million characters per second.

Table 1 also shows times for the three filter stages in the LIKEIT process. The patterns evident in this table are consistent with the algorithm's design. Regarding the qualitative nature of the timings, it will be noted that the F1 time is essentially constant for each query and varies little between queries. This is expected because the role of F1 amounts to counting polygraphs in database records and there is very little dependency on the query. Filter 2 time depends rather strongly on query string length since very similar processing takes place for each character of the database and query strings. For a fixed query string F2 is essentially constant through 100 requested matches--but has increased considerably at 500. This is explained by the choice to set the number of matches output by F1 to the greater of ten times the number of requested matches, and 1000. So up to 100 requested matches, F1 always outputs 1000 candidates for consideration by F2. Requesting 500 matches forces F1 to output 5000 candidates thus increasing almost linearly the F2 time required. For a fixed query it is expected that F3 time will increase with the number of candidates output by F2. For table's values of 1, 5, 25, 100, 500, filter F3 outputs 31, 70, 125, 316, and 1581 records respectively as determined from the search "problmoptimldictionry". Analysis of F3 time is complicated by the variable number of realignment steps performed. It is expected, however, fewer such steps to be necessary as query string length approaches the length of the database string. The table's F3 times are consistent with these two observations.

In summary, for each query string the time varies by roughly 2:1 as the number of requested matches ranges from 1 to 500. The variation is somewhat less than this within columns. The corner-to-corner variation is just above 3:1. Excluding the 500 matches column the variation is much smaller.

Application designers can approximate performance well, at least for budgetary estimation purposes, by simply assuming:

Each database character requires .apprxeq.400 ns to process, corresponding to a rate of 2.5 MB/second. However, this assumption fails in extreme cases such as databases of very short records. Here, per-record overhead dominates.

While there has been described and illustrated a preferred method of performing a multistage intelligent string comparison, it will be apparent to those skilled in the art that variations and modifications are possible without deviating from the broad teachings and spirit of the present invention which shall be limited solely by the scope of the claims appended hereto.

Claims

1. A method of searching a database for a query comprising the steps of:

(a) providing a database of strings of characters;

(b) providing a query string;

(c) identifying polygraphs that occur in said query string and also in said database strings;

(d) providing a cost to each said identified polygraph;

(e) positioning the query string relative to each database string;

(f) matching polygraph occurrences in the query string with those in each database string, the cost of matching providing a numerical indication of the similarity between said query string and each said database string;

(g) realigning said query string to reduce the cost by examining edges present in the matching solution;

(h) repeating said matching and said realigning a predetermined number of times or until the cost of matching fails to improve; and

(i) repeating the steps (c) to (h) above for each database string for the purpose of identifying those database strings most similar to said query string.

2. A method of searching as set forth in claim 1, where polygraphs are longer than a predetermined fixed minimum length and shorter than a predetermined fixed maximum length.

3. A method of searching as set forth in claim 1, where the query string is initially left-aligned with each database string.

4. A method of searching as set forth in claim 1, where said matching is approximated by relying only on the number of matching of said polygraph.

5. A method of searching as set forth in claim 1, where said matching is approximately identified by dividing matching polygraphs into levels so that occurrences alternate between said query string and database string such that: when the number of occurrences is even, the matching is uniquely defined and when the number of occurrences is odd, omitting either the first or last occurrence.

6. A method of searching as set forth in claim 1, where said matching is identified by a quasi-convex matching method.

7. A method of searching as set forth in claim 1, where said examination of edges comprises computing the mean of the edge lengths.

8. A method of searching as set forth in claim 1, where said examination of edges comprises computing the median of the edge lengths.

9. A method of searching as set forth in claim 1, where said matching following realigning is approximated by mathematically adjusting edge lengths.

10. A method of searching as set forth in claim 1, where said identification of most similar database strings is performed using a heap data-structure.

11. A method of searching as set forth in claim 1, where said query strip and database string are mathematically padded to a predetermined length.

12. A multistage method of searching a database for a query comprising the steps of:

providing a database of strings of characters;

providing a query string;

processing each database string using a first method selected on the basis of computational time efficiency for identifying database strings that are similar to said query string;

processing said similar database strings using a second method selected to be more accurate but less time efficient than said first method for identifying smaller numbers of database strings that are similar to said query string, and

providing final identification of most similar database strings based on the result of processing said similar database strings.

13. A multistage method of searching as set forth in claim 12, where said first method and said second method are selected from the group consisting of relying on the number of matchings of a polygraph, dividing matching polygraphs into levels so that occurrences alternate between said query string and database string such that when the number of occurrences is even, the matching is uniquely defined and when the number of occurrences is odd, omitting either the first or last occurrences, quasi-convex matching, and mathematically adjusting edge length.

14. A multistage method of searching as set forth in claim 13, where said providing final identification of most similar database strings is performed using a heap data-structure.

15. A multistage method of searching as set forth in claim 12, where said providing final identification of most similar database strings is performed using a heap data-structure.