The present invention relates to techniques for extracting information from text data sources, and more particularly, to methods and apparatus for inferring regular expressions that parse and extract information from line-oriented data.
A number of tools exist for extracting information from text data sources, such as a web page or another document. Such tools are typically designed to interact with structured data, such as comma-separated files, database tables, or tag-structured data, such as XML. Unfortunately, most real-world data does not naturally exist in this form. While many data sources generate completely unstructured data, such as news articles, a lot of data is well-structured, in that it may be reasonably converted into structured or semi-structured data, but the conversion function or wrapper may not be obvious.
For example, line-oriented files, such as log files, are relatively structured. Line-oriented files are typically parsed using regular expressions. A number of techniques exist for inferring such regular expressions. For example, the Whisk algorithm finds rules about certain types of text, such as classified advertisements and natural language. See, for example, Stephen Soderland, “Learning Information Extraction Rules for Semi-Structured and Free Text,” Machine Learning, V. 34, Nos. 1-3, 233-272 (1999),
In addition, “Potter's Wheel” uses Minimal Description Length (MDL) patterns to infer regular expressions over text. Generally, the MDL principle attempts to encode the sample data compactly. Potter's Wheel provides a set of interactive tools that help transform data from one form to another. In addition, Potter's Wheel has an automatic inference engine that tries to find structure in the input data and then uses this structure to detect outliers, which are likely errors. See, for example, V. Raman and J. M. Hellerstein, “Potter's Wheel: An Interactive Data Cleaning System,” Proc. VLDB 2001, Rome, Italy (2001), downloadable from http://control.cs.berkeley.edu/abc/ and http://control.cs.berkeley.edu/pwheel-vldb.pdf.
A need exists for methods and apparatus for user-guided inference of regular expressions for information extraction. A further need exists for improved methods and apparatus for inference of regular expressions for information extraction where the test data has inter-line similarities and differences that are important cues for pattern inference.
Generally, methods and apparatus are provided for inferring regular expressions that parse and extract information from line-oriented data. According to one aspect of the invention, a regular expression is generated that matches a line of text by: evaluating a plurality of characters of the line of text to identify one or more domains associated with each of the plurality of characters; assigning a run-length to each of the identified domains; populating a data structure having a data position corresponding to each of the characters with the identified domains and corresponding run-lengths; and generating the regular expression based on the data structure.
According to another aspect of the invention, a user interface is provided that generates a regular expression that matches a line of text by: evaluating a plurality of characters of the line of text to identify sub-groups of characters that belong to one or more domains; presenting the identified sub-groups of characters to a user for review; allowing the user to adjust the sub-groups of characters using a visual interface; and generating the regular expression based on the adjusted sub-groups of characters.
A more complete understanding of the present invention, as well as further features and advantages of the present invention, will be obtained by reference to the following detailed description and drawings.
The present invention provides methods and apparatus for user-guided inference of regular expressions for information extraction. As discussed further below in conjunction with
The regular expressions are obtained by identifying a domain (data type), such as integer, word, space, or punctuation, that matches a prefix of the data sample. Consider the phrase “hello 123”, with domains Anything, Word, Space, and Digits. The phrase may be expressed as any of the following: “Anything+”; “Word+ Anything+”; “Word+ Space+ Anything+”; “Word+ Space+ Digits+”; or “Word+ Space+ Word+”. Consider a sequence of n integers, and assume that the inference engine has a domain library that includes both integers and real values. Each of the numbers in the sequence could be considered either a real value or an integer, leading to 2n possible patterns to consider. Since n is proportional to the number of characters on a line, the full search space rapidly becomes large.
According to one aspect of the invention, a progressive (depth-limited) search heuristic is employed that breaks the problem into manageable sub-problems. Generally, the progressive search heuristic reduces the search space from cn to c(k+n/k). Instead of generating all let-to-right regular expressions, the progressive search heuristic initially limits the search to k domains. On subsequent executions, the Anything domains are refined on the subparts of the samples that go with the corresponding Anything part. Consider the sequence of n numbers, which are either integers or real values. As indicated above, there would be 2n patterns to search with conventional techniques. The first pass yields 2k patterns. All of them will likely end with anything (if k<n). Thereafter, the algorithm is repeated on the text that matches the ‘Anything’ domain, and only the top m choices are taken. This is performed recursively. The progressive search heuristic yields approximately 2k*m(n/k) patterns to search. If 2 equals m, or assuming the base is a constant c, the number of patterns to search is reduced to approximately c(k+n/k).
According to another aspect of the invention, a greedy parameterization heuristic is employed. Consider the following exemplary lines:
aaa bbb ccc
aaa bcd efg
After an initial evaluation, the pattern “word word word” might be inferred. The first word in the pattern could then be parameterized in two different ways, as the constant “aaa” or any three letter word. Alternatively, the first word can be left alone, and considered to be any word. If this parameterization step considers each word separately, and there are n words, then there are 3n possible parameterizations to consider. Instead, the present invention considers each parameterization separately and independently, which requires only 3*n choices.
To handle line oriented data in which multiple “types” of lines exist, it is necessary to infer regular expressions with ‘disjuncts’ or the “|” operator. The present invention consider cases in which disjunction appears as the outermost operator, that is “pattern1|pattern2|pattern2”, not “ab (pat1|pat2) cd”. The previous work on expression induction with MDL (Potters Wheel) did not consider inferring disjunctive expressions.
The algorithms performed by the regular expression interactive editor 100 are discussed further below in conjunction with
Thereafter, during step 220, the regular expression generation process 200 computes the costs of all the patterns computed during step 210. For example, the cost of each pattern may be computed using a Minimal Description Length (MDL) technique. See, for example, P. Grunwald et al. (eds.), Advances in Minimum Description Length: Theory and Applications, M.I.T. Press (MIT Press), April 2005 (ISBN0-262-07262-9). Generally, to compute a minimal description length cost, you consider the number of bits required to code the string. For example, if a string is known to match “(A|B)*”, then clearly only one bit is needed per character, to distinguish the A from the B, while if it matches “(A|B|C|D)”, two bits are needed and so on. However, with the more specific pattern, (A|B)*, there may be samples that don't match, and then you have to pay with a higher cost (the normal 8 bits per letter) for each of these samples.
Finally, during step 230, the regular expression generation process 200 refines the patterns, as discussed further below in conjunction with
As previously indicated, the regular expression generation process 200 finds the best patterns during step 210. Exemplary pseudo-code 300 for finding the best patterns is shown in
In one exemplary embodiment of the invention, the patterns can be parameterized at the beginning (early) or end (late) of the process. Thus, during line 4, a test is performed to determine if the candidate patterns are to be parameterized early. The cost of each candidate pattern is computed during line 5, for example, using the MDL technique referenced above for all generated patterns.
Finally, during line 6, all patterns with no remaining disjuncts are discarded that don't match any minNeeded strings.
As previously indicated, the regular expression generation process 200 refines the patterns during step 230. Exemplary pseudo-code 400 for refining the patterns is shown in
As previously indicated, candidate patterns are generated for each string in the extracted sample during line 3 of the pseudo-code 300 of
In one exemplary embodiment, the “generate candidate patterns” function fills out the matchingChoices table only for those locations in the string that are of interest. For example, using the exemplary “hi 48.3” string shown in
The algorithm for populating the structure 510 matches each known domain at the start of the string. If a match is found, the position where the match ends is noted (i.e., the run-length). For example, the string match at the beginning of “hi 48.3” ends at offset 1. Now, if offset 2 is already in the table, then the process is complete, since the table has already been filled out starting at offset 2. Otherwise, the function is recursively called on the rest of the string, say “48.3,” but with the correct offsets to fill the table.
In an exemplary implementation, matching is greedy, so “h” and “i” are not each matched by “word,” since all of “hi” can be matched.
The matchingChoices data structure 510 summarizes a number of patterns that match the sample string. These patterns are returned as the candidate patterns generated by this string.
Optionally, as indicated above, a ‘halving pattern’ can be introduced, once a large part of the string is matched, as discussed further below in a section entitled “Refine Pattern.” The halving pattern matches the rest of the string with a simple “.*” or “match anything” pattern. In this manner, if the first part fails to match many strings, further significant processing is avoided on the second part.
A prioritization scheme can be established for the domains, such that more specific domains are assigned to characters over more general domains. For example, If one domain matches a phone number “999-999-9999” and another matches an integer ‘9*’, then we may want to prioritize the phone number higher than just integer, since the fact that it matches provides significant evidence that the match is correct.
The “Anything” domain that is assigned to the remainder of the text can be updated to one or more actual domains for the remaining characters during an optional refinement stage. See the fourth position of the structure 520 of
Improve by Adding Disjuncts
As indicated above, an optional maxDisjuncts parameters indicates the number of disjuncts (|'s) allowed in a pattern (i.e., how many different patterns the regular expression interactive editor 100 can generate). To attempt to add additional disjuncts to a pattern being evaluated, the regular expression generation process 200 removes from the input sample set all the sample strings that match the pattern. Thereafter, the regular expression generation process 200 attempts to find a good pattern on the remaining samples, using the techniques described above. This process is repeated until the maximum number of disjuncts is reached, or until the computed cost doesn't decrease.
For example, if there are three allowed disjuncts (maxDisjuncts), and none have been used so far, the score of a pattern being evaluated can be computed as follows. Take the cost for the samples the pattern matches, and divide the computed costs by the maximum number of samples the pattern matches, and n/3 (where n is the number of samples). In other words, it is the cost per match, but it should match at least ⅓ of the samples when there are three disjuncts available.
Refine Patterns
Some domains are not true domains but are designed to match larger sections of text, such as “.*” (match anything) or “anything but a ‘,’”. During an optional refinement stage, the regular expression generation process 200 tries to find a pattern to match the text skipped by these patterns.
Strings do not always begin with a domain pattern, although they typically eventually return to a regular domain pattern. Consider the following data samples:
(abc),
def,
1234,
A pattern generated in the left-to-right manner described above will not be sufficient. Instead, the halving pattern can be employed. The general form of a halving pattern, given some token t, is (.*)t(.+). Furthermore, if it is a single character, the pattern is ([̂t]*)t(.+).
To accomplish this, each line is broken into a set of tokens. The tokens are derived by breaking up each line into the four mutually exclusive (exemplary) domains of Letter+, Digit+, Space+, and Punctuation. The first line would have the following four tokens: “(”, “abc”, “)”, and “,”.
For each token, the path of the domains that precede it is stored. The exemplary process finds tokens that (a) appear on most of the lines, and (b) exhibit at least some variation in domain paths. (If all domain paths are the same, there's no need for a halving pattern.)
The exemplary embodiment considers up to three halving tokens:
Often, the three choices for halving tokens will match.
Matching a Sample
To match a sample, it not only has to match like a regular expression would, it has to match each user-highlighted region of the sample with a single domain.
As previously indicated, the regular expression interactive editor 100 provides a user interface that allows the user to modify and guide the inference algorithm.
According to one aspect of the invention, the regular expression interactive editor 100 allows the user to select multiple fields from the data in window 610 to be grouped. This process will collapse multiple fields into a single group.
In addition, the interface 700 optionally provides a function to allow a user to add or delete groupings from the patterns.
System and Article of Manufacture Details
As is known in the art, the methods and apparatus discussed herein may be distributed as an article of manufacture that itself comprises a computer readable medium having computer readable code means embodied thereon. The computer readable program code means is operable, in conjunction with a computer system, to carry out all or some of the steps to perform the methods or create the apparatuses discussed herein. The computer readable medium may be a recordable medium (e.g., floppy disks, hard drives, compact disks, memory cards, semiconductor devices, chips, application specific integrated circuits (ASICs)) or may be a transmission medium (e.g., a network comprising fiber-optics, the world-wide web, cables, or a wireless channel using time-division multiple access, code-division multiple access, or other radio-frequency channel). Any medium known or developed that can store information suitable for use with a computer system may be used. The computer-readable code means is any mechanism for allowing a computer to read instructions and data, such as magnetic variations on a magnetic media or height variations on the surface of a compact disk.
The computer systems and servers described herein each contain a memory that will configure associated processors to implement the methods, steps, and functions disclosed herein. The memories could be distributed or local and the processors could be distributed or singular. The memories could be implemented as an electrical, magnetic or optical memory, or any combination of these or other types of storage devices. Moreover, the term “memory” should be construed broadly enough to encompass any information able to be read from or written to an address in the addressable space accessed by an associated processor. With this definition, information on a network is still within a memory because the associated processor can retrieve the information from the network.
It is to be understood that the embodiments and variations shown and described herein are merely illustrative of the principles of this invention and that various modifications may be implemented by those skilled in the art without departing from the scope and spirit of the invention.