A portion of this patent document contains material subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyrights whatsoever. The following notice applies to this document: Copyright© 2007, Thomson Reuters Global Resources.
Various embodiments of the present invention concern machine-learning based classification systems and methods, particularly as used in record linkage for updating databases and/or paraphrase generation for event extraction.
A classifier is a computerized tool that classifies an input set of data according to a finite set of classes. One general form of classifier uses machine learning techniques to make classification decisions. Before using this type of classifier, it is necessary to train the classifier using preclassified data, referred to as labeled data. In general, the more labeled data that is available to train the classifier, the more accurate the classifier can be. However, labeled data is typically produced manually and is therefore expensive in terms of time and money. The expense of this type of data ultimately limits the practical application of classifiers.
To address this and/or other needs, the present inventors devised, among other things, new classification methods and systems that do not require training using large sets of labeled data.
This document, which incorporates the drawings, the abstract, and the appended claims, describes one or more specific embodiments of one or more inventions. These embodiments, offered not to limit but only to exemplify and teach the invention(s), are shown and described in sufficient detail to enable those skilled in the art to implement or practice the invention(s). Thus, where appropriate to avoid obscuring the invention(s), the description may omit certain information known to those of skill in the art.
One or more of the exemplary system and methods involve semi-supervised learning, which refers to the case or situation when the learner or classifier exploits (a presumably large quantity of) unlabeled data to supplement a relatively small labeled sample, for accurate induction. At least some of the exemplary systems and methods embody a simple semi-supervised learning technique that relies on assumptions of class-conditional feature independence. In particular, when a feature set can be partitioned into two class-conditionally independent sets, the original learning problem can be reformulated in terms of the problem of learning a predictor from one of the partitions to the other. That is, the latter partition acts as a surrogate for the class variable. The novel technique is generally referred to herein as surrogate learning. The technique or methodology is described generally and then in two specific applications: one for record-linkage and the other for paraphrase generation.
The general surrogate learning technique can be understood in considering the problem of learning a classifier from the feature spade X to the set of classes y={0,1}. Let the features be partitioned into X=X1×X2. The random feature vector xεX will be represented correspondingly as x=(x1, x2). Since the exemplary embodiment is restricted to a two-class problem, the construction of the classifier involves the estimation of the probability P(y=0|x1, x2) at every point (x1, x2)εX.
The exemplary embodiment makes the following assumptions on the joint probabilities of the classes and features.
Considering the quantity P(y, x1|x2) allows one to derive the following equations.
Estimating the quantity P(x1|y) from a smaller labeled sample. If X1 is finite and ‘small’, this can be done accurately.
Thus, the exemplary embodiment decouples the prediction problem into two separate tasks, one of which involves predicting x1 from the remaining features. In other words, x1 serves as a surrogate for the class label. Furthermore, for the two steps above there is no necessity for complete samples. The labeled examples can have the feature x2 missing.
The following example illustrates the intuition behind surrogate learning. Consider a two-class problem, where x1 is a binary feature and x2 is a one dimensional real-valued feature. The class-conditional distribution of x2 for the class y=0 is Gaussian, and for the class y=1 is Laplacian as shown in
Because of the class-conditional feature independence assumption, the joint distribution P(x1,x2,y) can now be completely specified by fixing the joint probability P(x1, y). Let P(x1=0,y=0)=0.3, P(x1=0,y=1)=0.1, P(x1=1,y=0)=0.2, and P(x1=1,y=1)=0.4. The full joint distribution is depicted in
Assume a classifier to decide between x1=0 and x1=1 from the feature x2. If this classifier is used to classify a sample that is from class y=0, it will most likely be assigned the ‘label’ x1=0 (because, for class y=0, x1=0 is more likely than x1=1), and a sample that is from class y=1 is often assigned the ‘label’ x1=1. Consequently the classifier between x1=0 and x1=1 provides information about the true class label y. This can also be seen in the similarities between the curves P(y=0|x1, x2) to the curve P(x1|x2).
Another way to interpret the example is to note that if a predictor for P(x1|x2) were built on only the samples from the class y=0, P(x1=0|x2) will be a constant value (0.6). Similarly the value of P(x1=0|x2) for samples of class y=1 will also be a constant value (0.2). That is, the value of P(x1=0|x2) for a sample is a good predictor of its class. However, surrogate learning builds the predictor P(x1 1×2) on samples from both classes and therefore additionally requires P(x1|y) to estimate the boundary between the classes.
The independence assumptions made in the setting above may seem too strong to hold in real problems, especially because the feature sets are required to be class-conditionally independent for both classes. The two applications described below are based on a special tailoring of the generalized surrogate learning described above.
The problem is set us as learning a classifier from X=X1×X2 to the set of classes y={0,1}. The following assumptions are made:
Assumption 3 simplifies the learning task to the estimation of the probability P(y=0|x1=1, x2) for every point x2εX2. We can proceed as before to obtain the expression in Equation 3, with x1=1.
Equation 4 shows that P(y=0|x1=1, x2) is a monotonically increasing function of P(x1=0|x2). This means that after a predictor from X2 to X1 is built, one only need to establish the threshold on P(x1=0|x2) to yield the optimum classification between y=0 and y=1. Therefore the learning proceeds as follows.
We consider a problem with distributions P(x21y) identical to Example 1, except with the joint probability P(x1, y) given by P(x1=0, y=0)=0.3, P(x1=0, y=1)=0.0, P(x1=1, y=0)=0.2, and P(x1=1, y=1)=0.5. The class-and-feature joint distribution is depicted in
Note that on the samples from the class y=0 it is impossible to determine whether it is a sample from the target or background better than random by looking at the x2 feature, whereas a sample from the positive class is always a target. Therefore the background samples serve to delineate the positive examples among the targets.
Below are described applications of the surrogate learning approach to the problems of record linkage and paraphrase generation. These applications satisfy the assumptions in the second (100% recall) setting.
Record linkage generally refers to process of identification and merging of records of the same entity in different databases or the unification of records in a single database. The exemplary embodiment is presented as a solution to the problem of merging each of 20000 physician records, which are called the update database to the record of the same physician in a master database of one million records. The update database has fields that are absent in the master database and vice versa. The fields in common include the name (first, last and middle initial), several address fields, phone, specialty, and the year-of-graduation. Although the last name and year-of-graduation are consistent when present, the address, specialty, and phone fields have several inconsistencies owing to different ways of writing the address, new addresses, different terms for the same specialty, missing fields, etc. However, the name and year alone are insufficient for disambiguation. The exemplary embodiment had access to 500 manually matched update records for training and evaluation (about 40 of these update records were labeled as unmatchable with the information available).
As
Blocking, in the exemplary system and method, entails querying the master record database 410 with the last name of one or more update records from update database 420. As a result, a small set of candidate records is retrieved from the master record database, with the set of records having a high probability of containing the correct match for one or more of the update records.
Matching at module 440 generally defines features vectors for the one or more update records and corresponding candidate records, comparing and scoring these feature vectors, and identifying a most likely matching candidate record for each update record. Within the feature vectors, the feature values are either binary (verifying the equality of a particular field in the update and a master record) or continuous (for example, a normalized string edit distance between fields like street, address, first name, etc.).
More particularly, in the exemplary embodiment, the surrogate learning solution to the matching problem entails designating the binary feature of equality of year of graduation as the ‘100% recall’ feature x1, and the remaining features are relegated to x2. The required conditions for exemplary surrogate learning are satisfied because 1) in the exemplary physician data it is highly unlikely for two records with different year-of-graduation to belong to the same physician and 2) if it is known that the update record and a master record belong to two different physicians, then knowing that they have the same (or different) year of graduation provides no information about the other features. Therefore all the feature vectors with the binary feature indicating equality of equality of year-of-graduation are targets and the remaining are backgrounds.
The exemplary embodiment uses feature vectors obtained from the records in all blocks from all 20000 update records to estimate the probability P(x1|x2). Logistic regression can be used for this prediction task. For learning the logistic regression parameters, the exemplary embodiment discarded the feature vectors for which x1 was missing and performed mean imputation for the missing values of other features. Second, the probability P(x1=1|y=0) was estimated straightforwardly from the counts of the different years-of-graduation in the master record database.
These estimates were used to assign the score P(y=1|x1=1, x2) to the records in a block (cf. Equation 4). The score of 0 is assigned to feature vectors which have x1=0. The only caveat is calculating the score for feature vectors that had missing x1. For such vectors, the exemplary embodiment assigns the score P(y=1|x2)=P(y=1|x1=1, x2)P(x1=1|x2). Estimates for both quantities on the right-hand side of this equation are readily obtainable. The highest scoring record in each block was then flagged as a match if it exceeded some appropriate threshold.
Linking module 450 links or merges the update record with the matching master record identified by matching module 440.
Sentence classification is often a preprocessing step for event or relation extraction from text. One of the challenges posed by sentence classification is the diversity in the language for expressing the same event or relationship. What is described here and shown in
In addition to processors 501, system 500 includes a memory 502 which stores a sentence corpus or database 510, a source sentence extractor module 520, a target and background sentence extractor module 530, a paraphrase classifier module 540, training sentences 550, and a sentence classifier module 560. (These components are implemented using machine-readable data and/or machine-executable instructions.) Processors 501 and memory 502 may take a variety of consolidated and/or distributed forms.
The exemplary embodiment assumes that the unlabeled sentence corpus is time-stamped and named entity tagged with organizations. Also assumed is that an MA sentence must mention at least two organizations. The exemplary approach to generate paraphrases is the following.
At source sentence extractor module 520, the exemplary embodiment extracts a set of source sentences from the corpus 510 that match a few high-precision seed patterns (data structures). An example of a seed pattern used for the MA event is
‘<ORG1>acquired<ORG2>’
where <ORG1> and <ORG2> are place holders for strings that have been tagged as organizations. An example of a source sentence that matches the seed is
At target and background sentence extractor module 530, the exemplary embodiment extracts all sentences in the corpus (510) that contain at least two organizations such that at least one of them matches an organization in the extracted source sentences, and are within a time window of the matching source sentence. Of this set of sentences, all that contain two or more organizations from the same source sentence are designated as target sentences, and the rest are designated as background sentences.
In relation to surrogate learning, the binary “organization-pair equality” feature (both organizations in the current sentence being the same as the those in a source sentence) serves as the ‘100% recall’ feature x1. Word unigram, bigram, and trigram features were used as x2. This setup satisfies the required conditions for surrogate learning because 1) if a sentence is about MA, the organization pair mentioned in it must be the same as that in a source sentence, (i.e., if only one of the organizations match those in a source sentence, the sentence is unlikely to be about MA) and 2) if an unlabeled sentence is non-MA, then knowing whether or not it shares an organization with a source does not provide any information about the language in the sentence. Since an organization is unlikely to have a MA relationship with two different organizations in the same time period, the backgrounds are unlikely to contain MA sentences, and moreover the language of the non-MA target sentences is indistinguishable from that of the background sentences. If the original unlabeled corpus is sufficiently large, the target set is expected to cover most of the paraphrases for the MA event but may contain many non-MA sentences as well.
At block 540, the task of classifying paraphrases entails filtering the target sentences that are non-MA and flagging the rest of the targets as paraphrases. To this end, the exemplary embodiment defines a classifier between the targets and backgrounds. The feature set used for this task was a bag of word unigrams, bigrams and trigrams, generated from the sentences and selected by ranking them by mutual information. A support vector machine (SVM) is used to learn to classify between the targets and backgrounds and the sentences were ranked according to the score assigned by the SVM (which is a proxy for P(x1=1|x2) The scores are then thresholded to obtain the paraphrases, output and held at training sentences 550 for use in training sentence classifier module 560.
Example 3 below lists some sentences to illustrate the surrogate learning approach. Note that the targets may contain both MA and non-MA sentences but the backgrounds are unlikely to be MA.
Seed Pattern: “bought<ORG>”
Source Sentences
Target Sentences (SVM score)
3.<ORG>US Airways<ORG> is asking <ORG>Delta<ORG>'s official creditors committee to support postponing that hearing. (−0.99914371)
Background Sentences (SVM score)
The embodiments described above are intended only to illustrate and teach one or more ways of practicing or implementing the present invention, not to restrict its breadth or scope. The actual scope of the invention, which embraces all ways of practicing or implementing the teachings of the invention, is defined only by the issued claims and their equivalents.
This application claims priority to U.S. Provisional Application No. 61/063,966, which was filed on Feb. 6, 2008, and to U.S. application Ser. No. 12/341,926, which was filed on Dec. 22, 2008, and as a continuation of U.S. application Ser. No. 12/367,371, which was filed Feb. 6, 2009, and which are all incorporated herein by reference. U.S. Provisional Applications Nos. 61/008,714 and 61/063,047, which were filed respectively Dec. 21, 2007 and Jan. 30, 2008, are also incorporated herein by reference.
Number | Date | Country | |
---|---|---|---|
61063966 | Feb 2008 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 12367371 | Feb 2009 | US |
Child | 13361460 | US |