a-c illustrate exemplary tables and a package produced to facilitate understanding of aspects of the innovation.
a-c illustrates exemplary package generation structures.
a-c illustrates an example of package optimization.
The various aspects of the subject innovation are now described with reference to the annexed drawings, wherein like numerals refer to like or corresponding elements throughout. It should be understood, however, that the drawings and detailed description relating thereto are not intended to limit the claimed subject matter to the particular form disclosed. Rather, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the claimed subject matter.
The task of record matching is modeled herein not as a single monolithic operation as previously done but as that of designing a record matching query (also referred to herein as a package) obtained by composing primitive operators including relational operators and data cleaning operators, such as approximate joins between relations based on similarity functions, attribute value translations based on joins with standard reference tables, derived column values and attribute value correction based on a measure of functional dependency. By formulating record matching over a program built over primitive operators, the innovation significantly generalizes the class of current record matching approaches.
The flexibility in allowing design of packages over primitive operators, however, causes the task of developing a record matching package to become much harder. To address this hardness in design, a technique is provided for assisting programmers in designing accurate record matching packages. More specifically, a package can be automatically generated or learned from a set of training examples that identify matches and non-matches. These packages can then be optimized for execution. Furthermore, the technique for learning packages is flexible in that it can allow and respect constraints specified by designers to control the learning, for example based on knowledge of a domain or efficiency.
Referring initially to
The generation component 130 can produce a matching query or package, based on the training examples and optional constraints, which is composed of one or more primitive operators. Referring briefly to
Formally, the goal of record matching is to identify matching record pairs between tables or relations R and S. Assume that the schemas of R and S can be reconciled (e.g. attribute names in R have been mapped to corresponding attribute names in S). Further, assume for reasons of clarity that the corresponding attribute names in R and S are identical. Let the schema of R (and S) be Σ={ID, A1, . . . , An}. Dom(A) is used where A⊂Σ, to denote the domain (set of values) of attribute set A. In fact, if A={A1, . . . , An}, Dom(A)=Dom(A1)× . . . ×Dom(An). Dom(Σ)×Dom(Σ) is referred to as the cross-product space. Informally, the answer U to the record matching problem is a subset of the cross-product space. That is, U⊂R×S⊂Dom(Σ)×Dom(Σ). In sum, let R and S be two relations with identifier attributes R.ID and S.ID, respectively. The goal of a record matching problem between two relations R and S is to identify a relation U of matching pairs, where ΠR.ID, S.IDU⊂ΠR.ID, S.ID(R×S).
Observe that in general a record matching package can be an arbitrary function, which can include arbitrary operations. Record matching packages can be restricted to be built upon a pre-defined set of primitive operators—including both data cleaning and relational operators. This restricted class of packages is fairly general and includes all generic record matching techniques known in the art. However, the innovation is not so limited. It should be appreciate that system 100 can support plug-ins that identify new and/or different primitive operators that are not known by or native to the system. Throughout this description, a set of available operators is meant to encompass those native to a system as well as pluggable operators.
Given two relations, there is a large number of choices for creating a record matching package using a set of available primitive operators. Further, these choices usually lead to different solutions (i.e., different subsets of R×S). To identify the best or close to the best record matching package among all possible candidates, the quality of a record matching package should be quantified. The notion of quality can be quantified using a set of examples. An example is a record pair belonging to R×S along with a binary label (e.g., 1 or 0) indicating whether or not the two records match. For instance, one can correspond to a match and zero can correspond to a non-match. Let Δ be a set of examples, each of which belongs to Dom(Σ)×Dom(Σ)×Dom(Class), where Dom(Class)={0, 1}. Informally, the quality of a package with respect to the example set Δ is the amount of (weighted) error a package makes over Δ. Consider the recommendations of a package over the pairs of tuples in the example set Δ. If a package classifies a pair of records as matching (respectively, non-matching) when it is marked as a non-matching (respectively, matching) pair in Δ, then it is an error. The total error quantifies the quality of a package.
Formally, package error can be defined as follows: Let P be a record matching package over input relations R and S, and Δ⊂R×S×Dom(Class) be a set of examples. Let n01 (and n10) be the number of non-matching (matching) examples in Δ that are misclassified as matches (non-matches) by P, and c01 (and c10) be the cost of misclassifying a non-matching (matching) example as a matching (non-matching) one. Then, the error of P on Δ is c01*n01+c10*n10.
The package creation problem solved by generation component 130 can now be defined more formally as: Given relations R and S, and a set Ω of primitive operators, determine the record matching package P* which has the minimum error on R×S×Dom(Class) among all record matching packages built using only operators in Ω.
The above formulations suggest that it be ascertained whether or not each pair in R×S matches. However, that is exactly the information needed to obtain to be obtained from the package. Therefore, a standard machine learning approach can be employed where the goal is to learn a package which does well on a set Δ of training examples and also generalizes well to examples not in Δ (i.e., R×S−Δ). This generalization requirement has been addressed in machine learning literature while building models over training datasets. Thus, the innovation is able to exploit known machine learning techniques. Additionally, many techniques for generating training example collections such as active learning can also be incorporated herein.
For purposes of understanding, a number of primitive operators are now described that can be combined to form record matching packages. It should be noted that what follows is a description of only some of the operators and types of operators that may be utilized to generate matching query packages. The scope of the innovation is not to be construed as limited thereto.
The generation component 130 can utilize one or more of relational and data cleaning operators. The innovation can employ standard relational operators including select, union and cross product. However, a distinction is made with respect to classes of selections, namely regular selections with predicates based on attribute values and those with predicates based on similarity functions (e.g., edit distance, longest common prefix, cosine similarity . . . ). The latter class is particularly important with respect to comparing record pairs. Besides the relational operators, many data-cleaning operators can be employed. For example, the similarity join operator, which joins pairs of similar records across two relations, can be utilized. Further, the fuzzy lookup operators can be used, which generalizes the standard lookup operation and has been shown to be very useful for standardizing erroneous values using domain-specific reference tables. In addition to these previously known operators a new functional dependency (FD)-correction operator is disclosed for correcting attribute values using functional dependencies (e.g., soft). These operators are described in further detail below. First, however, is a description of another new operator: split.
The split operator partitions its input relation into multiple disjoint subsets, thus it can be seen as the inverse of a union operator. The split operator can be used as syntactic sugar to simplify description of the learning algorithm. In practice, this operator can be utilized or alternatively replaced with standard relational selections. Herein, splits are considered that split based on the values assumed by a single attribute, either numeric or categorical. Given a numeric attribute A and a threshold value α, the split operator partitions a relation into two parts. The first partition includes records with A greater than or equal to α; the second partition includes the rest of the records. For a categorical attribute A, the split operator paritions the given relation into n parts, where n is the number of distinct values in Dom(A). It should be appreciated that the split operator can be generalized to return fewer partitions by grouping attribute values into sets. The numeric split operator can be defined more formally as σA≧α: Given a relation T, a numeric attribute A, and a threshold value α, σA≧α(T) splits T into two parts σA≧α(T) and σA<α(T). The categorical split σA can be defined as: Given a relation T, a categorical attribute A, σA (T) splits T into |Dom(A)| subsets: σA=α(T), for all aεDom(A).
Turning attention to data cleaning operators we consider similarity selection, similarity join, fuzzy lookup and FD-correction in more detail. The similarity selection operator filters out record pairs whose similarity, as measured by a specific function, is less (or greater) than a given threshold. More formally, let f(R.A, S.A):Dom(R.A)×Dom(S.A)[0, 1] be a similarity function, where R.A and S.A are sets of attributes, and f(R.A, S.A)=1 iff the corresponding attribute values are identical. The similarity selection operator is a filter σp, where the predicate p is of the form f(R.A, S.A)≧α or f(R.A, S.A)<α.
The similarity join operator between two relations R and S is a cross product between R and S followed by a similarity selection operator. In order to facilitate efficient implementations, it is defined as a distinct operator. Note that the similarity join generalizes the equi-join operator (instantiated when the similarity threshold is set to 1). Formally, the similarity join operator RpS, where the join predicate p is of the form f(R.A, S.A)≧α outputs all record pairs in R×S with similarity f(R.A, S.A)≧α.
For both the similarity selection and the similarity join operators, the similarity function used to compare attribute values is itself configurable. The subject innovation allows the choice to be specified by a programmer or to be selected automatically according to the desired criteria of creating a good record matching package. Standard similarity functions such as Jaccard similarity with (IDF) weights on tokens, edit similarity, longest common prefix, and the like can be considered for use.
A lookup operator is useful for standardizing values based on domain specific lookup tables. For example, state values in the U.S.A. may be abbreviated to two-character postal codes based on a translation table obtained from the U.S. postal service. Turning briefly to
The lookup translation would proceed as follows. Given an input relation R and the lookup table L, one could lookup the value r[State] for each record rεR against L. If r[State] is equal to l[State] for some record lεL, r[State] can be replaced with l[StateCode]; otherwise r[State] remains unchanged. Observe that this operation may be implemented as a left outer-join between R and L on the State attribute followed by a case (switch) statement for replacing the R.State value with that from L. For example, lookup translation using the state code in table 230 in
In general, the value r[State] may not match exactly with any value in L[State] due to errors in r, for instance, ‘Texus’ instead of ‘Texas’ in table 210 of
The fuzzy lookup operator generalizes the lookup operator (instantiated when the threshold is set to 1). Thus, herein the fuzzy lookup operator is considered. Like for the similarity selection and join operators, the similarity function for the fuzzy lookup operator is configurable and can be chosen by a learning based technique or specified by a user.
Formally, the fuzzy lookup operator can be defined as follows: Let R be the input relation, L the lookup table, B the set of join attributes of R and L, Y the set of output attributes in L. Attribute set Y is compatible with Bin that attribute values r[B] of record rεR may be substituted by l[Y] of record lεL. Let f(R.B, L.B) be a similarity function and 0<α≦1. The fuzzy lookup (FL) operator FL(L, B, Y, f, α) for record rεR returns the record with r[B] being replaced by the value l[Y] of the record lεL that maximizes the value f(r[B], l[B]) if f(r[B], l[B])≧α; if no such l exists, the FL operator returns r unchanged. The fuzzy lookup on relation R returns a relation where each rεR is transformed as above.
Turning attention to the FD-correction operator, this operator exploits reference tables and the knowledge of functional dependencies to correct missing and incorrect values in a record, even when the functional dependency is soft in that it holds for a large subset of the data, or when there is not a perfect reference table.
As an illustrative example, consider a relation consisting of USA addresses {Address, City, State, Zip}. The functional dependency Zip→City is an example of a soft functional dependency. This dependency does not always hold because a few zip codes may actually be associated with multiple cities. Further, there may not be a perfect reference table recording the ideal {Zip, City} combinations. However, this information may still be used to correct several errors in the city values. For example, in
The FD-correction operator can be defined formally as follows: Given a multi-set S of values, it can be said that a value νεS is α-dominant in S if more than α % of values in S are equal to value ν. Let R be the input relation. Given a reference relation L, a (soft) functional dependency A→B, where A and B are sets of attributes both in R and L, and a threshold 50<α≦100, the FD-correction operator FD(L, A, B, α) on a record rεR returns the record where r[B] is replaced by the α-dominant B value in σA−r[A](L) (i.e., the group of all records in L that share the same A value as r[A]); if no dominate value exists, the FD-correction returns r unchanged. The FD-correction operator on relation R transforms each record in R.
Both fuzzy lookup and the FD-correction transform an attribute value based on the value in attribute set A. The transformation itself may use external sources of information such as reference tables or rules, or be based on other functions. This can be accomplished by a rename operator. The rename operator can be defined ρg(A, θ):B, where g(A, θ) is a rename function that returns a value in Dom(B)∪{NULL} based on set A of attributes and parameter (threshold) value θ, and B is the set of attributes to be renamed. For record r, ρg(A, θ):B (r) returns the record with r[B] being replaced by g(r[A], θ) if g(r[A], θ)≠NULL; otherwise, the record is unchanged. For relation R, ρg(A, θ):B (R) returns ∪rεR{ρg(A, θ):B(r)}.
The fuzzy lookup operator can be written as ρg1(A, θ):A, where g1(A, θ) is the function that returns the standardized A value from the lookup table if the similarity on the join attributes is at least θ. Similarly, the FD-correction operator for A→B can be written as ρg2(A, θ):B, where g2(A, θ) is the function that returns the θ-dominant B value in the group (e.g., in the reference table) sharing the same A value if the dominant value exists. Note that for simplicity of expression, the lookup table (in the fuzzy lookup case) and the reference table (in the FD-correction case) are encapsulated into the rename function g. Since both the fuzzy lookup and the FD-correction can be represented by the general rename operator, hereinafter discussion with be focused solely on the rename operator.
It should be appreciated that the subject innovation can be extended with a rich class of instantiations of the rename operator, beyond fuzzy lookup and FD-correction. More operators enrich the expressive power of the packages, and quite likely the accuracy of the resulting package. What is characterized now is a class of operators that can instantiate the rename operator. Any operator that produces exactly one tuple for each input tuple and which commutes with the union operator can instantiate the rename operator. That is, applying the operator to a relation R is equivalent to applying the operator to each individual record rεR and then taking the union over the results of each application.
Consider another operator, segmentation, which takes an attribute value R.A and segments it into constituent attribute values A1, . . . , Ak based on regular expressions. Such an operator is extremely useful for identifying target attribute values across a variety of domains: addresses, media databases, etc. For instance, conventional commercial address cleaning tools (e.g., Trillium) rely heavily on this operator, which they call “parsing.” The output for each record r is independent of the output for another record t, and thus commutes with the union operator. Hence, it can instantiate the rename operator. The concatenation operator, which is the inverse of segmentation, concatenates multiple columns together, can also instantiate the rename operator. Note that both these operators modify the schema of input relations. Even though the discussion herein assumes, for reasons of clarity, that schema is not modified we note that the subject innovation is general enough to handle segmentation and concatenation operators.
Yet another operator can be the association rule application operator. Consider such an operator defined using the set of association rules, mined from a relation R, whose confidences are above a threshold α (e.g., α>50%). An input record r may trigger application of several association rules. Applying these rules in a consistent (e.g., predefined) order would yield a new record r′. The association rule application operator can be applied in a number of scenarios, where the input values may be missing or incorrect (e.g., like FD-operator but with broader applicability). This operator satisfies the properties required for instantiating the rename operator.
Referring now to
It is assumed that the generation component 130 has access to a set Δ of training examples, and the goal is to learn an accurate record matching package utilizing this set. The class of packages described herein after is restricted to those having a cross product operator between R and S as the first operator. This restriction facilitates understanding and enables the system to adapt existing machine learning techniques to the package creation problem. It should be appreciated that the innovation is not limited thereto.
As discussed supra, the generation component 130 is can learn packages over a set Ω of available primitive operators from a training set Δ of examples. Recall that it is assumed that the first operator in the package is going to be a cross product between input relations. It is said that a partially specified operator is one where only attribute names and table names (where required) are specified. In contrast, a fully specified operator also specifies threshold values (e.g., similarity threshold . . . ). In accordance with one embodiment, exemplary table 1, below, depicts information to be specified by a user and that to be automatically determined for each operator in Ω that is chosen.
(Note that in this section partially specified operators will be simply referred to as operators.) Thus, for example σf1(A1, A2)≧α, σf2(A1, A2)≧α amd σf1(A3, A4)≧α are considered different operators in Ω because they either use different similarity functions or are based on different numbers.
To facilitate clarity and understanding, a simpler case is first considered in which Ω includes only cross product, similarity selections and unions. This case is closely related to the decision-tree learning problem, thus those techniques can be employed to create a package. Later, this adaptation is extended to a more general case where Ω also includes rename and split operators. By exploiting the identified relationship between package creation and decision tree construction problems, the innovation can benefit from a large body of research on decision tree classification. For instance, the innovation can benefit from techniques for building decision trees and for improving the ability to generalize to examples not in the training set Δ. Although using learning techniques to solve record matching problems is not per se new, no earlier techniques consider learning packages over a set of primitive operators (e.g., data cleaning, relational . . . ). More significantly, previous learning techniques are not able to handle rename operators, which are extremely important for obtaining accurate record matching packages.
Selection-only packages decide whether or not an example represents a matching pair based only on the similarity scores (i.e., values of similarity functions in Ω). Conceptually, the training set Δ is extended by adding new derived columns E1, . . . , En. Each Ei value of a tuple t= r,s
εΔ corresponds to the similarity, measured by similarity function ƒi, between tuples r and s. The relation including IDS and the derived columns can be referred to as a feature table or similarity feature table provided by feature table component 310. It is observed that a selection-only package is closely related to a decision tree classifier which predicts whether or not examples are matches using only the derived attributes E1, . . . , En. This decision tree is produced by tree component 320. Given such a decision tree, the corresponding selection-only package can be constructed by package component 330 as the union of all branches in a decision tree, which predict examples to be matches. The following example illustrates the approach of constructing an extended feature table, building the decision tree and then translating it into a package, as performed by components 310, 320 and 330, respectively.
Consider the running example in
Presented hereinafter is a discussion of how a decision tree can be generated within the context of a record matching package. The first operator in the package is initialized to be a cross product between input relations R and S. The package is then grown one operator at a time. Each point p in the frontier of the package is associated with a subset Δp of the initial training examples Δ. In the beginning, all the training examples Δ are assigned to the first frontier point (i.e., the point after the cross product operator). For each frontier point p, the best operator is chosen based on an operator selection procedure (described below), and the operator is added into the current package at point p. Subsequently, the newly added operator, for instance σƒi(A1, A2)≧α, is applied to the example set Δp, partitioning Δp into two subsets: Δp1 and Δp2, which are in fact the two outputs of σƒi(A1, A2)≧α (the right output Δp1 is the set of examples that satisfy ƒi(A1, A2)≧α the left output is the set Δp2 that does not). These two output points are now added to, and the point p deleted from, the list of current frontier points. The procedure can then recurse on each of the current frontier points to grow the package until a stopping criteria is met. For instance, recursion can be stopped when the number of training examples at the current point is less then a threshold or if all the examples are all either matches or non-matches. After stopping, various techniques such as decision tree pruning can be utilized.
At any frontier point p, candidate selection predicates can be searched and the predicate ƒi(A1, A2)≧α that maximizes a benefit function measured over the training set Δp can be selected. Informally, the benefit quantifies the quality of using a specific operator at a node. In the example illustrated in
Although not limited thereto, herein the error measure is utilized as the benefit function. The error on data Δp at point p is the smaller package error between the package that treats all Δp as matches and the one that treats all Δp as non-matches. The error after applying a similarity selection to Δp is the sum of the error on Δp1 and that on Δp2. The amount of error reduced by this operator is its benefit. Note that the choice of benefit function is orthogonal to the tree construction. Commonly used impurity functions such as informational gain, gini index and the like can also be utilized. However, they do not handle variable costs of misclassification as required by the package error measure.
At each point in p, the frontier in decision tree is labeled either a match or a non-match. The weighted errors e1 and e0 obtained by classifying all examples Δp associated with p in the frontier as matches and non-matches, respectively, are calculated. If e1 is smaller, the point p is labeled as match; otherwise, the point is labeled non-match. Finally, the leaves labeled match are connected to a union operator as shown in
Turning to
Although other classification models (e.g., support vector machines . . . ) may be applied within the subject framework, the resulting models cannot always be translated to a package built with the primitive operators discussed thus far. In particular, more sophisticated similarity selection and similarity join operators allowing predicates involving linear combinations over similarity functions and thresholds can be utilized.
Now the package generating/learning procedure performed by generation component 130 is generalized to consider all primitive operators (e.g., including rename and split) previously mentioned. The primary challenge here is that with rename operators the order of transformation operator invocation becomes important even for preparing the feature table. For instance, the orders FD(L, A→B, 0.8), FD(L, B→C, 0.9), edit-similarity>0.90 and FD(L, B→C, 0.9), FD(L, A→B, 0.8), edit-similarity>0.90 result in different feature tables. Considering all possible orders of transformation operators and all possible parameters, where transformation operators are parameterized is infeasible. Therefore, the decision tree construction procedure is extended to incorporate transformation operators in its search.
The top-level process is almost the same as that for selection-only packages. After the first cross product, adding an operator into the current package is considered. All the operators in Ω with all possible threshold values can be tried. Their benefits based on training examples can be measured and the one with the highest benefit chosen. The process can then recur on the output(s) of the chosen operators. The stopping criterion can be the same. The benefit measures for split and rename operators are as follows:
A rename operator just modifies the data so that subsequent similarity selections can better identify matches. For instance, converting the organization name to a standard representation (e.g. using fuzzy lookup against a reference table from the Yellow Pages™) may improve the chances of two records representing the same organization being recognized as matches. Therefore, as illustrated in
The above discussion describes the case where only the attributes of one of R and S need to be renamed. However, there are cases where both tables need to be renamed before similarity comparisons. For example, R.State and S.State may both contain “dirty” values. In this case, to better measure the benefit of renaming R.State, S.State should also be renamed before appending a temporary similarity selection. To handle this case, we alternatively evaluate the benefit of a rename operator on R.A by appending the same rename on S.A using the same threshold value followed by a temporary similarity selection. Since both R.A and S.A are renamed using the same threshold, the cost of the search is not increased.
Referring to reference 520 of
Turning attention to
Packages returned by the package-learning component 110 are similar to a regular relational operator tree. The main difference is that the packages may contain data cleaning operators, which cannot be handled by a conventional query optimizer. However, properties of these new operators can allow optimization techniques to be applied. It is to be appreciated that the properties identified herein can also be used to extend a cost-based query optimizer to handle data cleaning operators. Thus, optimization component 620 can correspond to such an extended optimizer.
A primary observation is that the sequence, cross product between relations R and S followed by a “higher-than threshold” selection predicate f(A1,A2)≧α, can be replaced by a single similarity join f(A1, A2)≧α. In order to facilitate such a rewrite however, the package needs to be transformed such that all operators between a cross product and a higher-than threshold selection predicate are either pushed down or pulled up.
a-c provide an example to illustrate optimization functionality in accordance with an aspect of the innovation.
We now briefly introduce some equivalence properties based on which a package can be transformed without changing its output. Recall that each package can be a union of paths each of which start with a cross product and may also have similarity selection, rename, and standard relational operators. Note that the split operators can be translated to relational selections. Operators op1 and op2 commute with each other if exchanging the order of adjacent operators op1 and op2 does not change the output of the package on any input.
In order to state the commutativity properties for the new operators (e.g., renames), recall the notion for the operators here. The threshold values do not influence the commutativity properties and they are dropped from the notation. σA denotes a standard relational selection that selects data based on the attribute A=c, A≧c or A<c, where c is a constant. σƒ(B1, B2) denotes a similarity selection that filters data based on the similarity between B1 and set B2 of attributes. ρg(G):H denotes a rename operator that modifies attribute H based on a set G of attributes. For clarity, assume that H is a single attribute; the discussion generalizes to a set. × denotes the cross product operator.
More formally, let ρg(G):H be a rename operator:
H ∉ B2
H ∉ C
D ≠ H
By way of example, consider
The aforementioned systems have been described with respect to interaction between several components. It should be appreciated that such systems and components can include those components or sub-components specified therein, some of the specified components or sub-components, and/or additional components. Sub-components could also be implemented as components communicatively coupled to other components rather than included within parent components. Further yet, one or more components and/or sub-components may be combined into a single component providing aggregate functionality. The components may also interact with one or more other components not specifically described herein for the sake of brevity, but known by those of skill in the art.
Furthermore, as will be appreciated, various portions of the disclosed systems and methods may include or consist of artificial intelligence, machine learning, or knowledge or rule based components, sub-components, processes, means, methodologies, or mechanisms (e.g., support vector machines, neural networks, expert systems, Bayesian belief networks, fuzzy logic, data fusion engines, classifiers . . . ). Such components, inter alia, can automate certain mechanisms or processes performed thereby to make portions of the systems and methods more adaptive as well as efficient and intelligent.
In view of the exemplary systems described supra, methodologies that may be implemented in accordance with the disclosed subject matter will be better appreciated with reference to the flow charts of
Referring to
One of the main insights identified previously (Table 2) is that a rename operator ρg(G):H commutes with all the other operators if H does not overlap with the attributes in the other operations. Thus, intuitively whenever one needs to push a rename operator below another operator that does not commute with a rename, a new derived attribute H′ (that has not been used in any other operator) can be added and ρg(G):H can be replaced with ρg(G):H′. This is illustrated using the following example.
Consider an example package such as the one in ƒ2(City)≧0.9. The rightmost path is also optimized similarly.
Now, consider the leftmost path. This path cannot be transformed using the commutatively properties summarized in Table 2. However, the rename operator can still be pushed down as follows: Add a new attribute S.State1 to S, which is a copy of attribute S.State (where attribute values are actually replicated) and the corresponding attribute R.State1 to R, which is just an alias of R.State (where attributes values are not copied). The rename operator ρg(S.City, 0.9):S.State can be changed to ρg(S.City, 0.9):S.State1. To ensure equivalence of the transformation, every occurrence of State after the rename operator can be changed to State1. In this case, the predicate of the following similarity selection becomes ƒ3(City, State1)≧0.8. Now, the rename operator commutes with all the operators before it. Thus, one can push the rename down to S. Since the renaming operators commute with each other, the most selective one (e.g., ƒ3(City, State1)≧0.8) can be moved next to the cross product and pack them into a similarity join.
Method 1000 generalizes the above example with respect to rewriting to optimize each path in the learnt package. More specifically, rename operators can be pushed down at 1010. For each rename operator ρg(R.A, θ):R.B (exchange R and S if the rename operator is on S) from the bottom one to the top one, a new attribute R.B′ can be added to R and ρg(R.A, θ):R.B can be changed to ρg(R.A, θ):R.B′ where B′ is an unused attribute name. Further, the corresponding attribute S.B′ that is just an alias of S.B can be added and every occurrence of B in any subsequent (upper) operator to B′ For example, consider the following path: σƒ1(B)ρg1(R.B):R.B σƒ1(B)ρg2(R.B):R.B σƒ3(B)(R×S), where thresholds are omitted for ease of expression. After the rewriting, the path becomes: σf1(B″)ρg1(R.B′):R.B″σf1(B′)ρg2(R.B):R.B′σf3(B)(R×S). Note that B′ and B″ are new attributes and, according to Table 2, each rename commutes with all the similarity selections to its right (below it). As a final step, all the rename operators are pushed down to the base table, preserving order. Then, the example becomes σf1(B″)σf1(B′)σf3(A)((ρg1(B′):B″ρg2(B):B′R)×S). Now all operators above the cross product commute with one another. At reference numeral 1020, single-table selections can be pushed down. Single-table selections (e.g. resulting from split operator) can be pushed down to the base table. At numeral 1030, cross products are replaced with similarity joins. The similarity selection or the most selective one, if selectivities can be estimated, is moved next to the cross product and the cross product is packed into the similarity selection if possible.
According to Table 2, all the above re-orderings are safe. Thus, the resulting package is equivalent to the original one. Note that it is at least in principle possible that a greater-than threshold similarity selection may not be found to generate the join for a path. This is an unusual phenomenon: dissimilar pairs of records match each other. In this case, either the similarity functions are inadequate or the training examples contain errors. A user can be alerted to prompt further information in such a scenario.
There are several other opportunities (e.g., user driven, cost-base automation . . . ) for optimizing a package. Because the output packages are interpretable by a domain expert and the semantics of the operators are well defined, the expert may rewrite the package to yield semantically equivalent packages by considering both the properties of the operators, common sub-expressions, and the data characteristics.
In particular, one can exploit the shared intermediate results between paths. For instance, the middle path and the right path of tree 720 (σƒ1(State)≧0.7S. Many conventional ETL (Extract, Transform, Load) engines can execute directed acyclic graphs over operators and hence already allow shared execution for common paths. Further, view materialization may be effectively employed for sharing intermediate results. A cost-base optimization can also be utilized to rewrite packages and evaluate the benefit of materializing views for shared intermediate results. However, the challenge here is to consider the additional data-cleaning operators such as similarity join and rename operators along with the standard relational operators.
Recall that the goal of the package creation problem is to minimize the error on R×S, while the techniques discussed above built a package for a given training set. Even approaches based on random sampling for collecting training examples provide probabilistic guarantees on the accuracy of the overall package learnt. Under the probabilistically approximately correct (PAC) model, if a teacher classifies a random sample of R×S without any error, and the learner builds a perfect (100% accurate) model on this sample, then this model is expected to have a very high accuracy on the entire dataset R×S.
In many record matching scenarios, the number of non-matching pairs is significantly higher than the number of matching pairs. In such cases, the number of examples required in a sample for it to contain a reasonable number of matching pairs is very high. Thus, the burden on the teacher increases significantly. Herein, active learning techniques can be leveraged such that examples are chosen carefully to maximize the overall gain.
Two other significant issues are now addressed. First, the issue of letting users control the resulting package structure addressed. Secondly, improving the efficiency of package learning is discussed.
Often a package designer may desire to control the structure of the output of the package learnt. Requirements may stem either from knowledge of the domain or from an efficiency standpoint, among other things. Consider once again the example provided in
The subject innovation allows users to specify such structural constraints on the desired package. The intuition is to divide all operators into equivalence groups. The groups are then partially ordered where any operator in a group has to be preceded in the package by at least one operator each in all preceding groups.
It is fairly straightforward to enforce the structural constraints while building decision trees and hence the package. While learning a package, whenever an operator is chosen to grow the package only candidate operators that satisfy constraints are considered. Package rewriting has to also respect these constraints.
As per improving efficiency of the learning algorithm, the main observation is that during the package-learning phase, similarity scores according to all functions are computed repeatedly. Therefore, if the computed similarity values are cached, learning efficiency can be improved significantly, especially when some of the similarity functions such as edit distance are expensive to compute. The feature table 410 in
Recall that to measure the benefit of ρg(A, θ):B all possible values of θ are tried, and for each θ, the best similarity selection needs to be found on the renamed data. In general, the rename operate should be executed once for each θ value to obtain the renamed data. However, for threshold-based renames, this repeated computation can be avoided by caching similarities between tuples. For this class of rename operators, the benefit of a higher threshold, such as 0.9, can be obtained from the information when the threshold was set at 0.8. Thus, the computed benefit for a small threshold value can be reused for higher threshold.
More formally, a rename operator ρg(A, θ):B is threshold-base if its rename function g(A, θ) is threshold based. g(A, θ) is threshold based if it can be decomposed into two functions: a potential function p(A) that returns a number and a transformation function t(A) that returns a value DOM(B), such that g(A, θ) returns t(A) if p(A)≧θ; otherwise, it returns NULL.
The fuzzy lookup and the FD-correction operators are threshold based renames. The potential function p(A) for a fuzzy lookup is a similarity function that returns similarity between the join attributes. The potential function p(A) for an FD-correction with A→B returns the level of the dominant B value in the A group.
Referring to
As used herein, the terms “component” and “system” and the like are intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to being, a process running on a processor, a processor, an object, an instance, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a computer and the computer can be a component. One or more components may reside within a process and/or thread of execution and a component may be localized on one computer and/or distributed between two or more computers.
The word “exemplary” is used herein to mean serving as an example, instance or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs. Furthermore, examples are provided solely for purposes of clarity and understanding and are not meant to limit the subject innovation or relevant portion thereof in any manner. It is to be appreciated that a myriad of additional or alternate examples could have been presented, but have been omitted for purposes of brevity.
Furthermore, all or portions of the subject innovation may be implemented as a method, apparatus or article of manufacture using standard programming and/or engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computer to implement the disclosed innovation. The term “article of manufacture” as used herein is intended to encompass a computer program accessible from any computer-readable device or media. For example, computer readable media can include but are not limited to magnetic storage devices (e.g., hard disk, floppy disk, magnetic strips . . . ), optical disks (e.g., compact disk (CD), digital versatile disk (DVD) . . . ), smart cards, and flash memory devices (e.g., card, stick, key drive . . . ). Additionally it should be appreciated that a carrier wave can be employed to carry computer-readable electronic data such as those used in transmitting and receiving electronic mail or in accessing a network such as the Internet or a local area network (LAN). Of course, those skilled in the art will recognize many modifications may be made to this configuration without departing from the scope or spirit of the claimed subject matter.
In order to provide a context for the various aspects of the disclosed subject matter,
With reference to
The system memory 1216 includes volatile and nonvolatile memory. The basic input/output system (BIOS), containing the basic routines to transfer information between elements within the computer 1212, such as during start-up, is stored in nonvolatile memory. By way of illustration, and not limitation, nonvolatile memory can include read only memory (ROM). Volatile memory includes random access memory (RAM), which can act as external cache memory to facilitate processing.
Computer 1212 also includes removable/non-removable, volatile/nonvolatile computer storage media.
The computer 1212 also includes one or more interface components 1226 that are communicatively coupled to the bus 1218 and facilitate interaction with the computer 1212. By way of example, the interface component 1226 can be a port (e.g., serial, parallel, PCMCIA, USB, FireWire . . . ) or an interface card (e.g., sound, video, network . . . ) or the like. The interface component 1226 can receive input and provide output (wired or wirelessly). For instance, input can be received from devices including but not limited to, a pointing device such as a mouse, trackball, stylus, touch pad, keyboard, microphone, joystick, game pad, satellite dish, scanner, camera, other computer and the like. Output can also be supplied by the computer 1212 to output device(s) via interface component 1226. Output devices can include displays (e.g. CRT, LCD, plasma . . . ), speakers, printers and other computers, among other things.
The system 1300 includes a communication framework 1350 that can be employed to facilitate communications between the client(s) 1310 and the server(s) 1330. The client(s) 1310 are operatively connected to one or more client data store(s) 1360 that can be employed to store information local to the client(s) 1310. Similarly, the server(s) 1330 are operatively connected to one or more server data store(s) 1340 that can be employed to store information local to the servers 1330.
What has been described above includes examples of aspects of the claimed subject matter. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the claimed subject matter, but one of ordinary skill in the art may recognize that many further combinations and permutations of the disclosed subject matter are possible. Accordingly, the disclosed subject matter is intended to embrace all such alterations, modifications and variations that fall within the spirit and scope of the appended claims. Furthermore, to the extent that the terms “includes,” “has” or “having” or variations in form thereof are used in either the detailed description or the claims, such terms are intended to be inclusive in a manner similar to the term “comprising” as “comprising” is interpreted when employed as a transitional word in a claim.