Single identifier platform for storing entity data

Abstract

Systems and methods are described for generating record clusters. The methods comprise receiving a plurality of records from data sources and providing at least a subset of the records to a scoring model that determines scores for various pairings of the records, a score for a given pair of the records representing a probability that the given pair of records contain data elements about the same entity. The method further comprises generating a graph data structure that includes a plurality of nodes, individual nodes representing a different record from the records. The method also comprises assigning a different unique identifier to individual clusters of the final clusters and responding to a request for data regarding a given entity by providing aggregated data elements from those records of the records associated with a cluster of the final clusters having an identifier that represents the given entity.

Description

BACKGROUND

Field

The present disclosure relates to generating and/or implementing database systems that enable record matching and/or searching of records from a plurality of selectable sources, and implementing a unified identifier database system such that records from various sources can be associated with a single identifier representing with a given entity, such as an individual.

Description of Related Art

Current database systems associate records with particular entity or individual identifiers based on similarity of identifying information. For example, records identifying the same entity (for example, first name, last name, and so forth) may be determined to correspond to the same entity in existing systems based on various predefined rules or heuristics. However, some identifying information in records (for example, phone number, address, and so forth) may be less indicative of the records corresponding to the same entity. Thus, the database systems may have to make decisions regarding which records correspond to the same entity and which do not in the face of varying information that may or may not match. For example, records with matching names and social security numbers may be matched to the same entity while those with matching phone numbers and similar names may not be matched to the same entity. Improved systems, devices, and methods for efficiently and effectively performing entity resolution in databases with billions of records based on varying information between different records and where various rules and/or restrictions regarding matching different records or records from particular sources exist are desired.

SUMMARY

The systems, methods, and devices described herein each have several aspects, no single one of which is solely responsible for its desirable attributes. Without limiting the scope of this disclosure, several non-limiting features will now be described briefly.

Embodiments of the present disclosure relate to a database system (also herein referred to as “the system”) for clustering records from a plurality of discrete sources for one or more individual entities.

In one aspects, a computer-implemented method is disclosed. The method comprises receiving a plurality of data records from one or more data sources and providing at least a subset of the data records to a scoring model that determines scores for various pairings of the data records, a score for a given pair of the data records representing a probability that the given pair of records contain data elements about the same entity. The method also comprises generating a graph data structure that includes a plurality of nodes, individual nodes of the nodes representing a different record from the plurality of data records, where edges between given node pairs are associated with the corresponding determined scores for the respective pairs of records and performing a connected component analysis of the graph data structure, including pruning one or more edges that fall below a threshold score. The method further comprises performing optimal weighted clustering of the graph data structure to determine final clusters of the plurality of nodes, assigning a different unique identifier to individual clusters of the final clusters, where individual identifiers represent different entities, and responding to a request for data regarding a given entity by providing aggregated data elements from those data records of the plurality of data records associated with a cluster of the final clusters having an identifier that represents the given entity.

In some embodiments, the threshold score indicates whether records represented by a node pair of the given node pairs connected by an edge of the edges belong to the same entity as each other, wherein a first given node pair having a first edge of the edges associated with a first score that exceeds the threshold score belong to the same entity as each other, and wherein a second given node pair having a second edge of the edges associated with a second score that is less than the threshold score belong to different entities than each other. In some embodiments, the threshold score varies dynamically according to one or more parameters. In some embodiments, pruning one or more edges that fall below the threshold score comprises removing the one or more edges associated with the score that falls below the threshold score such that one or more of the given node pairs connected by the removed one or more edges are no longer connected via the removed one or more edges. In some embodiments, the scoring model comprises a machine learning algorithm that learns weights associated with different attributes of a portion of the data records to generate the scores for individual pairs of records. In some embodiments, the method further comprises excluding at least one node of the nodes from at least the optimal weighted clustering based on a source of the at least one node according to a rule or restriction that limits combinations of sources of the data records. In some embodiments, the method further comprises merging at least two of the final clusters into a larger cluster based on the at least two of the final clusters being associated with the same entity.

In another aspect, a data aggregation and computation system is disclosed. The system comprises a data store configured to store a plurality of records related to a plurality of individual entities and a hardware processor. The hardware processor is configured to receive a request for clustering the plurality of records according to one or more entity identifiers based on information contained within each of the plurality of records, the request including identifiers for one or more specific data sources from which the plurality of records is sourced and identify one or more entity resolution rules that limit clustering of a given pair of records of the plurality of records based on the respective data sources of the one or more data sources for the given pair. The hardware processor is further configured to perform optimal weighted clustering of the plurality of records in consideration of the one or more entity resolution rules and identify a plurality of clusters of records based on the optimal weighted clustering. The hardware processor is also configured to assign a different unique identifier to individual clusters of the plurality of clusters, where individual identifiers represent different entities of the individual entities and respond to a request for data regarding a given entity of the individual entities by providing aggregated data elements from those records of the plurality of records associated with a cluster of the plurality of clusters having an identifier that represents the given entity.

In some embodiments, the hardware processor is further configured to, in individual clusters of the plurality of clusters, perform a connected component analysis on a set of records of the plurality of records that form the individual cluster and prune records from the set of records that fall below a threshold score, wherein the threshold score indicates whether a pair of records belong to the same entity as each other. In some embodiments, the threshold score varies dynamically according to one or more parameters. In some embodiments, the hardware processor configured to prune comprises the hardware processor configured to remove the records associated with a score that falls below the threshold score from the set of records. In some embodiments, the hardware processor is further configured to apply a scoring model to at least a subset of the plurality of records, wherein the scoring model determines scores for various pairings of the records, a score for a given pair of the records representing a probability that the given pair of records contain data elements about the same entity. In some embodiments, the scoring model comprises a machine learning algorithm that learns weights associated with different attributes of a portion of the records to generate the scores for individual pairs of records. In some embodiments, the hardware processor configured to perform optimal weighted clustering of the plurality of records in consideration of the one or more entity resolution rules comprises the hardware processor configured to exclude at least one record from the optimal weighted clustering based on a source of the at least one record when the one or more entity resolution rules limit combinations of sources of the records in the plurality of clusters. In some embodiments, the hardware processor is further configured to merge at least two of the plurality of clusters into a larger cluster based on the at least two of the plurality clusters being associated with the same individual entity.

In another aspect, a non-transitory computer-readable medium is disclosed. The non-transitory computer-readable medium comprises instructions that, when executed by one or more hardware processors, cause the one or more hardware processors to perform the steps of receiving a plurality of data records from one or more data sources and providing at least a subset of the data records to a scoring model that determines scores for various pairings of the data records, a score for a given pair of the data records representing a probability that the given pair of records contain data elements about the same entity. The instructions further cause the one or more hardware processors to perform the steps of generating a graph data structure that includes a plurality of nodes, individual nodes of the nodes representing a different record from the plurality of data records, where edges between given node pairs are associated with the corresponding determined scores for the respective pairs of records and performing a connected component analysis of the graph data structure, including pruning one or more edges that fall below a threshold score. The instructions also cause the one or more hardware processors to perform the steps of performing optimal weighted clustering of the graph data structure to determine final clusters of the plurality of nodes, assigning a different unique identifier to individual clusters of the final clusters, where individual identifiers represent different entities, and responding to a request for data regarding a given entity by providing aggregated data elements from those data records of the plurality of data records associated with a cluster of the final clusters having an identifier that represents the given entity.

In some embodiments, the threshold score indicates whether records represented by a node pair of the given node pairs connected by an edge of the edges belong to the same entity as each other, wherein a first given node pair having a first edge of the edges associated with a first score that exceeds the threshold score belong to the same entity as each other, and wherein a second given node pair having a second edge of the edges associated with a second score that is less than the threshold score belong to different entities than each other. In some embodiments, the threshold score varies dynamically according to one or more parameters. In some embodiments, pruning one or more edges that fall below the threshold score comprises removing the one or more edges associated with the score that falls below the threshold score such that one or more of the given node pairs connected by the removed one or more edges are no longer connected via the removed one or more edges. In some embodiments, the scoring model comprises a machine learning algorithm that learns weights associated with different attributes of a portion of the data records to generate the scores for individual pairs of records.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart of a method of providing entity data based on records from a plurality of data sources in a database system.

FIG. 2A shows a visual representation of a portion of graph data in a system containing entity records, the visual representation including a plurality of nodes and links, as well as scoring information associated with each link.

FIG. 2B shows a diagram of a visual representation of a portion of graph data in the system containing entity records, the visual representation including four nodes and links and scoring information associated with each link connecting pairs of the nodes.

FIG. 3 shows a visual representation of a portion of graph data in the database containing entity records, including three cluster candidates, each cluster candidate comprising nodes bounded by a different polygon.

FIG. 4 shows the visual representation of FIG. 3 having applied thereto a plurality of regions indicating potential clusters of the nodes.

FIG. 5 shows a flow chart of a process that a processor of the database system may perform to determine the optimal grouping of nodes into clusters of entities.

FIG. 6 shows a visual representation of a portion of graph data in the database containing entity records in a clustered arrangement, where the individual nodes within the various clusters are linked.

FIG. 7 shows a flow chart of a process of using simplex and/or interior point solvers to solve a problem.

FIG. 8 shows an exemplary system architecture for implementing entity resolution as described herein.

FIG. 9 shows a visualization of a portion of graph data in the database containing entity records, the visual representation including two potential clusters and each having records from a number of sources and showing weighted links between related records.

FIG. 10 shows an example of an iterative process of adjusting the weights of different attributes based on different parameters, according to an expectation maximization algorithm described herein.

FIG. 11 shows another visual representation of a portion of graph data in the system containing entity records, the visual representation including three different clusters of entities as identified using the entity resolution methods and systems described herein.

FIG. 12 is a block diagram showing example components of a data processing system.

FIGS. 13A and 13B show how an arrangement of a cluster of nodes and edges may be visualized or represented differently with a focus on the nodes.

FIG. 14 shows a visual representation of a portion of graph data in a system containing entity records, the visual representation including a plurality of nodes and links, as well as scoring information associated with each link.

FIG. 15 shows an example comparison of two personal identification data records.

DETAILED DESCRIPTION

The systems and methods described herein perform entity resolution of records for one or more entities when the records are received from one or more sources of a database system. The entity resolution of the records may involve identifying which records from the sources belong to or are about one entity, such as a single person, and then associating those records with each other and a unique identifier representing the person. An aggregated data record may be stored for the person that includes various information about the person as gathered or received from a potentially large number of different sources, which did not all include a common identifier for the person originally. When a system includes a small number of records, entity resolution may not be a resource intensive process or may be relatively simple to complete. However, as the number of records increase, the entity resolution process may become more resource intensive. In some embodiments, entity resolution may involve examining multiple records from a plurality of sources, where each of the records includes a variety of information that can be analyzed as part of the entity resolution of the records.

In today's data-centric world, large numbers of records may be available for analysis and use in decision-making. In some embodiments, the large numbers of records may include multiple records from the same individual or entity, duplicate records, records with restrictions on use, and so forth. Thus, in addition to simply analyzing the records, database systems may also identify and mark duplicate records (de-duping), identify records that are related to each other or are associated with the same person or entity (for example, entity resolution), and show relationships between records that can be dynamically changed based on any associated record restrictions. The systems and methods described herein provide for such analysis and determinations.

The systems and methods described herein may utilize a graph or similar data structure for storage of the records from the various sources. In the graph data structure example, records from a plurality of sources may each be represented as nodes or vertices. Relationships between nodes are represented by links or edges between individual pairs of nodes. In some embodiments, the links may each comprise information indicating a strength, reliability, ranking, or confidence of the association between the pair of nodes connected by the link.

In a database of graphs with billions of nodes and links (for example, vertices and edges, respectively), each link may have a pre-calculated weight (for example, the information indicating a strength, reliability, ranking, or confidence) indicating a probability that the two nodes coupled by the link actually belong or refer to the same entity (and thus belong to the same cluster). Given such a database of graphs, an optimal clustering maximizes an overall probability of similar nodes being grouped or clustered together. In some embodiments, identifying the optimal clustering of the nodes in the database of graphs may be similar to a non-deterministic polynomial-time hardness or graph partition problem. The methods and systems described herein introduce a novel approach to solve such problems in databases of graphs including such a large number of records. As described below, the novel approach may involve breaking the large database of graphs into a set of smaller graphs or groups of connected nodes and/or other components and applying an optimal weighted clustering algorithm on the smaller groups to determine the final, optimal set of clusters for the entire database of graphs.

In some embodiments, the weights for each link are used to determine whether the link reflects a true relationship or not (for example, to determine whether the edge between two vertices should actually be maintained as an edge or cut). The determination of whether the link represents an actual relationship or not may be made using the weights of each link and comparing the weights to a threshold value. For example, in one embodiment, all links with weights greater than or equal to the threshold may be considered edges or links in a connected component calculation and all links with weights less than the threshold may be determined to not be edges or links. Thus, in such an embodiment, as weights approach 1, corresponding links are attractive (indicating a relationship exists between the nodes) while as weights approach 0, corresponding links are repulsive (indicating no relationship exists between the nodes). The threshold value may vary dynamically or according to one or more parameters. If two nodes are only linked by a repulsive link, then such repulsive link may be used to break up the nodes into subsets without detracting from the optimality of the end result from solving the multiple subsets individually as opposed to the larger set at once.

Based on the nodes, the links, and the information of each of the links, the systems and methods described herein may perform entity resolution. Entity resolution may comprise determining when two or more records are determined to belong to or be related to a single entity. In some embodiments, entity resolution may be based, at least in part, on the information associated with the links between pairs of nodes. For example, when the information corresponds to a ranking or confidence value of the association between the pair of nodes, then the determination whether the two records are associated with the single entity may be made based on whether the information value exceeds a particular threshold. For example, pairs of nodes with ranking or confidence values of greater than 0.8 may be determined to be associated with the single entity. More details regarding entity resolution are provided below.

The systems and methods described herein may use estimated probabilities of pairs of records belonging to a same person to build a weighted graph, where the nodes are the records and the link weights are log-odds of the linked records belonging to the “same person”. The systems and methods may further “cut” or “prune” links based on threshold weights and run an algorithm (for example, a connected component algorithm) to group the nodes into clusters based on the link weights. In some embodiments, for example with large connected components graphs generated from the weighted graph, an additional algorithm (for example, an optimal weighted clustering (OWC) algorithm) is applied to further parse or decompose the graph(s). In some embodiments, for such “same person” probability estimation, a conditional independence (CI) or similar model is generated and trained using an expectation maximization algorithm, as described herein.

Exemplary Term Descriptions

To facilitate an understanding of the systems and methods discussed herein, a number of terms are described below. The terms described below, as well as other terms used herein, should be construed to include the provided descriptions, the ordinary and customary meaning of the terms, and/or any other implied meaning for the respective terms. Thus, the descriptions below do not limit the meaning of these terms, but only provide examples.

Data Store: Includes any computer readable storage medium and/or device (or collection of data storage mediums and/or devices). Examples of data stores include, but are not limited to, optical disks (for example, CD-ROM, DVD-ROM, and so forth), magnetic disks (for example, hard disks, floppy disks, and so forth), memory circuits (for example, solid state drives, random-access memory (“RAM”), and so forth), and/or the like. Another example of a data store is a hosted storage environment that includes a collection of physical data storage devices that may be remotely accessible and may be rapidly provisioned as needed (commonly referred to as “cloud” storage).

Database: Includes any data structure (and/or combinations of multiple data structures) for storing and/or organizing data, including, but not limited to, relational databases (for example, Oracle databases, MySQL databases, and so forth), non-relational databases (for example, NoSQL databases, and so forth), in-memory databases, spreadsheets, as comma separated values (“CSV”) files, eXtendible markup language (“XML”) files, TeXT (“TXT”) files, flat files, spreadsheet files, and/or any other widely used or proprietary format for data storage. Databases are typically stored in one or more data stores. Accordingly, each database referred to herein (for example, in the description herein and/or the figures of the present application) is to be understood as being stored in one or more data stores.

Database Record and/or Record: Includes one or more related data items stored in a database. The one or more related data items making up a record may be related in the database by a common key value and/or common index value, for example.

Entity: depending on the context, may refer to a person, such as an individual, consumer, or customer, and/or may refer to a person having one or more records that are provided to the system and for whom a cluster of records is generated. Thus, the terms “entity,” “individual,” “consumer,” and “customer” should be interpreted to include single persons. Additionally, the terms may be used interchangeably. In some embodiments, the terms refer to an entity other than a person, such as in an embodiment in which the entity for each cluster is a company or other entity represented by a unique identifier in the system and for which data records are aggregated from multiple sources.

A user may generally refer to a party requesting information about the records stored in the database or data store. In some implementations, the party may be an administrative user associated with the system, a credit bureau, a merchant or other provider of goods or services to one or more users, a financial institution, a bank, a credit card company, an individual, a lender, or a company or organization of some other type.

A model may generally refer to a machine learning construct which may be used to automatically generate a result or outcome. A model may be trained. Training a model generally refers to an automated machine learning process to generate the model that accepts an input and provides a result or outcome as an output. A model may be represented as a data structure that identifies, for a given value, one or more correlated values. For example, a data structure may include data indicating one or more categories. In such implementations, the model may be indexed to provide efficient look up and retrieval of category values. In other embodiments, a model may be developed based on statistical or mathematical properties and/or definitions implemented in executable code without necessarily employing machine learning.

A vector encompasses a data structure that can be expressed as an array of values where each value has an assigned position that is associated with another predetermined value. For example, a cost vector will be discussed below. A cost vector may be used to represent the cost associated with a particular column or cluster of values. In some implementations, a vector may be a useful way to provide meaningful analysis regarding the clusters of nodes in a database.

Machine learning generally refers to automated processes by which received data is analyzed to generate and/or update one or more models. Machine learning may include artificial intelligence such as neural networks, genetic algorithms, clustering, or the like. Machine learning may be performed using a training set of data. The training data may be used to generate the model that best characterizes a feature of interest using the training data. In some implementations, the class of features may be identified before training. In such instances, the model may be trained to provide outputs most closely resembling the target class of features. In some implementations, no prior knowledge may be available for training the data. In such instances, the model may discover new relationships for the provided training data. Such relationships may include similarities between data elements such as personal identifying information.

Example Clustering Platforms

FIG. 1 is a flowchart of a method 111 of providing entity data based on records from a plurality of data sources in a database system. The method 111 comprises a plurality of blocks or steps performed by one or more processors or systems for analyzing data. In some embodiments, the blocks or steps of method 111 are performed by a system that includes a database of graphs as described herein.

The method 111 begins at block 110 with receiving a plurality of data records from one or more data sources. In some embodiments, receiving the data records comprises receiving data from data sources as described in further detail below. Once the records are received and stored, the method 111 proceeds to block 112, where the processor or system provides at least a subset of the data records to a scoring model that determines scores for various pairings of the data records, the score for a given pair of the data records representing a probability that the pair of records contain data elements about the same entity. Thus, at block 112, the processor or system identifies the weights or values for links between pairs of data records. In some embodiments, the block 112 comprises applying the expectation-maximization (“EM”) (or similar) algorithm described in further detail below. In other embodiments, the score may be based at least in part on applying one or more supervised, semi-supervised, or unsupervised machine learning algorithms that include training a model to learn weights associated with each of a number of different attribute types associated with a group of training records. For example, the unsupervised learning algorithm may comprise an EM algorithm where no known labels are used. The semi-supervised learning algorithm comprises the unsupervised EM algorithm that is extended by allowing the incorporation of some labeled dataset to guide the derivation of the model. In some embodiments, the EM algorithm is used to calculate the linkage probability as a linear combination of the matching features. Therefore, it allows the system to explain why a data record is linked to the others data records.

The method 111 proceeds to block 114, where the processor or system generates a graph data structure that includes a plurality of nodes, each of the nodes representing a different data record from the plurality of data records received in block 110, where the edges between given node pairs are associated with the corresponding score determined for the represented pair of data records (such as a score determined for the given pair of records at block 112). Example visualizations of the graph data structure are provided in the Figures and described below with reference to at least FIGS. 2A and 2B, among others. At block 116, the method 111 comprises performing a connected component analysis of the graph data structure, including pruning one or more edges that fall below a threshold score. Details of an example connected component analysis is provided below. At block 118, the method 111 comprises performing optimal weighted clustering of the graph data structure to determine final clusters of the plurality of nodes. Optimal weighted clustering may comprise an extensive algorithm to ensure that all nodes are placed into appropriate clusters, as described in more detail below. At block 120, the method 111 comprises assigning a different unique identifier to each cluster, where each identifier represents a different entity. In some embodiments, the same unique identifier will apply to all records associated with an entity due to the entity resolution performed as described herein. At block 122, the method 111 responds to a request for data for a given entity based on the clusters generated and the information contained therein, as described herein.

For example, the request may be from a requesting party (such as an external computing system or an application) for a data profile or entity data regarding a specific person having a certain identifier. Responding to the request at block 122 may include finding the cluster associated with the certain identifier and returning a set of data pulled from the data records represented by nodes in that cluster. In some embodiments, as will be described below, the graph may have been generated subsequent to receiving the request in a manner that is specific to the requesting party and the request, such that data governance rules and thresholds may have been applied that were specific to a matching strictness of the requesting party and/or rules associated with the use of data from individual data providers.

FIG. 2A shows a visual representation 100 of a portion of graph data in a database of graphs containing entity records, the visual representation 100 including a plurality of nodes 102 and links 104, as well as scoring information 106 associated with each link 104. The visual representation 100 includes four nodes 102a-102d and links 104 coupling individual node pairs. The links 104 include the scoring information 106 that indicates a score representing a probability that the information or data record represented by each of the paired nodes is about the same individual or entity. For example, the node 102a includes the name “John Smith” while the node 102b includes the name “John Smith Sr.”. The link 104 connects the nodes 102a and 102b and the information 106 for the link 104 shows that the two nodes 102a and 102b share a phone number and that the shared phone number results in a score (for example, likelihood or probability of similarity) of 0.7. The visual representation 100 shows that examples of other shared information between pairs of nodes 102 may have greater or lesser likelihoods or probabilities of similarity. For example, when two nodes 102 share a social security number (SSN), the likelihood of similarity is 0.95 while when two nodes share an address, the likelihood of similarity is 0.6. As will be discussed below, the actual scores may be determined in a more involved manner, but are described in a simplified manner with respect to FIG. 2A to discuss below certain graph-based or clustering problems that may arise after the scores are determined.

As described above, the large number of nodes of the visual representation 100 may be grouped into a plurality or set of clusters such that each individual node 102 is part of only one cluster. One way of performing such clustering may integrate optimal weighted clustering, as described in more detail below. The optimal weighted clustering may greatly simplify solving of an integer linear programming (ILP) problem having a million or more constraints that would otherwise be impractical to solve, for example using brute force. Though the optimal weighted clustering described herein is applied to a database of graphs for use in entity resolution, the optimal weighted clustering may be applied to various other database applications not necessarily described herein.

As described above, the visual representation 100 shows how different records from different sources may be linked based on shared or similar attributes or values. The visual representation 100 also shows how such shared attributes may result in likelihoods or probabilities of similarities that may represent a confidence level of the link. For example, the nodes 102a and 102b share the same phone number and have a 0.7 likelihood of similarity or confidence level. The 0.7 likelihood of similarity or confidence level indicates that the nodes 102a and 102b are more likely to indicate or represent the same people as compared to nodes having a smaller numerical likelihood of similarity or confidence level. Similarly, nodes 102 having larger likelihood of similarity or confidence levels than 0.7 are more likely to indicate or represent the same people as compared to nodes 102 having the 0.7 or smaller levels.

Pair-wise scoring may be used to perform entity resolution between pairs of nodes 102. For example, based on the 0.8 and 0.95 likelihood of similarity or confidence levels, the corresponding nodes (for example, the three individual nodes) may be determined to belong to or be about the same entity. Nodes 102 coupled with likelihood of similarity or confidence levels that are less than 0.8 may not necessarily be determined to belong to the same entity. However, as will be described in further detail below, looking only at these likelihood of similarity or confidence levels may generate contradicting information regarding shared records between entities. For example, the nodes 102d (“John Smith Jr.”) and 102b (“John Smith Sr.”) are linked with no indicated level, which suggests that the link does not meet the threshold indicating that they belong to the same entity. However, nodes 102d and 102a are linked with a level of 0.6, meeting the threshold indicating that the nodes 102d and 102a belong to the same entity and the nodes 102a and 102b are linked with a level of 0.7, also meeting the threshold indicating that the nodes 102b and 102a belong to the same entity. This means that node 102b and 102d are linked (implicitly) even though the direct link does not meet the desired threshold. Thus, even though 102d and 102b may have a higher likelihood of representing different entities, the entity resolution based purely on the links, nodes, and corresponding information may be insufficient to produce accurate results of entity record grouping or clustering. Accordingly, a comprehensive algorithm for generating an optimal grouping of nodes 102 considering all the links 104 is further described below. Such an algorithm may consider or compensate for the implicit associations that appear to contradict weighted links.

FIG. 2B shows a diagram of a visual representation 200 of a portion of graph data in the database of graphs containing entity records, the visual representation 200 including four nodes 201-204 and links and scoring information associated with each link connecting pairs of the nodes. The four nodes 201, 202, 203, and 204 are coupled by links that include a different representation of the relationships (negative logit of probability scores). As such, negative values for the links may represent favorable groupings for the single entity, while positive weights suggest different entities. The link between nodes 202 and 204 indicate a high penalty or positive weight (1000) that would mathematically prevent node 202 from being grouped with node 204 into the same entity cluster. For example, if nodes 202 and 204 represented nodes 102b and 102d of FIG. 2A, the high positive weight between the nodes 202 and 204 would suggest that those two nodes are not the same entity.

The visual representation 200 may be represented by an edge-weight matrix Φ. The matrix Φ may be defined as one of an upper triangular matrix, a lower triangular matrix, a symmetric matrix, and so forth. The matrix Φ may include the nodes as rows and columns where the weights of the links pairing the nodes correspond to the values in the matrix for the different row/column pairings where the matrix Φ, for each subsequent row, only shows weights not shown in any previous row.

A matrix G, also shown in FIG. 2B, includes as each column one possible cluster from the visual representation 200. For example, each row in the matrix G represents one of the nodes 201-204. A value of “1” in the matrix G indicates that the respective node 201-204 is part of the cluster identified by the particular column. For example, when multiple nodes 201-204 have a value of “1” for a particular column, those nodes are part of the same cluster identified by that particular column. The matrix G provides all possible clusters for the visual representation 200. As there are four nodes, there are 16 possible combinations of the nodes, so matrix G includes 16 columns.

Based on the matrix G, a cost vector Gamma (Γ) can be defined. The cost vector Γ has 16 columns like the matrix G. Thus, each column in the cost vector Γ may correspond to each column (or possible cluster) of the matrix G. Each column of the cost vector Γ may represent a sum of edge weights of the corresponding cluster of the matrix G. For example, the first 3 columns include no edges (only zero or one node for each of the first three columns, so no links or edges included), so the cost for the first three columns is 0. However, the 4^th column includes, for example, nodes 201 and 202 and thus represents the sum of the weights for the links or edges between the nodes 201 and 202 with no other links to be included (−2.7 as shown in the visual representation 200). The last column of the cost vector Γ includes all nodes 201-204 and, thus, corresponds to the sum of all edges, having a value of 994.

Based on the matrices and vector described above, for the corresponding visual representation 200, an optimal cost vector Γ is determined (for example, by a processor of the database system). The optimal cost vector Γ may provide an optimal combination of clusters from the 16 cluster candidates of the matrix G that minimizes the total cost while ensuring that no node 201-204 was chosen more than once.

A value of “1” in the optimal cost vector Γ means that the corresponding cluster from the matrix G is selected while the value of “0” means the corresponding cluster from the matrix G is not selected. This method of identifying optimal clusters may be resource and computation demanding as the size of the graph grows, because the computations involved grow exponentially as the size of graph grows. If the number of nodes in the graph as N, then the size of G matrix would be N×2^N, the size of cost vector Γ would be 2^N. Such an algorithm involving the matrices and vectors described above may be impractical for a graph with 20 nodes, for example. However, the algorithm and corresponding computations may be simplified by generating the matrix G and the optimal cost vector Γ corresponding to only possible clusters (for example, clusters that do not group nodes 202 and 204 together).

For example, as described above, such a problem may be interpreted as an ILP problem, as described above. However, based on the optimal weighted clustering described herein, the ILP problem may be reduced to a linear programming (IL) problem to which column generation is applied. For example, if the graph includes 20 nodes, instead of having to solve the problem in view of the 1+ million potential clusters (2²⁰ potential clusters), the optimal weighted clustering algorithm described herein instead starts from zero clusters and iteratively increases the number of clusters until the optimal clustering is determined for the graph. Such an algorithm may be applied to the graph of any size and similarly reduce processing times for determining the optimal weighted clustering for that graph. In some embodiments, the optimal weighted clustering algorithm may identify the optimal clustered solution for the 20 node graph by considering a fraction (for example, between 10-20 constraints) of the total 1+ million constraints.

In one example algorithm, the processor associated with the database of graphs may search for and/or generate a cluster of the nodes 201-204 from the database of graphs 200 that will potentially improve the overall objective. In an interaction of the search/generation process, if a new cluster is found, it is added to the matrix G and the vector Γ. If no new cluster is added for the iteration, then the global set of optimal clusters has been found. Further details of this algorithm are provided below.

For example, in the visual representation 300 of FIG. 3 showing a portion of graph data in the database of graphs containing entity records, three cluster candidates are shown, each cluster of nodes bounded by a different polygon 301, 302, and 303. Each of the nodes shown in the visual representation are assigned or comprise node values of 0.5, 0.5, 2.5, and 0.0 for the nodes 201-204, respectively. In some embodiments, these node values may be statically or dynamically assigned. In some embodiments, the node values may be associated or related to a previously determined and/or assigned link weight. Based on the analysis provided above, the polygon 301 has a cost of −3.5 (−0.3+−0.5+−2.7), the polygon 302 has a cost of −2.7, and the polygon 303 has a cost of −2.5. The two clusters 301 and 302 may be shown in matrix G′ and the cost vector Γ′ shown in FIG. 3. In simplified processing to determine the minimum cost vector Γ′, each row of the matrix G′ may correspond to a potential cluster.

As described above, the original problem (for example, the ILP problem) of identifying clusters in the visual representation 300 may be quantified by Equation #1.1 below:

$\begin{array}{cc}\underset{\gamma \in {\left\{0,1\right\}}^{2^N}}{\min }\Gamma ·\gamma s.t.G·\gamma \le 1& \mathrm{EQUATION}\mathrm{\#1}\text{.1}\end{array}$ Where:

• • Γ is the cost vector defined asΓ_q=ω+G_q^T·Θ+G_q^T·Φ·G_q   EQUATION #1.1.0Where:
  • ω is a uniform cluster offset, Θ is a 1*N vector, representing the cost of including each of the N nodes in the problem, and Φ is a N*N matrix that represents the edge cost of including 2 nodes simultaneously in the same cluster;
  • G is a N*2^N binary matrix consisting of all the possible clusters; and
  • γ is the selection vector to represent which ones of the 2N candidate clusters will be chosen.

Additional constraints can be imposed to confirm the validity of columns in G. The constraints can be added in 3 ways:

• • 1. Add more constraints into Equation #1.1.
  • 2. Consider only valid cluster/column in the “Column Generation” step. For example, the longest chain length is limited to be no more than 4-hops by considering clusters that are entirely within 2-hops from some node, called a “radius constraint”.
  • 3. Introduce an infinite cost to the objective function in Equation #1.1. For example, some pair of the nodes should never appear in one cluster, though they may be linked indirectly through other nodes. By setting the cost on the edge between those pairs to be infinitely high (i.e. Φ_ij=∞), those nodes having the high cost edges are not clustered together.

The above described constraints can reduce the number of valid columns to be considered in G, but it may still be an intractable number, which calls for the column generation algorithm to help. Equation #1.1 may be rewritten into the following form to consider only the valid clusters

$\begin{array}{cc}\underset{{\gamma \in \{0,1)}^{\hat{N}}}{\min }\hat{\Gamma }·\gamma s.t.\hat{G}·\gamma \le 1& \mathrm{EQUATION}\mathrm{\#1}\text{.1}\text{.1}\end{array}$ Where:

• • {circumflex over (N)} is the number of all valid clusters, which may be very close to 2N. Ĝ is a N×{circumflex over (N)} matrix consisting of all the valid clusters. And r is still the corresponding cost vector of the N valid clusters.
  • A column generation algorithm (outlined in further detail below) is implemented to solve this problem.

The Equation #1.2 below may represent the simplified LP problem corresponding to the ILP problem. Thus, Equation #1.2 may represent the processing required to solve the simplified LP problem:

$\begin{array}{cc}\underset{\gamma \in R^{\hat{N}}}{\min }\hat{\Gamma }·\gamma s.t.\hat{G}·\gamma \le 1& \mathrm{EQUATION}\mathrm{\#1}\text{.2}\end{array}$

The Equation #1.3 represents a dual problem corresponding to the LP problem:

$\begin{array}{cc}\underset{\lambda \in R^N\mathrm{and}\lambda \ge 0}{\max }-1·\lambda s.t.\Gamma +G^T·\lambda \ge 0& \mathrm{EQUATION}\mathrm{\#1}\text{.3}\end{array}$

The dual problem represented by Equation #1.3 may have the same optimal value as the primal problem (Equation #1.2), but involve reduced analysis to determine. In the primal problem, the processor or system optimizes a 2N-dimensional space with N constraints each having 2N terms. In the dual problem, the processor or system optimizes a N-dimensional space with 2N constraints each having N terms. Only by investigating from the dual side, the processor or system can decompose the 2N columns of G into 2N separate constraints and only consider a subset of the constraints.

When using a subset of K columns (for example, a matrix G′ (N×K) and cost vector Γ′ (1×K)), the LP Equation #1.2 can be reduced to Equation #1.4 and the equivalent dual equation (Equation #1.5) below. The processor or system may solve Equations #1.4 and 1.5 to solve the smaller LP problem and the corresponding equivalent dual problem for the subset of columns:

$\begin{array}{cc}\underset{\gamma \in R^K}{\min }{\Gamma }^{\prime }·\gamma s.t.G^{\prime }·\gamma \le 1& \mathrm{EQUATION}\mathrm{\#1}\text{.4}\end{array}$ $\begin{array}{cc}\underset{\lambda \in R^N\mathrm{and}\lambda \ge 0}{\max }-1·\lambda s.t.{\Gamma }^{\prime }+G^{\mathrm{\prime T}}·\lambda \ge 0& \mathrm{EQUATION}\mathrm{\#1}\text{.5}\end{array}$

• • Once an optimal solution γ, λ is determined for the equivalent dual problem, a new column/cluster that can potentially make the reduced problem closer to the full problem is determined by solving Equation #1.6.

$\begin{array}{cc}\overline{\Psi }=\underset{q\in \left\{1{\mathrm{\dots 2}}^N\right\}}{\min }{\Gamma }_q+{\left(G_q\right)}^T·\stackrel{\_}{\lambda }& \mathrm{EQUATION}\mathrm{\#1}\text{.6}\end{array}$ Where:

• • Γ_q is the qth element of the cost vector Γ, which is also the corresponding cost of the qth column of G.

Equation #1.6 may represent a quadratic problem (see definition of Γ (Equation #1.1.0)). However, considering that the elements in G_q are all binary, the quadratic problem represented by Equation #1.6 can be converted to a binary linear programming by introducing new variables y_ij to represent whether the edge between node i and node j is included. Constraints on y_ij ensure that y_ij=1 if and only if both node i and node j are included in the cluster or grouping (G_qi=1 and G_qj=1).

If Ψ<0, then the corresponding qth column of G is the most violated row regarding the dual constraint in the full problem. As such, the qth column can make the reduced problem closer to the full problem. The resulting new column will be included in the next iteration. Thus, this step is called “column generation step”.

If Ψ≥0, then the constraint for the full problem is also satisfied by this solution. Thus it is also the optimal for full problem.

In order to introduce the 2-hop constraint mentioned above, the minimization problem in the entire graph is broken into a minimization problem in all neighbourhoods of the graph, which is defined as a graph including all nodes within 2-hops of some node. Thus, every new column generated is a valid column satisfying the radius constraint when these radius constraint is maintained. For example, FIG. 4, which shows the visual representation of FIG. 3 having applied thereto a plurality of regions 402, 404, and 406 indicating potential clusters of the nodes, illustrates that the processor or system breaks the whole problem into 3 subproblems based on a similar 1-hop constraint. The yellow region 406 is the portion of the graph including two nodes that are 1-hop from node 202, the blue region is the portion of the graph 1-hop from node 201, and node 202, and the green region is the portion of the graph including all nodes that are 1-hop from nodes 202 and 203.

In summary, the whole iterative process, as illustrated in FIG. 5, shows a flow chart of the process that the processor of the database system may perform to determine the optimal grouping of nodes into clusters of entities, works as described below:

• • 1. The matrix G′ and the cost vector Γ′ start empty and λ=0.
  • 2. In each iteration
    • a. For each radius-constrained subproblem, find the most violated constraint/column/cluster in that neighbourhood
    • b. C=total number of new columns generated
    • c. If C>0:
      • i. Update G′, Γ′ with the new columns and solve the global LP (Equation #1.4 and #1.5) to update λ and γ
    • d. Else:
      • i. Converged, break out of the loop
  • 3. The final clusters will be the ones chosen by the final y vector.

FIG. 6 shows a visual representation of a portion of graph data in the database containing entity records in a clustered arrangement, where the individual nodes within the various clusters are linked. In some embodiments, for example with graph data similar to that shown in FIG. 6, processing all of the displayed nodes as described above may result in nodes ABHIJ being identified as belonging to one cluster, even though these nodes are not actually connected, because there is no restriction that prevents such an outcome of the process of optimization described above. Adjusting the cluster offset (ω) and node offset (Θ) in Eq.1.1.0 may provide for determination of the nodes clustered into groups to which they are actually connected. For example, if the cluster offset is negative and the node offset is positive, then the unconnected clusters will automatically break into smaller connected clusters because of the lower cost.

A robust LP solver may be used to solve the LP problem. There are 2 main algorithms for solving the LP problems, (1) an interior point solver and (2) a simplex solver. These 2 algorithms may give different solutions when there are degenerate solutions. Though the degenerate solutions are mathematically equivalent in terms of the objective of LP, in the column generation framework, solutions from the interior point solver are preferable because these solutions can lead to faster convergence, i.e., fewer iterations to converge. However, existing interior point solvers are not robust enough to handle all the cases. Accordingly, an backup simplex solver is added to the workflow in case the interior point solver fails, as illustrated in FIG. 7, which shows a flow chart of a process of using simplex and/or interior point solvers to solve a problem.

FIG. 8 shows an exemplary system architecture 800 for implementing entity resolution as described herein. The system 800 includes a plurality of data sources, distribution components, data storage components, components that perform analytics, components that perform services, and application components. The system 800 may provide a unified application program interface (API) that interfaces with the various application components. The application components may have specific accuracy/coverage requirements that are provided by the system 800. The system 800 may support deterministic and probabilistic linkages and/or provide connectivity to various record sources while tracking and limiting access to the various records according to one or more restrictions.

The data sources may include a variety of sources, with examples shown in FIG. 8 with respect to one embodiment. For example, the data sources may include a credit bureau that provides credit based entity records for one or more entities. The data sources also includes a marketing services group that provides records relating to marketing data received for various entities. The data sources also include identity data records, for example as sourced from fraud and identification sources. In some embodiments, the data sources include business services data records from one or more business services sources. The data sources also includes automobile records data received from one or more automobile data sources. The data sources may include device data regarding device fraud verification records. The data sources may also include data from consumer services providers. In some embodiments, the record matching and entity resolution described herein may involve rules that limit how data from different data sources is analyzed and/or combined. For example, rules regarding the records from the credit bureau source may limit use with records from, for example, the identity data source or the consumer services data source. In some embodiments, the rules may result in records from one or more data sources being excluded from the entity matching or resolution process. In some embodiments, excluding one data source from the entity resolution may result in one or more links being “broken” in the entity resolution, which may cause certain other records to not be considered part of a cluster. For example, if a consumer services data record is linking an automobile data record to a cluster and the consumer services data records cannot be used in entity resolution, then the automobile data record may no longer be part of the cluster because the consumer services data records are removed.

The applications may provide support to clients and/or provide access to records from one or more of the data sources. The APIs may support real-time inquiry and batch record processing. In some embodiments, the APIs may comprise the inquiry and data management service components. The entity resolution component may be a graph-based solution that links similar record identifiers and/or other information based on a desired certainty level. The entity resolution component may be one of the analysis components. The search & match analysis component may be a machine learning based scoring component that determines a likelihood that different identifiers for different records are matched. The search & match component may be one of the analysis components. The data distribution components may parse, standardize, and enrich diverse data sources and records therefrom. In some embodiments, the data distribution components may be limited according to the one or more restrictions. Governance & security components may provide specific configurations to control how data is used in the entity resolution. For example, the security, audit, and data governance and access components may be one of the governance & security components.

The database of graphs generated and utilized by the system 800 may be used for or have applied thereto two-phase searching and matching of records received from sources. For example, as noted above, the database of graphs used by the system 800 may represent relationships between records having identifiers with a probabilities or likelihood weight that provides for arbitrary matching features. The database of graphs may then be used for machine learning based analysis and entity resolution (unsupervised or semi-supervised machine learning) based on an expectation-maximization (EM) algorithm. In some embodiments, such database of graphs based entity resolution maintains probabilistic relationships between existing identifiers in the database of graphs and may not be impacted by ordering of records, unlike some existing pinning methods of entity resolution. Additionally, all links may be rescored as needed or at desirable intervals, and so forth. Additionally, in the database of graphs, excluding particular data sources and/or records may be easily performed if needed. Finally, various aspects of the entity resolution using the database of graphs may be adjusted to accomplish a trade-off between accuracy and coverage.

FIG. 9 shows a visualization 900 of a portion of graph data in the database of graphs containing entity records, the visual representation 900 including two potential clusters 902 and 904 each having data records from a number of sources and showing weighted links between related records. In some embodiments, each of the data records may have an identifier, for example an identifier assigned by the source from which the data records are received. In some embodiments, the machine-learning search and match algorithm described above determines the probability of a pair of records in the clusters 902 and 904 referring to the same entity.

As shown, the links within the clusters 902 and 904 each link pairing nodes has a weight value assigned thereto. Additionally, each link pairing nodes has a line weight or thickness that generally corresponds to the associated weight value assigned to that link. For example, links with higher weight values (for example, 0.7 or greater) have larger thickness or line weight than links with lower weight values (for example, 0.6 or lower). As such, either the thickness or weight value can be representative of the likelihood or confidence of the paired nodes being associated with the same entity.

Each of the nodes or records that are linked within the cluster 902 are associated with a single entity, for example ExpID*-1. Similarly, each of the records linked within the cluster 904 are associated with entity ExpID*-2. Each of the nodes may be sourced from one or more particular data sources. For example, the nodes of the cluster 904 come from the “credit bureau data”, “marketing services data”, and “identity” sources. The nodes of the cluster 902 come from the “credit bureau data”, “marketing services data”, and “identity”” sources as well. By being aware and tracking the sources of the records and nodes, specific sources (and thus, records/nodes) can be dynamically excluded from entity resolution and/or other analysis based on corresponding rules, restrictions, and so forth, as will be discussed in further detail below. For example, if a rule or restriction prevents records from the “marketing services data” and “identity” sources being used together, then the clusters 902 and 904 may be dynamically updated to exclude the nodes from the “marketing services data” and “identity” sources, which may have corresponding effects of reducing the number of nodes in the clusters 902 and 904.

In some scenario, the exclusion of certain sources can cause the previously assigned clusters to be no-longer connected with each other. For example, if node (EMSID-8, corresponding to a marketing services data node) and node (CISID-5, corresponding to a credit bureau data node) has to be excluded, then cluster 902 may be divided into an upper cluster and a lower cluster with different cluster IDs. Thus, for each run, of the system or processor (for example, the system 1200 or processor 1205 described below), for example, expectation-maximization is performed only once to derive the linkage probabilities. Then, depending on how each application requires which sources are allowed, the system or processor (for example, the system 1200 or processor 1205 described below) may perform the clustering (connected component and OWC) multiple times. Furthermore, depending on whether each performance focuses on the coverage or accuracy, different configuration are used in the OWC, hence the super-script l′ in the Exp{circumflex over ( )}k−1. This may reduces the run-time of the system or processor by leveraging the same linkage probability calculation and may allow flexibility needed by different applications.

In some embodiments, entity resolution as described herein may involve applying the expectation-maximization algorithm. For example, two records received from the same or different sources may include the following information:

Record #1
Name:    John Doe
Address: 123 Main St.
Phone:   858-321-4567
DOB:     Feb. 1, 1960
Record #2
Name:    Jon Doe
Address: 123 Main St.
Phone:   321-4567
DOB:     N/A

Some matching algorithms may apply rules or other heuristics to the provided information to determine whether or not the records refer to the same person or entity. For example, the similarity in names may indicate a “+5” weight, the matching address may indicate a “+20” weight, the similarity in phone numbers may indicate a “+8” weight, and the lack of matching DOB may indicate a “−5” weight. Given these weights or values, these two records may be found to indicate a match.

However, when additional information for each of the records was included (for example, an e-mail address in record #1 and an Internet address in record #2), the records may not continue to match according to the applied algorithm. Thus, a static algorithm may be unable to compensate for differences in records such as this. Furthermore, as most of the entity resolution problems, obtaining enough data with labels on whether the records are matched or not is extremely difficult. Therefore, it is difficult even for a domain expert to derive efficient rules without trial-and-error. It is also difficult to apply any supervised machine learning algorithm to learn a model to estimate the probability. Therefore, the expectation-maximization (EM) algorithm that does not require any labeled data can be applied to adapt to differences in records. For example, the EM algorithm applied may identify that the data in each record is structured and not random and identify informative features in the data which yield consistency in the observed similarities and differences over a variety of records. The EM algorithm may apply such capabilities and develop models with the weighting information to determine matches between records with improved accuracy, efficiency, and confidence.

The EM algorithm (or a similar algorithm) may learn a set of weights or similar parameters that maximize a total log likelihood of observed matching patterns according to Equation #2 below:

$\begin{array}{cc}\sum _n\log p\left(X^{\left(n\right)};\theta \right)& \mathrm{EQUATION}\mathrm{\#2}\end{array}$ Where X⁽ⁿ⁾ represents the observed matching patterns in the records.

For example, the EM or similar algorithm, applied by a processor or similar component of the system 800, determines a probability of the nth record pair being a match according to Equation #3:P(Z⁽ⁿ⁾=M|X⁽ⁿ⁾;θ)   EQUATION #3Where θ represents the current parameters or weights used in the matching determination. In some embodiments, the probability generated by the EM or similar algorithm may correspond to the weights shown by the links between pairs of nodes, for example the weights shown in the visual representations of graph data.

The EM or similar algorithm may include updating the applied parameters or weights based on an estimated probability that maximizes the log likelihood according to Equations #4-6 below:

$\begin{array}{cc}{m}_{\mathrm{ij}}=\frac{{\sum }_{n}P\left(Z=M|X^{\left(n\right)};\theta \right)1\left(X_i^{\left(n\right)}=j\right)}{{\sum }_{n}P\left(Z=M|X^{\left(n\right)};\theta \right)}& \mathrm{EQUATION}\mathrm{\#4}\end{array}$ $\begin{array}{cc}{u}_{ij}=\frac{{\sum }_{n}P\left(Z=U|X^{\left(n\right)};\theta \right)1\left(X_i^{\left(n\right)}=j\right)}{{\sum }_{n}P\left(Z=U|X^{\left(n\right)};\theta \right)}& \mathrm{EQUATION}\mathrm{\#5}\end{array}$ $\begin{array}{cc}p=\frac{{\sum }_{n}P\left(Z=M|X^{\left(n\right)};\theta \right)}{N}& \mathrm{EQUATION}\mathrm{\#6}\end{array}$ Where, the probabilities of Equations #4-6 are applied to the parameters or weights θ. Based on Equations #4-6, the system 800 applying the algorithm may pair over 200 million records (generate over 100 million record pairs) in approximately one minute, in one embodiment.

FIG. 10 shows an example of an iterative process of adjusting the weights of different attributes based on different parameters, according to the expectation maximization algorithm described herein. The expectation maximization initializes the weights of individual features randomly, which produces prediction using Equation #3 on whether pairs of records are matched or not. Based on the consistency in the prediction, it revised the estimation of the weights of the parameters using Equation #4-5. The process iterates until it converges with no changes in the parameters

In the embodiment, the expectation-maximization algorithm is enhanced by allowing the initialization of the weights of the individual features in Equation #4-5 using a limited set of labeled records that are collected when such data is available.

In some embodiments, additional optimizations may be implemented in conjunction with the optimized weighted clustering algorithm described above. For example, in some embodiments, the original clustering problem (Equation #1.1) can be converted into an equivalent multi-cut (or pruning) problem. Instead of selecting node clusters, in the multi-cut problem representation, edges that need to be cut (for example, pruned) are identified and/or selected. There are an exponential number of constraints to ensure that by cutting the selected edges, the graph will be cut into one or more clusters so that no edges within clusters are cut and all between-clusters edges are cut.

The formula of the multi-cut problem is

$\begin{array}{cc}\underset{x_{d_1{d}_2\in {\left\{0,1\right\}}^{❘ℰ❘}}}{\min }\sum _{\left(d_1,d_2\right)\in ℰ}{\varphi }_{d_1{d}_2}{x}_{d_1{d}_2}s.t.\sum _{e\in c\backslash \left\{e_0\right\}}{x}_e\ge x_{e_0},\forall e_0\in c^{-},c\in 𝒞& \mathrm{EQUATION}7.\end{array}$ Where:

• • d∈D: the set of nodes in the graph
  • (d₁,d₂)∈ε: the set of edges in the graph, indexed by the nodes
  • x_d1,x_d2∈{0,1}. An indicator if the edge between d₁,d₂ is cut. x_d1d2=1 indicate the edge between d₁,d₂ is cut, which also means node d₁ and node d₂ are in different clusters
  • ϕ_d1d2: the cost of creating a boundary (i.e. cutting the edge) between d₁ and d₂, which is essentially the opposite of Φ_d1d2 in the column generation representation.
  • c∈C: set of edge cycles that contains at least one edge with negative weight

$\underset{e\in c}{\min }{\varphi }_e<0$

• • c⁻⊂c: the set of edges in c that has a negative weight

The constraint is basically stating for any edge cycles, there are either no cuts at all or at least 2 cuts in the circle, otherwise the cut will be inside a cluster. In some embodiments, since only at least one edge with a negative weight is cut, constraints on only the negative edges may be used.

For example, in FIG. 14, cycle ACEDA, CDEC are all from set C, but cycle ABCA is not, because there is no negative edge on that cycle. The corresponding constraint may recite, in cycle CDEC for example, that cutting edge DE will involve cutting either edge CD or edge CE also.

The objective function in Equation #7.0 can be rewritten into

$\underset{x_{d_1{d}_2}\in {\left\{0,1\right\}}^{❘ℰ❘}}{\min }\sum _{\left(d_1,d_2\right)\in ℰ}{\varphi }_{d_1{d}_2}{x}_{d_1{d}_2}=\underset{x_{d_1{d}_2}\in {\left\{0,1\right\}}^{❘\epsilon ❘}}{\min }\left[\sum _{\left(d_1,d_2\right)\in {ℰ}^{+}}{\varphi }_{d_1{d}_2}{x}_{d_1{d}_2}+\sum _{\left(d_1,d_2\right)\in {ℰ}^{-}}{\varphi }_{d_1{d}_2}{x}_{d_1{d}_2}\right]=\sum _{e={ℰ}^{-}}{\varphi }_e+\underset{x_e\in {\left\{0,1\right\}}^{❘ℰ❘}}{\min }\left[\sum _{e\in {ℰ}^{+}}{\varphi }_e{x}_e-\sum _{e\in {ℰ}^{-}}{\varphi }_e\left(1-x_e\right)\right]=\sum _{e\in ℰ}{\varphi }_e^{-}+\underset{x_e\in {\left\{0,1\right\}}^{❘ℰ❘}}{\min }\sum _{e\in ℰ}\left[{\varphi }_e^{+}{x}_e-{\varphi }_e^{-}\left(1-x_e\right)\right]$ Where:

• • ε⁺={e|e∈ε and ϕ_e≥0}, ε⁻={e|e∈ε and ϕ_e<0}, and
  • ϕ_e⁺=max(0,ϕ_e), ϕ_e⁻=min(0,ϕ_e)

Obviously, the complexity of this problem resides mainly in the constraints. Writing/enforcing the exponential number of cycle inequalities may be avoided by utilizing the idea of Bender's decomposition.

A term Σ_n∈NQ_n(x) introduced into the objective function of Equation #2.1 may enforce the constraints.

Here n∈ is a set of nodes that whose connections cover all negative edges. For example, in FIG. 14, can be set {A, E} or set {B,D} or set {A, B, D} or {A,B,C,D,E}. In order to make the problem as simple as possible, is selected to have the minimum size. But choosing the minimum is a well-known NP-hard problem (min-set cover). In some embodiments, a greedy algorithm is implemented to choose a set with “almost” the smallest size.

The basic idea is to use a latent cluster label m_d to enforce the edge to be cut even though it may not be selected to cut by x_d1d2 in the master problem. In each subproblem n, m_n=0.

Then:

• • ε_n⁺ is the set of all the positive edges attached to node n
  • ε_n⁻ is the set of all the negative edges attached to node nand

$Q_n\left(x\right)=\underset{f\ge 0}{\min }\left[\sum _{d_1{d}_2\in {ℰ}^{+}\setminus {ℰ}_n^{+}}{\varphi }_{d_1{d}_2}^{+}{f}_{d_1{d}_2}+\sum _{n{d}_1\in {ℰ}_n^{+}}{\varphi }_{n{d}_1}^{+}{h}_{d_1}^{+}-\sum _{n{d}_1\in {ℰ}_n^{-}}{\varphi }_{{\mathrm{nd}}_1}^{-}{h}_{d_1}^{-}\right]$ So that:

• • If m₁≠m₂, then either (d₁,d₂) is cut in the master problem already (x_d1d2=1), or it is cut in the subproblem (f_d1d2=1)
  • If x_nd=1 for a negative edge (n,d)∈ε_n⁻, then either label m_d=1 or edge is uncut (h_nd⁻=1)
  • If m_d=1 and edge (n d)∈ε_n⁺, then either the edge is cut in master (x_nd=1), or it is cut in the subproblem (h_nd⁺=1)

Q_n(x) is an LP problem, with x_d1d2 appearing only in the constraints. Thus, its corresponding dual problem is an LP problem with x_d1d2 appearing only in the objective function, whose optimal value is still

$Q_n\left(x\right)=\underset{{\omega }^{\left(n\right)}}{\max }\left({\omega }_0^{\left(n\right)}+{\sum }_{e\in ℰ}{x}_e{\omega }_e^{\left(n\right)}\right),$ with some additional constraints on the dual variables ω.

Now it is converted into a standard form for Benders' decomposition with some additional linear constraints on ω,

$\begin{array}{cc}\underset{x_{d_1{d}_2}\in {\left\{0,1\right\}}^{|ℰ❘}}{\min }\left[\sum _{e\in ℰ}{\Omega }_e{x}_e+\sum _n\underset{{\omega }^{\left(n\right)}}{\max }\left({\omega }_0^{\left(n\right)}+\sum _{e\in ℰ}{x}_e{\omega }_e^{\left(n\right)}\right)\right]& \mathrm{EQUATION}7.2\end{array}$

Here, Σ_eΩ_ex_e represents the complicated expression on EQUATION #7.1. Benders approach can be applied by the processor or system to solve this problem.

Because the maximum of Q_n(x) must happen at some extreme point of the feasible region of ω, Equation 7.2 can be rewritten into Equation 7.3 below:

$\begin{array}{cc}\underset{x_{d_1{d}_2\in {\left\{0,1\right\}}^{❘ℰ❘}}}{\min }\left[\sum _{e\in ℰ}{\Omega }_e{x}_e+\sum _n{q}_n\right]\mathrm{with}& \mathrm{EQUATION}7.3\end{array}$ $q_n\ge {\left({\omega }_z^{\left(n\right)}\right)}_0+{\sum }_{e\in ℰ}{x_e\left({\omega }_z^{\left(n\right)}\right)}_e,\forall z\in {𝒵}^{\left(n\right)}$

Here z∈Z⁽ⁿ⁾ is the set of extreme points of the feasible region of ω⁽ⁿ⁾. In Bender's approach, the extreme point z is added one by one. In some embodiments, a number of extreme points to consider may be reduced by selecting the extreme points appropriately.

Such an approach may avoid fractional solutions caused by loose relaxation from Equation #1.1 to Equation #1.2.

Additional Optimizations

In some embodiments, if numerous columns exist for the optimal cluster candidate, non-essential columns may be identified and removed. Criteria for the “non-essential” columns may be flexible. For example, the columns that are not selected in this primal solution, or the columns that have not been selected in the last 10 iterations.

In some embodiments, the neighborhoods created for simplified analysis may be too large for efficient analysis and may be further separated or divided into sub-neighborhoods. Additionally, results of iterations may be reused to improve efficiencies and enable faster solving of various problems, such as the pricing problem.

Ensuring Consistent Cluster ID when Updating the Data.

In some embodiment, the cluster ID that matched to the inquiry from clients may be returned. It is important to ensure the same inquiry or very similar inquiries will still return the same cluster ID as much as possible when the data records in the database of graphs has changed. For example, when new nodes are added or retired nodes are removed, or the node content has been updated (resulting in the edges around the node being updated), results from inquiries or requests should generally return consistent results.

Sometimes the best way to assign a new cluster ID occurs when merging clusters. For example, when merging clusters shown in FIG. 11, merging clusters into the larger original cluster may be a default method of merging clusters. However, in some embodiments, like the update in FIGS. 13A and 13B, merging clusters may be much more complicated and there may not be one assignment that is clearly better than others.

In some embodiments, a 2-step solution is implemented to systematically assign the cluster IDs consistently.

In the first step, the possible candidates to inherit the cluster ID are grouped together. The grouped together nodes may be viewed in 2 different snapshots as hyper-nodes and the edges as hyper-edges. The hyper-nodes in 2 snapshots that share the same nodes will be conceived as connected by a hyper-edge, as illustrated in FIGS. 13A and 13B. In some embodiment, the clusters connected by the hyper edges can be considered as a possible heir to inherit the cluster ID. By running a connected component iteration or analysis on this graph of hyper-nodes and hyper-edges, each group of the clusters is considered together. The non-trivial part here is to pull those 4 clusters and around 15 nodes together from potentially billions of nodes, which can be solved by applying a connected component algorithm when viewed as hyper-nodes and hyper-edges.

In the second step, the cluster ID is assigned systematically within each group, with the following process:

• • 1. Calculate the jaccard similarity index between each new cluster-old cluster pair, with jaccard score defined as Equation 8 below:

$\begin{array}{cc}{S}_{jaccard}=\frac{\left|\left\{\mathrm{nodes}\mathrm{in}\mathrm{new}\mathrm{cluster}\right\}\bigcap \left\{\mathrm{nodes}\mathrm{in}\mathrm{old}\mathrm{cluster}\right\}\right|}{\left|\left\{\mathrm{nodes}\mathrm{in}\mathrm{new}\mathrm{cluster}\right\}\bigcup \left\{\mathrm{nodes}\mathrm{in}\mathrm{old}\mathrm{cluster}\right\}\right|}& \mathrm{EQUATION}8\end{array}$

• • 2. match old clusters and new clusters using max bipartite matching algorithm. (greedy approach can also be applied here to reduce computation cost by sacrificing some optimality of the result.)
  • 3. If there is any new cluster left, generating a new ID by hash the update timestamp and the nodes ID together, so that there is only infinitesimal probability that the newly generated ID would collide with any other existing cluster IDs.

By doing so, the highest total jaccard score between the new and old clusters with the same cluster ID is obtained, or an approximate one is obtained via greedy approach.

Example System Implementation and Architecture

FIG. 12 is a block diagram showing example components of a data processing system 1200. The system 1200 or variations thereof may be used, in some embodiments, as part of the database systems for databases 100, 200, 300, 400, 600, 800, and 1000 or that performs the process 500 or embodies the system architecture 700. The processing system 1200 includes, for example, a personal computer that is IBM, Macintosh, or Linux/Unix compatible or a server or workstation. In one embodiment, the processing system 1200 includes a server, a laptop computer, a smart phone, a personal digital assistant, a kiosk, or a media player, for example. In one embodiment, the processing system 1200 includes one or more central processing unit (“CPU”) 1205, which may each include a conventional or proprietary microprocessor specially configured to perform, in whole or in part, one or more of the machine learning recommendation/result model features described above. The processing system 1200 further includes one or more memory 1232, such as random access memory (“RAM”) for temporary storage of information, one or more read only memory (“ROM”) for permanent storage of information, and one or more mass storage device 1222, such as a hard drive, diskette, solid state drive, or optical media storage device. A specially architected database of graphs 1208 may be provided. The database of graphs 1208 may be optimized for storing records as nodes or vertices and storing relationships between the records as links or edges having confidence weights or similar information, as described above. In some implementations, the database of graphs 1208 may be designed to handle large quantities of data and provide fast retrieval of the records and fast determination of relationships between records. In some embodiments, the database of graphs 1208 may dynamically include and/or exclude specific records or records from specific sources based on one or more rules.

Typically, the components of the processing system 1200 are connected using a standards-based bus system 1290. In different embodiments, the standards-based bus system 1290 could be implemented in Peripheral Component Interconnect (“PCI”), Microchannel, Small Computer System Interface (“SCSI”), Industrial Standard Architecture (“ISA”) and Extended ISA (“EISA”) architectures, for example. In addition, the functionality provided for in the components and modules of processing system 1200 may be combined into fewer components and modules or further separated into additional components and modules.

The processing system 1200 is generally controlled and coordinated by operating system software, such as Windows XP, Windows Vista, Windows 7, Windows 8, Windows Server, Unix, Linux, SunOS, Solaris, iOS, Blackberry OS, Android, or other compatible operating systems. In Macintosh systems, the operating system may be any available operating system, such as MAC OS X. In other embodiments, the processing system 1200 may be controlled by a proprietary operating system. The operating system is configured to control and schedule computer processes for execution, perform memory management, provide file system, networking, I/O services, and provide a user interface, such as a graphical user interface (“GUI”), among other things.

The processing system 1200 may include one or more commonly available input/output (I/O) devices and interfaces 1212, such as a keyboard, mouse, touchpad, and printer. In one embodiment, the I/O devices and interfaces 1212 include one or more display devices, such as a monitor, that allows the visual presentation of data to a user. More particularly, a display device provides for the presentation of GUIs, application software data, and multimedia presentations, for example. The processing system 1200 may also include one or more multimedia devices 1242, such as speakers, video cards, graphics accelerators, and microphones, for example.

In the embodiment of FIG. 12, the I/O devices and interfaces 1212 provide a communication interface to various external devices. The processing system 1200 may be electronically coupled to one or more networks, which comprise one or more of a LAN, WAN, cellular network, satellite network, and/or the Internet, for example, via a wired, wireless, or combination of wired and wireless, communication link. The networks communicate with various computing devices and/or other electronic devices via wired or wireless communication links, such as the credit bureau data source and financial information data sources.

In some embodiments, information may be provided to the processing system 1200 over a network from one or more data sources. The data sources may include one or more internal and/or external data sources that provide transaction data, such as credit issuers (e.g., financial institutions that issue credit cards), transaction processors (e.g., entities that process credit card swipes at points of sale), and/or transaction aggregators. The data sources may include internal and external data sources which store, for example, credit bureau data and/or other data associated with individual consumers or other entities. In some embodiments, one or more of the databases or data sources may be implemented using a relational database, such as Sybase, Oracle, CodeBase and Microsoft® SQL Server as well as other types of databases such as, for example, a flat file database, an entity-relationship database, and object-oriented database, and/or a record-based database.

In general, the word “module,” as used herein, refers to logic embodied in hardware or firmware, or to a collection of software instructions, possibly having entry and exit points, written in a programming language, such as, for example, Java, Lua, C or C++. A software module may be compiled and linked into an executable program, installed in a dynamic link library, or may be written in an interpreted programming language such as, for example, BASIC, Perl, or Python. It will be appreciated that software modules may be callable from other modules or from themselves, and/or may be invoked in response to detected events or interrupts. Software modules configured for execution on computing devices may be provided on a computer readable medium, such as a compact disc, digital video disc, flash drive, or any other tangible medium. Such software code may be stored, partially or fully, on a memory device of the executing computing device, such as the processing system 1200, for execution by the computing device. Software instructions may be embedded in firmware, such as an EPROM. It will be further appreciated that hardware modules may be comprised of connected logic units, such as gates and flip-flops, and/or may be comprised of programmable units, such as programmable gate arrays or processors. The modules described herein are preferably implemented as software modules. They may be represented in hardware or firmware. Generally, the modules described herein refer to logical modules that may be combined with other modules or divided into sub-modules despite their physical organization or storage.

In the example of FIG. 12, the modules 1210 may be configured for execution by the CPU 1205 to perform, in whole or in part, any or all of the process or systems discussed above, such as those described above with respect to FIGS. 1-11 and FIGS. 13A-15.

Identity Searching and Matching

As described above, a single person may be associated with a number of records. Associating the number of records with that single person may involve applying one or more searching and/or matching models to the records. For example, the system 1200 may generate and/or apply the searching and/or matching models to the records, for example in the database of graphs 1208, to the single person. An initial step may comprise the system 1200 generating a subset of candidate identity records that most likely belong to the single person, for example by applying one or more searching models. Once the subset of candidate identify records is generated, the system 1200 may apply one or more matching models (for example, unsupervised statistical models) to evaluate a probability of “same-personness” between an identity record and the subset of candidate identity records. In some embodiments, the modules 1210 of the system 1200 includes a searching module that applies the one or more searching models and a matching module that applies the one or more matching models. In some embodiments, the CPU 1205 applies the corresponding models and/or performs one or more actions for or instructed by the searching and/or matching modules.

In some embodiments, the system 1200 may train the one or more matching models using a model training method that is designed to produce explainable matching model. In some embodiments or use cases, the model is expandable to enable operation with and/or integration with various regulations and compliance requirements.

Model Training

In some embodiments, a personal identity record comprises a plurality of fields of information. For example, the personal identity record may include one or more of a name field, address field, driver's license field, phone number(s) field, email address field, social security number field, Internet protocol (IP) address field, mobile advertiser identity (MAID) field, and so forth. The process or method of matching one record including one or more of these fields with other records including one or more of these fields may involve the system 1200 employing a conditional, independent matching model to evaluate the probability of “same personness” for a pair of candidate identity records. In some embodiments, the matching model(s) perform matching of the pair of candidate identity records on an element-by-element analysis of each of the fields. For example, the name field for the pair of candidate identity records includes four elements: first name, middle name, last name, and generation indicator.

Before the candidate identity records are compared, the records may be pre-processed (for example, by the CPU 1205 or another processing system. Such pre-processing may comprise parsing the name, address, e-mail, and other fields in the records. The pre-processing may also comprise standardizing the records into a common format (for example, with given or expected fields).

For each element of the respective field, the name fields for the pair of candidate identity records may be compared, for example using string comparison or some similar matching process. In some embodiments, the matching is a comparison of the full name field, a subset of the characters of the name field (for example, the first X characters), and so forth. The matching process may be performed by one or more of the CPU 1205, an element of the matching model, and so forth and result in a categorical output. For example, when analyzing a first name element of the name field for the pair of candidate identity records, the following examples are four possibilities of the comparison:

• • (Bob, Bob)→exact_match (0)
  • (Bob, Bobby)→partial_match (1)
  • (Bob, Robert)→alias_match (2)
  • (Bob, B)→fst_char_match (3)

Each element comparison or matching may be defined as matchers X₁, X₂, . . . , X_K, assuming a total of K elements that are compared between the pair of personal identity records. The matchers may be observable values, for example, indicating whether corresponding fields in two records match (or how much they match) for the machine learning/modeling (which may be unsupervised or semi-supervised) approach using expectation-maximization (EM) algorithm to train the model. For purposes of the matching model, these elements may be handled as random variables (r.v.), and more specifically, be identified as independent variables. The weights associated with matching fields in the pair of personal identity records (for example, the +50, +20, +8, −5 the “John Doe” and “Jon Doe” records introduced above) may be parameters for the EM algorithm. Binary random variables, Y, that indicate whether a given pair of identity records belong to the same person are identified as dependent variables, or hidden variables of the EM algorithm. The system 1200 applying the matching model described herein may be consistent, data driven, easily adaptable to data changes and/or format changes, and flexible to accommodate data variation, such as missing elements between records or missing fields.

FIG. 15 shows an example comparison of two personal identification data records. The two records “Bob M. Smith Jr” and “Robert K. Smith” are compared, with results from the comparison of the pair of records highlighted in the table below the records. As shown, the first names “Bob” and “Robert” are a “Nickname Match” while the middle names “M.” and “K.” are in “Conflict” and the last names “Smith” are an “Exact Match”. Additionally, the street IDs are an “Exact Match” and the Unit ID is “One Missing”.

The matching model applied by the system 1200 may be a generative model with dependent variable Y being latent (for example, unobservable) and X₁, X₂, . . . , X_K, being independent given Y. Equation 9.1 below provides an example of the conditional independence (CI) assumption probability identified by the matching model described herein:P(X₁, . . . ,X_K|Y)=P(X₁|Y)*P(X₂|Y)* . . . *P(X_K|Y)   EQUATION 9.1P(X₁, . . . ,X_k, . . . ,X_K|Y)=Π_kP(X_k|Y)   EQUATION 9.1.1

Training of the matching model described herein may involve estimating parameters, for example:

• • m_k:(m_k,1, . . . , m_k,j, . . . , m_k,|Xk|), Σ_jm_k,j=1, where m_{k,j}≙P(X_k=j|Y=1)
  • u_k:(u_k,1, . . . , u_k,j, . . . , u_k,|Xk|), Σ_ju_k,j=1, where u_{k,j}≙P(X_k=j|Y=0)
  • p≙P(Y=1)Where k=1, . . . , K, j=1, . . . , |X_k|.CI may be applied as a generative model with a random draw y_i based on p. Based on y_i=1 (or 0), independently draw X_{k,k=1 . . . K} based on m (or u).

Evaluating the matching model may involve calculating a conditional probability based on Equation 9.2:P(Y|X₁=x₁,X₂=x₂, . . . ,X_K==x_k)   EQUATION 9.2

The CPU 1205 may apply a Bayesian theorem based on the parameters (m,u,p) estimated based on the above training step and output from matchers x₁, x₂, . . . , x_K. The training method is described in more detail below.

In some embodiments, the matching model applied by the CPU 1205 may achieve explainability based on Equation 9.3, which may be a model evaluation formula:

$\begin{array}{cc}P\left(Y=1|X_1=x_1,X_2=x_2,\dots ,X_K==x_k\right)=\frac{P\left(X_1=x_1,X_2=x_2,\dots ,X_K=x_k,|Y=1\right)*P\left(Y=1\right)}{P\left(X_1=x_1,X_2=x_2,\dots ,X_K=x_k\right)}=\frac{(P\left(X_1=x_1❘Y=1\right)*P\left(X_2=x_2❘Y=1\right)*\dots *P\left(X_K=x_k❘Y=1\right)*P\left(Y=1\right)}{P\left(X_1=x_1,X_2=x_2,\dots ,X_k=x_k\right)}& \mathrm{EQUATION}9.3\end{array}$

In log-odd space, the explainability based on Equation 9.3 may be presented by Equation 9.4:

$\begin{array}{cc}\log \frac{P\left(Y=1|X_1=x_1,\dots ,X_K=x_K\right)}{P\left(Y=0|X_1=x_1,\dots ,X_K=x_K\right)}=\log \frac{P\left(X_1=x_1❘Y=1\right)*\dots *P\left(X_K=x_K❘Y=1\right)*P\left(Y=1\right)}{P\left(X_1=x_1❘Y=0\right)*\dots *P\left(X_K=x_K❘Y=0\right)*P\left(Y=0\right)}=\log \frac{P\left(X_1=x_1❘Y=1\right)}{P\left(X_1=x_1❘Y=0\right)}+\dots +\log \frac{P\left(X_K=x_K❘Y=1\right)}{P\left(X_K=x_k❘Y=0\right)}+\log \frac{P\left(Y=1\right)}{P\left(Y=0\right)}={\sum }_k\log \frac{P\left(x_k❘Y=1\right)}{P\left(x_k❘Y=0\right)}+\log \frac{P\left(Y=1\right)}{P\left(Y=0\right)}& \mathrm{EQUATION}9.4\end{array}$

Equation 9.4 may be further simplified in terms of the parameters (m,u,p) as Equation 9.5:

$\begin{array}{cc}\log \frac{m_{\left\{1,x_1\right\}}}{u_{\left\{1,x_1\right\}}}+\dots +\log \frac{m_{\left\{K,x_K\right\}}}{u_{\left\{K,x_K\right\}}}+\log \frac{p}{1-p}& \mathrm{EQUATION}9.5\end{array}$ ${\sum }_k\log \frac{m_{k,x_k}}{u_{k,x_k}}+\log \frac{p}{1-p}$

Accordingly, the log of odd ratio (log-odd) of the target variable Y given an observation point (x₁, x₂, . . . , x_K), is a summation of the base (unconditional) log-odd:

$\log \frac{p}{1-p}$

The log-odds of matching results from the identity elements may be:

$\log \frac{m_{\left\{k,x_k\right\}}}{u_{\left\{k,x_k\right\}}},k=1,\dots ,K$

For example, assuming that the pair of identity records contains the following identity elements having the corresponding matchers and furnished log-odds:

• • First Name (X₁)
    • Exact_match
      • label: 0
      • log_odd: 2.1
    • Partial_match
      • Label: 1
      • Log_odd: 1.1
    • Nickname_match
      • Label: 2
      • Log_odd: 1.3
    • Not match (else)
      • Label: 3
      • Log_odd: −0.5
  • Last Name (X₂)
    • Exact_match
      • Label: 0
      • Log_odd: 1.5
    • Partial match
      • Label: 1
      • Log_odd: 1.1
    • Not match (else)
      • Label: 2
      • Log_odd: −0.2
      • Phone Number (X3)
    • Exact_match
      • Label: 0
      • Log_odd: 0.8
    • Plus_minus_1
      • Label: 1
      • Log_odd: 0.5
    • Not match (else)
      • Label: 2
      • Log_odd: −0.3
  • Assume base log-odd to be 0.

Based on these parameters, and the matching model Equations described above, the log_odd for an exemplary pair of identity records (Bob, Kelly, 8588119023) vs (Robert, Kelly 8588119022) may operate as follows:

• • 1. Obtain matching result
    • a. X₁=2, first name: nickname_match
    • b. X₂=0, last name: exact_match
    • c. X₃=1, phone number: plus/minus_1
  • 2. Sum up corresponding log-odds based on matching result, which results in log-odd of target Y

$\log \frac{P\left(Y=1❘X1=2,X2=0,X3=1\right)}{P\left(Y=0❘X1=2,X2=0,X3=1\right)}=0+1.3+1.5+0.5=3.3$

• • 3. Convert log-odds to probability

$P\left(Y=1❘X1=2,X2=0,X3=1\right)=\frac{e^{3.3}}{1+e^{3.3}}=0.8997$

From the above example, the log-odds of matching cases may act as a weight

• • They are combined linearly
  • Between different matchers and different matching outcome of same matcher, they indicate relative importance in determine “same person-ness” probability.

As such, the characterizers may form a basis of explainability.

The likelihood function described herein may be maximized using one or more parameters, such as (m, u, p), Equations 9.6 may approximate how the parameters (m, u, p) best fit to maximize the likelihood function:

$\begin{array}{cc}& \mathrm{EQUATION}9.6\end{array}$ $L\left(X=x;m,u,p\right)=E_{\left(Y^i,i=1..n\right)}L\left(X=x,Y;m,u,p\right)=E_{\left(Y^i,i=1¨n\right)}p\left(X=x,Y❘m,u,p\right)=E_{\left(Y^i,i=1..n\right)}\prod _{i=1}^{n}p\left(x_1^i,¨,x_k^i,¨,x_K^i,Y^i\right)=E_{\left(Y^i,i=1..n\right)}\text{⁠}\prod _{i=1}^{n}p\left(\left(x_1^i,¨,x_k^i,¨,x_K^i\right)|Y^i\right)p\left(Y^i\right)=E_{\left(Y^i,i=1..n\right)}\prod _{i=1}^n\left[p\left(Y^i\right)\prod _{k}p\left(x_1^i❘Y^i\right)\right]$

Due to latent variables Yⁱ, optimizing the likelihood function may be difficult. The system 1200 may apply an iterative algorithm (for example, the EM algorithm) to find maximum likelihood estimates of parameters in models that depend on latent variables.

In some embodiments, weights may be controlled using monotonicity. The matching model as described above may be trained using an expectation-maximization (EM) algorithm in a purely data driven-fashion. In some embodiments, the log-odds/weights obtained do not satisfy one or more requirements that manifest in terms of monotonicity. As such, the EM may replace the maximization step, which solves an unconstrained optimization problem analytically, with constrained optimization by numerically solving an optimization problem that respects monotonicity constraint as follows, based on the EM training algorithm provided below.

• • Expectation-Maximization (EM) trainer for our matching model
    • Initialize m⁰, u⁰, p⁰
    • At step t
  • Compute cond. prob. of latent variable p^t(Yⁱ|xⁱ) based on m^t−1, u^t−, p^t−1, using, for example, Bayes' theorem by taking p^t−1 as prior See model evaluation formula
    • Update m^t, u^t, p^t based on p^t(Yⁱ|xⁱ)

$p^t=\frac{{\sum }_i{p}^t\left(Y^i=1❘x^i\right)}{n}$ $m_{k,j}^c=\frac{{\sum }_{i:x_k^i=j}{p}^t\left(Y^i=1❘\left(x_1^i,¨,x_K^i\right)\right)}{{\sum }_i{p}^t\left(Y^i=1\right)},j=1..❘X_k❘$ $u_{k,j}^t=\frac{{\sum }_{i:x_k^i=j}{p}^t\left(Y^i=0❘\left(x_1^i,¨,x_K^i\right)\right)}{{\sum }_i{p}^t\left(Y^i=0\right)},j=1..❘X_k❘$

• • • • x_kⁱ=j: j observed for kth feature on ith instance

In general, the EM algorithm may include a plurality of steps: expectation steps (E-Steps) and maximization steps (M-Steps). The E-step may be defined as Q(θ|θ^t), which may be an expected value of a log likelihood function of θ, with respect to a current conditional distribution of Z given X and current estimates of the parameters θ^(t), according to Equation 9.7.1 below. The M-step may be defined as Q(θ^t+1)=arg_θ max Q(θ|θ^t), according to Equation 9.7.2 below. The E-step and M-step equations may be applied to the likelihood function described herein based on θ: m, u, p and Z: Yⁱ, i=1 . . . n.Q(θ|θ^t)=E_(Z|X,θ(t))[log[L(θ;X,Z)]θ^(t+1)=arg_θ max Q(θ|θ^t)   EQUATIONS 9.7.1-2

As described herein, marginalization over the latent variable Y makes computing and/or optimizing the EM algorithm difficult. On the other hand, unmarginalized likelihood function L(X, Y; m, u, p) may make computing and/or optimizing the EM algorithm more easy, especially when based on the CI assumption, Equation 9.7.3, and the log of the unmarginalized likelihood function, Equation 9.7.4:

$\begin{array}{cc}& \mathrm{EQUATIONS}9.7\text{.3}-4\end{array}$ ${\prod }_{i=1}^{n}p\left(x_1^i,¨,x_K^i,Y^i\right)={\prod }_{i=1}^{n}p\left(x_1^i|Y^i\right)*¨*p\left(x_K^i❘Y^i\right)p\left(Y^i\right)$ $\mathrm{Log}\left[L\left(X,Y;m,u,p\right)\right]={\sum }_{i=1}^n\left\{{\sum }_k\log \left(p\left(x_k^i❘Y^i\right)\right)+\log \left(p\left(Y^i\right)\right)\right\}$

Given the estimated parameters m^t−1, u_t−1, p^t−1 described above, where

• • m_k:(m_k,1, . . . , m_k,j, . . . , m_k,|Xk|), Σ_jm_k,j=1, where m_{k,j}≙P(X_k=j|Y=1)
  • u_k:(u_k,1, . . . , u_k,j, . . . , u_k,|Xk|), Σ_ju_k,j=1, where u_{k,j}ϑP(X_k=j|Y=0)
  • p≙P(Y=1),

For each data instance xⁱ, p^t(Yⁱ|Xⁱ) is estimated based on applying, for example, Bayes Theorem:

$p_i^t\stackrel{\Delta }{=}{p}^t\left(Y^i=1❘x^i\right)=\frac{{p\left(x^i|Y^i=1\right)}_p\left(Y^i=1\right)}{p\left(x^i\right)}=\frac{p\left(x^i❘Y^i=1\right)p\left(Y^i=1\right)}{p\left(x^i❘Y^i=1\right)p\left(Y^i=1\right)+p\left(x^i❘Y^i=0\right)p\left(Y^i=0\right)}$

As previously discussed, monotonicity on log-odds (weight) solve a constrained optimization problem in each M-step instead of an unconstrained EM training as shown below.

• • Initialize m⁰, u⁰, p⁰
  • At step t
    • Compute Q(θ|θ^t−1) with expectation taken w.r.t. p^t(Yⁱ|xⁱ) as

$Q\left(\theta |{\theta }^{t-1}\right)=E_{p^t\left(Y^i|x^i\right)}\mathrm{Log}\left[L\left(X,Y;m,u,p\right)\right]={\sum }_{i=1}^n\left\{{\sum }_k{p}^t\left(Y^i❘x^i\right)\log \left(p\left(x_k^i|Y^i\right)\right)+p^t\left(Y^i❘x^i\right)\log \left(p\left(Y^i\right)\right)\right\}={\sum }_{i=1}^n\left\{{\sum }_k{p}_i^t\log \left(p\left(x_k^i|Y^i=1\right)\right)+p_i^t\log \left(p\left(Y^i=1\right)\right)+\left(1-p_i^t\right)\log \left(p\left(x_k^i|Y^i=0\right)\right)+\left(1-p_i^t\right)\log \left(p\left(Y^i=0\right)\right)\right\}={\sum }_{i=1}^n{\sum }_k{p}_i^t\log \left(p\left(x_k^i|Y^i=1\right)\right)+{\sum }_{i=1}^n{\sum }_k\left(1-p_i^t\right)\log \left(p\left(x_k^i|Y^i=0\right)\right)+{\sum }_{i=1}^n\left\{p_i^t\log \left(p\left(Y^i=1\right)\right)+\left(1-p_i^t\right)\log \left(p\left(Y^i=0\right)\right)\right\}$ where θ and θ^t−1 refers, collectively, to the model parameters (m,u,p) and (m,u,p)^t−1.

$\begin{array}{c}{\sum }_{i=1}^n{\sum }_k{p}_i^t\log \left(p\left(x_k^i|Y^i=1\right)\right)\mathrm{is}m;\\ {\sum }_{i=1}^n{\sum }_k\left(1-p_i^t\right)\log \left(p\left(x_k^i|Y^i=0\right)\right)\mathrm{is}u;\mathrm{and}\\ {\sum }_{i=1}^n\left\{p_i^t\log \left(p\left(Y^i=1\right)\right)+\left(1-p_i^t\right)\log \left(p\left(Y^i=0\right)\right)\right\}\mathrm{is}p.\end{array}$

• • Note that Q is a function of m and u. In EM, an optimizer (m* and u*) of Q is obtained analytically so that it yields m and u update formula above. In the constrained EM, the Q function is optimized numerically. In other words, m^t and u^t are updated by solving following constrained optimization problemmax_{m,u}Q(θ|θ^t−1)=E_pt(Yi|xi)Log[L(X,Y;m,u,p)]
  • such that
    • c₁(m,u)
    • c₂(m,u)
    • . . .
    • c_t(m,u)
  • Where c_{{1,2, . . . ,t}} are constraints assuming a total t constraints.

Further details regarding the EM algorithm described herein are provided below. p may be updated as follows:

$Q\left(m,u,p|m^{t-1},u^{c-1},p^{t-1}\right)={\sum }_{i=1}^n\left\{{\sum }_k{p}_i^t\log \left(p\left(x_k^i|Y^i=1\right)\right)+\left(1-p_i^t\right)\log (p\left(x_k^i|Y^i=0\right)\right\}+{\sum }_{i=1}^n\left\{p_i^t\log \left(p\right)+\left(1-p_i^t\right)\log \left(1-p\right)\right\}={\sum }_{i=1}^n\left\{{\sum }_k{p}_i^t\log \left(p\left(x_k^i|Y^i=1\right)\right)+\left(1-p_i^t\right)\log (p\left(x_k^i|Y^i=0\right)\right\}+\log \left(p\right)\left({\sum }_{i=1}^n{p}_i^t\right)+\log \left(1-p\right)\left({\sum }_{i=1}^n\left(1-p_i^t\right)\right)={\sum }_{i=1}^n\left\{{\sum }_k{p}_i^t\log \left(p\left(x_k^i|Y^i=1\right)\right)+\left(1-p_i^t\right)\log (p\left(x_k^i|Y^i=0\right)\right\}+n*\left[p_0\log \left(p\right)+\left(1-p_0\right)\log \left(1-p\right)\right],$ $p_0=\frac{{\sum }_i{p}_i^t}{n}\arg {\max }_p{p}_0\log \left(p\right)+\left(1-p_0\right)\log \left(1-p\right)\to p^{*}=p_0=\frac{{\sum }_i{p}_i^t}{n}$

• • Jensen's inequality: E[log(X)]≤log(E[X])
  • Apply Jensen's inequality to obtain Gibbs' inequality
    • P₀ log(p)+(1−p₀)log(1−p)≤p₀ log(p₀)+(1−p₀)log(1−p₀) for any p
      • Take R.V. X with dist.

$X=\frac{p}{p_0}w·p·p_0$

• • • •  and X=(1−p)/(1−p₀) w.p. 1−p₀

$E\left[\log \left(X\right)\right]=p_0\log \left(\frac{p}{p_0}\right)+\left(1-p_0\right)\log \left(\frac{1-p}{1-p_0}\right)\le \log \left(E\left[X\right]\right)=\log \left(p_0*\frac{p}{p_0}+\left(1-p_0\right)*\frac{1-p}{1-p_0}\right)=\log 1=0$

• • • m and u may be updated as follows:
  • Use exactly same approach, maximizing Q against m and u

$Q\left(m,u,p|m^{t-1},u^{t-1},p^{t-1}\right)={\sum }_{i=1}^n\left\{{\sum }_k{p}_i^t\log \left(p\left(x_k^i|Y^i=1\right)\right)+\left(1-p_i^t\right)\log \left(p\left(x_k^i|Y^i=0\right)\right)\right\}+p_0\log \left(p\right)+\left(1-p_0\right)\log \left(1-p\right)$

Which yields an update formula of:

$m_{k,j}^t=\frac{{\sum }_{i:x_k^i=j}{p}_i^t}{{\sum }_i{p}_i^t},j=1..❘X_k❘,k=1..K$ $u_{k,j}^t=\frac{{\sum }_{i:x_k^i=j}\left(1-p_i^t\right)}{{\sum }_i\left(1-p_i^t\right)},j=1..❘X_k❘,k=1..K$ Where p_i^t: tag!,

• • All known: Naive Bayes (supervised)
  • Some known: Semi-supervised

For explainability and compliance, monotonicity constraints, for example, for first name matcher (X₁) may be imposed to ensure:

• • caeteris paribus (all other things being equal), two records with first name exact match should be weighted no lower than two records with first name only partially match.
  • caeteris paribus, two records with first name partial match should be weighted no lower than two records with one's first name is a nickname of the other's first name (nickname match).
  • caeteris paribus, two records with first nickname match should be weighted no lower than two records whose first names do not match: not exact match, not partial match, not nickname match).

Because the matching model is additive on log-odds, the above requirements may be represented mathematically, leveraging a monotonicity increasing property of the logarithmic function:

$\frac{m_{\left\{1,0\right\}}}{u_{\left\{1,0\right\}}}\ge \frac{m_{\left\{1,1\right\}}}{u_{\left\{1,1\right\}}}$ $\frac{m_{\left\{1,1\right\}}}{u_{\left\{1,1\right\}}}\ge \frac{m_{\left\{1,2\right\}}}{u_{\left\{1,2\right\}}}$ $\frac{m_{\left\{1,2\right\}}}{u_{\left\{1,2\right\}}}\ge \frac{m_{\left\{1,3\right\}}}{u_{\left\{1,3\right\}}}$ Where:

• • m_{1,0}:=P(X₁=0|Y=1):=P(first name exact match|Y=1),
  • u_{1,0}:=P(X₁=0|Y=1):=P(first name exact match|Y=0)
  • m_{1,1}:=P(X₁=1|Y=1):=P(first name partial match|Y=1)
  • u_{1,1}: =P(X₁=1|Y=1):=P(first name partial match|Y=0)
  • m_{1,2}:=P(X₁=2|Y=1):=P(first name nickname match|Y=1)
  • u_{1,2}:=P(X₁=2|Y=1):=P (first name nickname match|Y=0)
  • m_{1,3}:=P(X₁=3|Y=1): =P(first name not match|Y=1)
  • u_{1,3}:=P(X₁=3|Y=1)=P(first name not match|Y=0)

In some embodiments, the monotonicity constraint may be imposed between different identity elements, for example, first name and last name such that total weights of the first name are no lower than those for the last name. This constraint can be mathematically represented

${\prod }_k\frac{m_{\left\{1,k\right\}}}{u_{\left\{1,k\right\}}}\ge {\prod }_k\frac{m_{\left\{2,k\right\}}}{u_{\left\{2,k\right\}}}$

By maximizing Q function in each step with the above constraints, m and u may satisfy explainability and compliance requirements.

Applying Expectation Maximization (EM) to Entity Resolution

As described above, each record may include different record parameters (for example, name, address, phone, and so forth) or have missing fields as compared to another record. Such differences in the records can introduce difficulties in performing direct comparisons between records to determine whether or not the compared records relate to the same person or entity. Each of the processes, methods, and algorithms described in the preceding sections may be embodied in, and fully or partially automated by, code modules executed by one or more computer systems or computer processors comprising computer hardware. The code modules may be stored on any type of non-transitory computer-readable medium or computer storage device, such as hard drives, solid state memory, optical disc, and/or the like. The systems and modules may also be transmitted as generated data signals (for example, as part of a carrier wave or other analog or digital propagated signal) on a variety of computer-readable transmission mediums, including wireless-based and wired/cable-based mediums, and may take a variety of forms (for example, as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The processes and algorithms may be implemented partially or wholly in application-specific circuitry. The results of the disclosed processes and process steps may be stored, persistently or otherwise, in any type of non-transitory computer storage such as, for example, volatile or non-volatile storage.

Additional Embodiments

Each of the processes, methods, and algorithms described in the preceding sections may be embodied in, and fully or partially automated by, code modules executed by one or more computer systems or computer processors comprising computer hardware. The code modules may be stored on any type of non-transitory computer-readable medium or computer storage device, such as hard drives, solid state memory, optical disc, and/or the like. The systems and modules may also be transmitted as generated data signals (for example, as part of a carrier wave or other analog or digital propagated signal) on a variety of computer-readable transmission mediums, including wireless-based and wired/cable-based mediums, and may take a variety of forms (for example, as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The processes and algorithms may be implemented partially or wholly in application-specific circuitry. The results of the disclosed processes and process steps may be stored, persistently or otherwise, in any type of non-transitory computer storage such as, for example, volatile or non-volatile storage.

The various features and processes described above may be used independently of one another, or may be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. In addition, certain method or process blocks may be omitted in some implementations. The methods and processes described herein are also not limited to any particular sequence, and the blocks or states relating thereto can be performed in other sequences that are appropriate. For example, described blocks or states may be performed in an order other than that specifically disclosed, or multiple blocks or states may be combined in a single block or state. The example blocks or states may be performed in serial, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The example systems and components described herein may be configured differently than described. For example, elements may be added to, removed from, or rearranged compared to the disclosed example embodiments.

Conditional language, such as, among others, “can,” “could,” “might,” or “may,” unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements and/or steps. Thus, such conditional language is not generally intended to imply that features, elements and/or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without user input or prompting, whether these features, elements and/or steps are included or are to be performed in any particular embodiment.

Any process descriptions, elements, or blocks in the flow diagrams described herein and/or depicted in the attached figures should be understood as potentially representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps in the process. Alternate implementations are included within the scope of the embodiments described herein in which elements or functions may be deleted, executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those skilled in the art.

All of the methods and processes described above may be embodied in, and partially or fully automated via, software code modules executed by one or more specially configured general purpose computers. For example, the methods described herein may be performed by a processing system, card reader, point of sale device, acquisition server, card issuer server, and/or any other suitable computing device. The methods may be executed on the computing devices in response to execution of software instructions or other executable code read from a tangible computer readable medium. A tangible computer readable medium is a data storage device that can store data that is readable by a computer system. Examples of computer readable mediums include read-only memory, random-access memory, other volatile or non-volatile memory devices, compact disk read-only memories (CD-ROMs), magnetic tape, flash drives, and optical data storage devices.

It should be emphasized that many variations and modifications may be made to the above-described embodiments, the elements of which are to be understood as being among other acceptable examples. All such modifications and variations are intended to be included herein within the scope of this disclosure. The foregoing description details certain embodiments. It will be appreciated that no matter how detailed the foregoing appears in text, the systems and methods can be practiced in many ways. As is also stated above, it should be noted that the use of particular terminology when describing certain features or aspects of the systems and methods should not be taken to imply that the terminology is being re-defined herein to be restricted to including any specific characteristics of the features or aspects of the systems and methods with which that terminology is associated.

Further detail regarding embodiments relating to the systems and methods disclosed herein, as well as other embodiments, is provided in the Appendix of the present application, the entirety of which is bodily incorporated herein and the entirety of which is also incorporated by reference herein and made a part of this specification. The Appendix provides examples of features that may be provided by a system that implements at least some of the functionality described herein, according to some embodiments, as well as specific system configuration and implementation details according to certain embodiments of the present disclosure.

Claims

1. A computer-implemented method, comprising: receiving a plurality of data records from one or more data sources;providing at least a subset of the data records to a scoring model that determines scores for various pairings of the data records, a score for a given pair of the data records representing a probability that the given pair of data records contains data elements about the same entity;generating a graph data structure that includes a plurality of nodes, each individual node of the plurality of nodes representing a different record from the plurality of data records, where edges between given node pairs are associated with corresponding determined scores for respective pairs of data records;performing a connected component analysis of the graph data structure, including pruning one or more edges that fall below a threshold score;performing optimal weighted clustering of the graph data structure to determine final clusters of the plurality of nodes, wherein computer processing time is reduced in performing the optimal weighted clustering at least in part by reduction to a linear programming problem such that only a subset of millions of potential clusters possible from the plurality of data records are analyzed in determining the final clusters;assigning a different unique identifier to each individual cluster of the final clusters, where different identifiers represent different entities; andresponding to a request for data regarding a given entity by providing aggregated data elements from those data records of the plurality of data records associated with a cluster of the final clusters having an identifier that represents the given entity.

2. The method of claim 1, wherein the threshold score indicates whether data records represented by a node pair of the given node pairs connected by an edge of the edges belong to the same entity as each other, wherein a first given node pair having a first edge of the edges associated with a first score that exceeds the threshold score belong to the same entity as each other, and wherein a second given node pair having a second edge of the edges associated with a second score that is less than the threshold score belong to different entities than each other.

3. The method of claim 1, wherein the threshold score varies dynamically according to one or more parameters.

4. The method of claim 1, wherein pruning one or more edges that fall below the threshold score comprises removing the one or more edges associated with the score that falls below the threshold score such that one or more of the given node pairs connected by the removed one or more edges are no longer connected via the removed one or more edges.

5. The method of claim 1, wherein the scoring model comprises a machine learning algorithm that learns weights associated with different attributes of a portion of the data records to generate the scores for individual pairs of data records.

6. The method of claim 1, further comprising excluding at least one node of the plurality of nodes from at least the optimal weighted clustering based on a source of the at least one node according to a rule or restriction that limits combinations of sources of the data records.

7. The method of claim 1, further comprising merging at least two of the final clusters into a larger cluster based on the at least two of the final clusters being associated with the same entity.

8. A data aggregation and computation system, comprising: a data store configured to store a plurality of data records related to a plurality of individual entities; anda hardware processor configured to: receive a request for clustering the plurality of data records according to one or more entity identifiers based on information contained within each of the plurality of data records, the request including identifiers for one or more data sources from which data records of the plurality of data records are sourced;identify one or more entity resolution rules that limit clustering of a given pair of data records of the plurality of data records based on the respective data sources of the one or more data sources for the given pair;perform optimal weighted clustering of the plurality of data records in consideration of the one or more entity resolution rules, wherein processor time used by the hardware processor in performing the optimal weighted clustering is reduced at least in part by reduction to a linear programming problem such that only a subset of millions of potential clusters possible from the plurality of data records are analyzed in determining final clusters;identify a plurality of clusters of data records based on the optimal weighted clustering;assign a different unique identifier to each individual cluster of the plurality of clusters, where different identifiers represent different entities of the individual entities; andrespond to a request for data regarding a given entity of the individual entities by providing aggregated data elements from those data records of the plurality of data records associated with a cluster of the plurality of clusters having an identifier that represents the given entity.

9. The system of claim 8, wherein the hardware processor is further configured to, for at least a first cluster of the plurality of clusters, perform a connected component analysis on a set of data records of the plurality of data records that form the first cluster and prune data records from the set of data records that fall below a threshold score, wherein the threshold score indicates whether a pair of data records belongs to the same entity as each other.

10. The system of claim 9, wherein the threshold score varies dynamically according to one or more parameters.

11. The system of claim 9, wherein the hardware processor configured to prune comprises the hardware processor configured to remove the data records associated with a score that falls below the threshold score from the set of data records.

12. The system of claim 8, wherein the hardware processor is further configured to apply a scoring model to at least a subset of the plurality of data records, wherein the scoring model determines scores for various pairings of the data records, a score for a given pair of the data records representing a probability that the given pair of data records contain data elements about the same entity.

13. The system of claim 12, wherein the scoring model comprises a machine learning algorithm that learns weights associated with different attributes of a portion of the data records to generate the scores for individual pairs of data records.

14. The system of claim 8, wherein the hardware processor configured to perform optimal weighted clustering of the plurality of data records in consideration of the one or more entity resolution rules comprises the hardware processor configured to exclude at least one data record from the optimal weighted clustering based on a source of the at least one data record when the one or more entity resolution rules limit combinations of sources of the data records in the plurality of clusters.

15. The system of claim 8, wherein the hardware processor is further configured to merge at least two of the plurality of clusters into a larger cluster based on the at least two of the plurality of clusters being associated with the same individual entity.

16. A non-transitory computer readable medium comprising instructions that, when executed by one or more hardware processors, cause the one or more hardware processors to perform the steps of: receiving a plurality of data records from one or more data sources;providing at least a subset of the data records to a scoring model that determines scores for various pairings of the data records, a score for a given pair of the data records representing a probability that the given pair of data records contains data elements about the same entity;generating a graph data structure that includes a plurality of nodes, each individual node of the plurality of nodes representing a different data record from the plurality of data records, where edges between given node pairs are associated with corresponding determined scores for respective pairs of data records;performing a connected component analysis of the graph data structure, including pruning one or more edges that fall below a threshold score;performing optimal weighted clustering of the graph data structure to determine final clusters of the plurality of nodes, wherein processor time used by the one or more hardware processors in performing the optimal weighted clustering is reduced at least in part by reduction to a linear programming problem such that only a subset of millions of potential clusters possible from the plurality of data records are analyzed in determining the final clusters;assigning a different unique identifier to each individual cluster of the final clusters, where different identifiers represent different entities; andresponding to a request for data regarding a given entity by providing aggregated data elements from those data records of the plurality of data records associated with a cluster of the final clusters having an identifier that represents the given entity.

17. The non-transitory computer readable medium of claim 16, wherein the threshold score indicates whether data records represented by a node pair of the given node pairs connected by an edge of the edges belong to the same entity as each other, wherein a first given node pair having a first edge of the edges associated with a first score that exceeds the threshold score belong to the same entity as each other, and wherein a second given node pair having a second edge of the edges associated with a second score that is less than the threshold score belong to different entities than each other.

18. The non-transitory computer readable medium of claim 16, wherein the threshold score varies dynamically according to one or more parameters.

19. The non-transitory computer readable medium of claim 16, wherein pruning one or more edges that fall below the threshold score comprises removing the one or more edges associated with the score that falls below the threshold score such that one or more of the given node pairs connected by the removed one or more edges are no longer connected via the removed one or more edges.

20. The non-transitory computer readable medium of claim 16, wherein the scoring model comprises a machine learning algorithm that learns weights associated with different attributes of a portion of the data records to generate the scores for individual pairs of data records.