The present invention relates generally to data synchronization, and more particularly to systems and methods for determining discrepancies between data sets.
The proliferation of digital information has created vast amounts of digital data. Digitized information such as, for example, sales records and customer databases, allow businesses to quickly access their information to increase their profitability and customer satisfaction. However, storing all of this information digitally frequently causes databases to reach terabyte levels in size. Large databases are beneficial when storing data but often become extremely problematic when attempting to manipulate the database, due to its sheer size. This becomes apparent when businesses who share common data attempt to store duplicate information at separate locations or when two different businesses try to work together and correlate their databases. For example, in a merger, two companies will try to correlate records for the same consumer in both company's databases. However, they may not be able to merge the two systems, so they must be kept in synchronization by propagating updates.
Over time, due to added and/or deleted information and other changes, the two different databases will “drift” or grow apart from each other. When this occurs, the databases are no longer identical and must be “synchronized” to ensure that the two databases remain the same.
One method of synchronizing the information is for a business to compare the information bit-by-bit. Obviously, this method is very time consuming and would not be able to keep up with the drift rate between the two databases. Thus, in the amount of time it took to review the databases, additional changes would have occurred and the review would have to restart before it was finished. Another possible method of synchronizing is for one business to send all of their information to the other business to ensure that the information is identical. The problem with this approach is that, due to the massive size of the information, it is extremely costly and time consuming. Additionally, if the companies wish to ensure each day, or multiple times each day, that the data has remained identical, their costs would substantially increase. For example, an international banking institution might have millions, or even possibly billions, of transaction records. Even worse, each transaction record could be composed of thousands of bits, thus dramatically increasing the amount of digital information that must be transferred, far beyond just the number of records. Therefore, this approach proves to be too costly for practical business applications. In fact, even though synchronization protocols might be continuously running to keep databases synchronized, because of system errors, two databases can become out of synchronization. Generally, it is very difficult to detect all of the places where the databases differ.
In more complex business models, each database might be an equivalent database rather than an identical copy of another database. This increases the complexity of determining which database has the correct information. Thus, it might require that even more digital information be exchanged or information be transformed into logically equivalent information between entities to ensure that the databases are equivalent in any necessary aspects. Therefore, businesses desire that a synchronization method be flexible enough to handle equivalent and identical databases on disparate platforms while, at the same time, be cost and time efficient such that frequent synchronizations are feasible. Businesses typically already have synchronization methods in place, and, thus, a means to facilitate these existing methods in order to obtain additional flexibility and error detection is highly desirable. This would allow a company to ensure that its information is correct and that their business is operating with the most up-to-date information as possible. The efficiency and cost effectiveness of business data transactions can directly increase both customer satisfaction and profitability.
The following presents a simplified summary of the invention in order to provide a basic understanding of some aspects of the invention. This summary is not an extensive overview of the invention. It is not intended to identify key/critical elements of the invention or to delineate the scope of the invention. Its sole purpose is to present some concepts of the invention in a simplified form as a prelude to the more detailed description that is presented later.
The present invention relates generally to data synchronization, and more particularly to systems and methods for determining discrepancies between data sets. Data hierarchies are leveraged to provide a systematic means to determine data differences between equivalent data. This allows disparate data storage systems to efficiently determine divergent data locations by utilizing, for example, data signatures representative of varying degrees of data granularity. Comparative analysis can then be performed between the databases by employing an iterative approach until the desired level of data granularity is obtained at which point sending details about records suspected to be mismatched becomes manageable. This allows, in one instance of the present invention, discrepant data to be determined without the transfer of large amounts of data and without requiring homogeneous data storage systems. Another instance of the present invention utilizes equivalent logical data views from non-identical data sets to determine data discrepancies. Yet another instance of the present invention determines discrepancies of a federated and/or integrated data system by employing reversible data statistical signatures, providing a simplistic transfer protocol and sheltering each data system from the other's complexities. Thus, the present invention provides a substantial improvement in data discrepancy determination, both in speed and cost.
To the accomplishment of the foregoing and related ends, certain illustrative aspects of the invention are described herein in connection with the following description and the annexed drawings. These aspects are indicative, however, of but a few of the various ways in which the principles of the invention may be employed and the present invention is intended to include all such aspects and their equivalents. Other advantages and novel features of the invention may become apparent from the following detailed description of the invention when considered in conjunction with the drawings.
The present invention is now described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It may be evident, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing the present invention.
As used in this application, the term “component” is intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a server and the server can be a computer component. One or more components may reside within a process and/or thread of execution and a component may be localized on one computer and/or distributed between two or more computers. A “thread” is the entity within a process that the operating system kernel schedules for execution. As is well known in the art, each thread has an associated “context” which is the volatile data associated with the execution of the thread. A thread's context includes the contents of system registers and the virtual address belonging to the thread's process. Thus, the actual data comprising a thread's context varies as it executes.
Additionally, a component can also include a human element. For example, a human can take a digest of databases manually to a second organization and compare it manually and/or a human can burn a CD with the data that is sent via courier to second organization. Though in-efficient, a human can also be the one creating the digest.
Enterprise software requires disparate entities to share information and collaboratively update the data. There are a number of available algorithms known to accomplish this, but for a variety of reasons such as strong assumptions required by an algorithm not holding up, errors in the implementation, and/or updates happening outside the implementation of the algorithms, the utilization of these algorithms results in copies of data maintained in two different places becoming or “drifting” out of synchronization. The present invention provides a way to locate these discrepancies as an ongoing process so that requisite cleanups can be done. In general, the systems and methods of the present invention are utilized to facilitate existing protocols that propagate and apply changes. However, instances of the present invention can also be utilized for detecting and fixing changes, though it is typically not as efficient as pro-actively propagating changes. One instance of the present invention employs two components, namely a partitioning component that partitions data into smaller chunks and a signature component that computes signatures for the smaller chunks. Another party then compares the signature of each chunk with signatures from their own chunks of data and identifies chunks whose signatures do not match with their own. For non-matching chunks, the chunks are then broken down into a lower level of granularity and re-signed and sent to the other party. The other party re-computes its corresponding chunk to determine which of the larger non-matching chunks do not match. The process is then repeated for smaller and smaller non-matching chunks until the specific non-matching records and/or data are found. Thus, the present invention can be employed to facilitate in locating discrepant data to allow requisite synchronization of the data by various data management entities. The present invention also facilitates to reduce the data set associated with a data mismatch between entities. The selection process is logarithmic, producing ‘n’ messages for ‘n’ elements in a data set. For example, if there are ‘d’ discrepancies, in the worst case, all of them can have an independent path from a master digest. So, it produces d*log(n) messages. In this case, all the data is erroneous, so, this produces n*log n messages. Thus, this protocol is useful when d is so small that d*log n is substantially smaller than n. This is superior to a linear process that requires that a complete data set be transmitted between entities, increasing transaction costs substantially.
In
The hierarchical drift detection component 102 of entity “1” 112 employs a digital signature technique to partitions associated with the structure of the associated data set 122. Partitioning is accomplished by domain specific algorithms. A ‘signature’ or digest is created for each individual partition. So, signatures are created post partitioning. However, the entire set <partition1-signature>, <partition2-signature> can be thought of as the signature of the whole data set and the partitioning algorithm to be just a part of the signature algorithm. This allows a condensed version of the data to be transmitted to the other data management entities. Likewise, the other hierarchical drift detection components 104-110 also employ digital signature techniques to their associated data sets 124-130 on equivalent data. If data management entity “1” 112 is considered the master, for example, it 112 can initiate a partitioning of its data set 122 based on a highest level of the data structure. This yields data partitions with the coarsest resolution of the data structure. A signature is then calculated for each coarse data partition by the hierarchical drift detection component 102, and a statistical signature is then utilized based on these individual data signatures to create a single signature representative of the coarse data partitions. The data management entity “1” 112 then transmits the statistical data signature to the other data management entities “2-P” 114-120. Each entity “2-P” 114-120 compares the statistical signature from data management entity “1” 112 to their own computed statistical signature of the equivalent level of coarse data partitions. If one of the entities “2-P” 114-120 finds a mismatch, it compares the signatures of the partitions to identify mismatched partitions. For each mismatched partition, it partitions at one level deeper and calculates signatures for this level of the data. The new signatures are then transmitted back to data management entity “1” 112. Data management entity “1” 112 then compares this new level of data signatures to its own signatures at that level. This iterative process continues until a criterion is reached such as, for example, a data subset is obtained that is small enough to be transmitted without substantial cost, an atomic data granularity level has been reached, a predetermined time limit has been reached, a predetermined granularity level has been reached, and/or a predetermined number of transmissions has occurred and the like.
The present invention can also utilize combined signatures such as utilizing a lower level signature and a higher level signature to form the signature that is transmitted between two entities. It can also incorporate techniques such that disparate data structures can be shielded (i.e., isolated) from another entity and non-identical data sets can also be synchronized through equivalent data sets formed by logical views. If two datasets are being dynamically updated while still detecting errors in a running system, a logical view can capture data as of event X. One skilled in the art will appreciate that there are multiple ways of marking event X, including synchronized time, Lamport's vector clock, etc. These aspects of the present invention are detailed infra.
Referring to
In this instance of the present invention, the single hierarchical drift detection component 202 communicates with the data management entities “1-Q” 204-210 to determine if any data mismatches have occurred. It 202 asks each of the entities 204-210 for the signatures and combines their signatures into one master signature. It 202 then receives a master signature from another entity and identifies the sub-partitions where there are mismatches. At this point, it 202 has at least two options (1) still stay in loop, ask the sub-partition to provide a more detailed signature, and merge them together in a detailed signature or (2) ask the sub-partitions to talk directly to the corresponding sub-partition on the other side in order to detect errors at a finer level of granularity. Generally speaking, it 202 does not start by asking sub-components for mismatches, since sub-components typically only know their data and have not received information about the other side.
This is accomplished, in one example of the present invention, via iterative processing of signatures generated on data provided by the individual data management entities “1-Q” 204-210. The signatures are received by the hierarchical drift detection component 202 and analyzed against signatures received from other data management entities. In this manner, the hierarchical drift detection component 202 can direct a data synchronization evaluation by requesting data signatures at appropriate data structure levels. The data structure levels themselves can also be dictated via the hierarchical drift detection component 202.
Turning to
The optional logical view component 312 is utilized when disparate data structures are associated with the data management entities “1-R” 304-310. The logical view component 312 interfaces with the data management entities “1-R” 304-310 and the iterative process control component 314 to determine an appropriate logical view that can be employed by the hierarchical drift detection system 300. In this manner, the detection of data discrepancies is independent of the structure of the data sets. This affords the present invention great flexibility in its deployment, substantially surpassing traditional data synchronization systems. Once a logical data view has been selected, if necessary, the iterative process control component 314 initiates the data signature component 316 to determine data signatures for a data set. The data signature is then passed to the iterative process control component which then transmits the data signature to an appropriate data management entity. A response from the data management entity is evaluated by the iterative process control component 314 to determine if any mismatched data has been detected. If mismatches have occurred, it 314 initiates the data signature component 316 to determine data signatures for one lower level of the data that has been partitioned according to its structure. This process continues until the iterative process control component 314 has determined that a stop criterion has been met as elaborated supra.
Moving on to
The supra systems of the present invention facilitate in eliminating the widespread problems surrounding data drifting. The present invention accomplishes this in a generic and expedited manner. The algorithm employed by instances of the present invention generally utilizes two components. The first component provides a way to partition data into smaller chunks. This partitioning scheme allows multiple levels of partitioning. For example, suppose the data being maintained is about customers as shown in the illustration 500 in
The two components are then utilized with the algorithm as follows. First, the data is broken up into chunks at a highest level (i.e., level 1), producing the coarsest chunks. Then the digest is computed for each chunk. Typically, the signature of a chunk is a tuple where the first element has the information required to identify the chunk and the second element is the ‘digest’ of the chunk. In the example supra, the ‘prefix’ in the name string utilized for grouping is sufficient to identify the chunk and number of customers in that chunk is the digest. The Statistical Signature of the data set is computed by the set of signatures of the chunks of data. The complete statistical signature of data is sent to another entity. The other entity then computes the Statistical Signature in an equivalent fashion. It compares the signature of each chunk and identifies the chunks whose signatures do not match. For each of these mismatched chunks, it partitions data one level deeper (e.g. utilizing two characters for a customer name), computes the signatures for the partitions, and sends the signatures back to the original entity. The signature of a data set is now more detailed for the mismatched chunks. Depending on the instance of the present invention, the present invention can mix these details with other high level signatures and/or send a special message with ‘mismatched’ chunks only. Entities continue sending data back and forth, successively refining it until the granularity comes to the level of a single row and/or the chunk becomes so small that the complete chunk can be sent. A comparison at this point identifies the rows that are missing on either side and/or have conflicting data. Conflict resolution can be done with standard resolution methodologies, for example, such as defining one of the sources as the master and winning the conflict every time, random decision making, and/or manual intervention and the like.
Additionally, other instances of the present invention utilize a structure with the signatures to further facilitate locating data discrepancies. For example, groups can be employed that represent a top half and/or a bottom half and the like. This allows a comparing entity to utilize prior knowledge to more quickly discern where the mismatched data is located. One skilled in the art can appreciate that prior knowledge and/or probabilistic data error likelihood information can be employed to converge the iterative process more quickly. Multiple replies can also be given by an entity to facilitate the iterative process. Instances of the present invention also allow the comparing entity to ascertain which data segments and what levels are necessary to retransmit back to the originating entity. It is also not necessary to start with the coarsest data. For example, if during a first run it is discovered that frequent mismatches are found in most of the level 1 chunks, the protocol can start directly at level 2. Since signatures are utilized, two different data sets can produce a substantially similar signature, and, all the problems might not be detected. The width of the signature can be controlled, in one instance of the present invention, to control the probability that some conflict might be missed. Furthermore, drift detection can be repeated to enhance detection of errors in the data. Thus, in one instance of the present invention, different signature algorithms can be employed in different ‘runs’ to reduce the probability that a conflict might be missed.
The costs associated with employing the present invention to detect data discrepancies include the cost of computing the signatures by an entity, the cost of exchanging the signature between entities, and the cost of exchanging the data between entities. Cost can also be a function of the error rate. If an error rate is substantially high, it is more cost efficient to send the data. If the error rate is substantially low, it is more efficient to utilize the present invention to determine any data discrepancies. Additionally, instances of the present allow a user to determine at what level of granularity they wish to pursue to find mismatched data. Generally speaking, this also indicates a cost level that the user is willing to accept.
There are many parameters for this algorithm that can be fine tuned based on application and/or user preferences and the like. These include, but are not limited to, at what point is it better to send a complete dump of a ‘set suspected to be out of sync’ rather than keep sending a digest, whether the send/receive of mismatches are separated from the send/receive of ‘signatures,’ how often and with what method to compute the signatures, and how good is the signature in catching the kind of errors expected and the like. Thus, parameters such as these can be utilized to extract maximum efficiency from a data synchronization scheme that employs the present invention.
The present invention also facilitates in synchronizing disparate databases as shown in the illustration 600 in
Turning to
In view of the exemplary systems shown and described above, methodologies that may be implemented in accordance with the present invention will be better appreciated with reference to the flow charts of
The invention may be described in the general context of computer-executable instructions, such as program modules, executed by one or more components. Generally, program modules include routines, programs, objects, data structures, etc., that perform particular tasks or implement particular abstract data types. Typically, the functionality of the program modules may be combined or distributed as desired in various instances of the present invention.
In
Referring to
Turning to
In order to provide additional context for implementing various aspects of the present invention,
As used in this application, the term “component” is intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and a computer. By way of illustration, an application running on a server and/or the server can be a component. In addition, a component may include one or more subcomponents.
With reference to
The system bus 1208 may be any of several types of bus structure including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of conventional bus architectures such as PCI, VESA, Microchannel, ISA, and EISA, to name a few. The system memory 1206 includes read only memory (ROM) 1210 and random access memory (RAM) 1212. A basic input/output system (BIOS) 1214, containing the basic routines that help to transfer information between elements within the computer 1202, such as during start-up, is stored in ROM 1210.
The computer 1202 also may include, for example, a hard disk drive 1216, a magnetic disk drive 1218, e.g., to read from or write to a removable disk 1220, and an optical disk drive 1222, e.g., for reading from or writing to a CD-ROM disk 1224 or other optical media. The hard disk drive 1216, magnetic disk drive 1218, and optical disk drive 1222 are connected to the system bus 1208 by a hard disk drive interface 1226, a magnetic disk drive interface 1228, and an optical drive interface 1230, respectively. The drives 1216-1222 and their associated computer-readable media provide nonvolatile storage of data, data structures, computer-executable instructions, etc. for the computer 1202. Although the description of computer-readable media above refers to a hard disk, a removable magnetic disk and a CD, it should be appreciated by those skilled in the art that other types of media which are readable by a computer, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, and the like, can also be used in the exemplary operating environment 1200, and further that any such media may contain computer-executable instructions for performing the methods of the present invention.
A number of program modules may be stored in the drives 1216-1222 and RAM 1212, including an operating system 1232, one or more application programs 1234, other program modules 1236, and program data 1238. The operating system 1232 may be any suitable operating system or combination of operating systems. By way of example, the application programs 1234 and program modules 1236 can include a data discrepancy detection scheme in accordance with an aspect of the present invention.
A user can enter commands and information into the computer 1202 through one or more user input devices, such as a keyboard 1240 and a pointing device (e.g., a mouse 1242). Other input devices (not shown) may include a microphone, a joystick, a game pad, a satellite dish, wireless remote, a scanner, or the like. These and other input devices are often connected to the processing unit 1204 through a serial port interface 1244 that is coupled to the system bus 1208, but may be connected by other interfaces, such as a parallel port, a game port or a universal serial bus (USB). A monitor 1246 or other type of display device is also connected to the system bus 1208 via an interface, such as a video adapter 1248. In addition to the monitor 1246, the computer 1202 may include other peripheral output devices (not shown), such as speakers, printers, etc.
It is to be appreciated that the computer 1202 can operate in a networked environment using logical connections to one or more remote computers 1260. The remote computer 1260 may be a workstation, a server computer, a router, a peer device or other common network node, and typically includes many or all of the elements described relative to the computer 1202, although for purposes of brevity, only a memory storage device 1262 is illustrated in
When used in a LAN networking environment, for example, the computer 1202 is connected to the local network 1264 through a network interface or adapter 1268. When used in a WAN networking environment, the computer 1202 typically includes a modem (e.g., telephone, DSL, cable, etc.) 1270, or is connected to a communications server on the LAN, or has other means for establishing communications over the WAN 1266, such as the Internet. The modem 1270, which can be internal or external relative to the computer 1202, is connected to the system bus 1208 via the serial port interface 1244. In a networked environment, program modules (including application programs 1234) and/or program data 1238 can be stored in the remote memory storage device 1262. It will be appreciated that the network connections shown are exemplary and other means (e.g., wired or wireless) of establishing a communications link between the computers 1202 and 1260 can be used when carrying out an aspect of the present invention.
In accordance with the practices of persons skilled in the art of computer programming, the present invention has been described with reference to acts and symbolic representations of operations that are performed by a computer, such as the computer 1202 or remote computer 1260, unless otherwise indicated. Such acts and operations are sometimes referred to as being computer-executed. It will be appreciated that the acts and symbolically represented operations include the manipulation by the processing unit 1204 of electrical signals representing data bits which causes a resulting transformation or reduction of the electrical signal representation, and the maintenance of data bits at memory locations in the memory system (including the system memory 1206, hard drive 1216, floppy disks 1220, CD-ROM 1224, and remote memory 1262) to thereby reconfigure or otherwise alter the computer system's operation, as well as other processing of signals. The memory locations where such data bits are maintained are physical locations that have particular electrical, magnetic, or optical properties corresponding to the data bits.
In one instance of the present invention, a data packet transmitted between two or more computer components that facilitates data discrepancy determination is comprised of, at least in part, information relating to a data discrepancy determination system that utilizes, at least in part, at least one data signature representative of at least one data partition based, at least in part, on a hierarchical structure of a data set and utilized in an iterative process to isolate mismatched data.
It is to be appreciated that the systems and/or methods of the present invention can be utilized in data discrepancy detection facilitating computer components and non-computer related components alike. Further, those skilled in the art will recognize that the systems and/or methods of the present invention are employable in a vast array of electronic related technologies, including, but not limited to, computers, servers and/or handheld electronic devices, and the like.
What has been described above includes examples of the present invention. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the present invention, but one of ordinary skill in the art may recognize that many further combinations and permutations of the present invention are possible. Accordingly, the present invention is intended to embrace all such alterations, modifications and variations that fall within the spirit and scope of the appended claims. Furthermore, to the extent that the term “includes” is used in either the detailed description or the claims, such term is intended to be inclusive in a manner similar to the term “comprising” as “comprising” is interpreted when employed as a transitional word in a claim.