Patent Application
20080027899
A further understanding of the various embodiments of the present invention may be realized by reference to the figures which are described in remaining portions of the specification. In the figures, like reference numerals are used throughout several to refer to similar components. In some instances, a sub-label consisting of a lower case letter is associated with a reference numeral to denote one of multiple similar components. When reference is made to a reference numeral without specification to an existing sub-label, it is intended to refer to all such multiple similar components.
The present invention relates to systems and methods for data production and distribution, and in particular to systems and methods for gathering and maintaining data.
Various embodiments of the present invention provide systems and methods for data exchange. One exemplary method in accordance with embodiments of the present invention includes identifying at least one target data receptacle and at least two source data receptacles. As used herein, the term “data receptacle” is used in its broadest sense to mean any repository of data. Thus, for example, a data receptacle may be, but is not limited to, a database server or a hard disk drive that is formatted to accept information. In addition, the method involves providing a map that includes a relationship between a source element of one of the source data receptacles and a target element of the target data receptacle, and between a source element of another source data receptacle and another target element of the target data receptacle. In some cases, the methods further include designing the target data receptacle. Designing the target data receptacle includes providing a name for the target data elements. Designing the target data receptacle may further include providing a relationship between the target elements. In one particular instance of the embodiments, the target data receptacle is an XML file defined by an XML schema. Based on the disclosure provided herein, however, one of ordinary skill in the art will recognize other files and/or schema types to which embodiments of the present invention may be applied.
Some embodiments of the present invention provide capability to view metadata of more than one data source, as well as a capability to map one or more data sources to one or more target stores. Thus, embodiments of the present invention may provide for mapping and/or merging multiple sources to multiple targets, multiple sources to one target, and/or one source to multiple targets. In these embodiments, a user can define relationships between fields of different data sources (e.g., a database column with a field in a delimited text file). After the relationships have been prepared, a user may be presented with a merged (hierarchical) view of source data, which the user can map to any of various target fields. Then, the merge may be completed in accordance with the aforementioned relationships. The merge may be done by merging information coming from multiple data sources in accordance with an association rule set up by the user and the mapping rules will then be applied to the merged data. Further, in some cases, a user may be able to specify if any field of a source metadata contains some data that shall be parsed using some other program. In such a case, the parsed metadata may be shown as part of original metadata hierarchy allowing a user to map data fields from original metadata as well as the parsed content metadata.
In one particular embodiment of the present invention, mappings defined in a data production engine or data mapper is converted to a standard XSL file where the data is a single XML stream in accordance with the merged structure of the actual data sources. During the data transfer process, the embodiment internally converts data received from various data sources to XML streams and then all these streams are merged to create a single and consolidated intermediate XML stream. The mapper generated XSL is applied over the aforementioned XML stream to generate an intermediate XML stream, which is then converted to target data structures and then imported to target data repositories.
Turning to
In addition, system 100 includes a data production engine 150. As illustrated, data production engine 150 is communicably coupled to the various data receptacles via a communication network 110. As used herein, the term “communicably coupled” is used in its broadest sense to mean any approach or mechanism whereby information may be exchanged between devices. Thus, for example, communication network may be, but is not limited to, the Internet, a virtual private network, an optical network, a cellular telephone network, a public switched telephone network, a wire between devices, combinations of the aforementioned, and/or the like.
Data production engine 150 may be any microprocessor based tool capable of communicating with one or more of the data receptacles via communication network 110. In one particular instance, data production engine 150 is a personal computer executing software instructions. Based on the disclosure provided herein, one of ordinary skill in the art will appreciate a variety of microprocessor based systems capable of providing the functionality associated with data production engine 150.
In operation, data production engine 150 receives a schema for a target data source that may be maintained on target data store 140. The map indicates a relationship between elements in the schema and elements from various of public database 120, public database 130, proprietary database 160, proprietary database 170, and proprietary database 180. As just one example, a field in proprietary database 160 may include an employee's name, a field in proprietary database 170 may include an employee's current compensation, a field in proprietary database 180 may include company products represented by a particular employee, and public databases 120, 130 may include a field that provides public marketing information about company products. The map associates these respective fields from databases 120, 130, 160, 170, 180 with corresponding fields in the target schema that may be maintained on target data store 140.
Turning to
Data production element 150 implements a map graphically portrayed as lines between respective data elements on graphical representation 200. In the depicted situation, the map causes information associated with data element G 231 to be merged in target data structure 240 as data element X 242. Similarly, data elements D.1221, D.2222, E.3223, F 224, F.1225, and F.2226 of source data structure 220 are respectively merged in target data structure 240 as data elements X.1244, X.2246, X.3248, Y 250, Y.1252, and Y.2254. Data elements A 211, A.1212, A.2213, and A.3214 of source data structure 210 are respectively merged in target data structure 240 as data elements Z 256, Z.1258, Z.2260, and Z.3262. In a typical scenario, source data structure 210, source data structure 220, and source data structure 230 may be associated with a public database or a proprietary database. Target data structure 240 may be associated with a target data store. Based on the disclosure provided herein, one of ordinary skill in the art will recognize that the source data structures may be associated with various different data sources. Indeed, the source data structures and target data structure may all exist on the same physical medium, each on distinct physical media, or some on the same physical media and others on distinct physical media.
Turning now to
Various data sources that include information that can be used to populate the target data store are also identified (block 310). Selection of the data sources may include selection of two or more data sources depending upon the level of distribution exibited by the information that is to be included in the target data structure. Selection may be done by putting in location information about the data source. For example, where one of the data sources is a public database accessible via the Internet, identifying a data source may include providing a URL address of the data source. Alternatively, where the data source is a proprietary data source available on a hard disk drive associated with a computer implementing the method, identifying the data source may include identifying a file holding the data source. Based on the disclosure provided herein, one of ordinary skill in the art will recognize a variety of methods by which data sources may be identified in accordance with one or more embodiments of the present invention.
A graphic is formatted that includes the designed target data structure displayed in relation to source data structures identified as the data sources (block 315). Thus, for example, two or more source data structures may be displayed to the left of the target data source. The individual elements of the source data structures and target data structure are displayed in such a way that the individual elements may be graphically connected one to another. Via the aforementioned graphic (block 315), graphical instructions may be received that connect particular elements of the various data structures (block 325). It is also determined whether all instructions have been received (block 330). Where reception of the instructions is not yet complete (block 330), additional instructions are received (block 325).
Alternatively, where reception of the instructions is complete (block 330), a map is formatted based on the previously received graphical instructions (block 335). The map includes a list of corresponding data elements including a data element from a source data structure that corresponds to (i.e., is mapped) an element of the target data structure. Where, for example, the target data structure includes ten elements, the map will include ten entries, with each of the ten entries identifying a source data element corresponding to a particular data element of the target data structure. Data is then merged from the data source to the target in accordance with the map (block 340). Thus, for example, where the map indicates that an element of one source data structure corresponds to a first element of the target data structure, then the first instance of the particular data element of the source data structure is accessed. This first instance is transferred and stored as the first instance of the corresponding data element of the target data store. This process is repeated for the second instance, with the second instance from the source data structure being stored as a second instance of the corresponding data element of the target data store. This process is continued until all instances of the particular data element have been transferred from the data source to the target data store. The process is then repeated for the next element and/or data sources, and for all instances thereof (block 345, 350). Once all of the elements and instances thereof have been merged, the process ends.
Where the number of instances is not the same for each of the data elements, or where the instances are not properly aligned, some alignment may be done. For example, where the first instance of a data element is the name FRED, and the first instance of the second data element is an employee number of JACK and the second instance of the second data element is the employee number for FRED some algorithm capable of assuring that the target data store is loaded with all of the information associated with FRED in one particular instance of the target data structure. Based on the disclosure provided herein, one of ordinary skill in the art will appreciate a variety of algorithms and/or methods that may be employed to assure that merges are aligned such that information in any particular instance of the target data structure are related.
Further, it should be noted at this juncture that while flow diagram 300 is particularly suited to transferring information from two or more data sources to a single data target, various embodiments of the present invention also provide for transferring information from one data source to multiple data targets, or from two or more data sources to two or more data targets. In some cases where data is being transferred to more than one target, data integrity is maintained by assuring that data is either supplied to all data targets or it is not supplied to any of the data targets. This may be achieved by extensive transaction support, which can rollback all data imported to any previous data target in the event that a tat transfer to a later data target identifies an error or inability to transfer particular data to another data target.
Turning to
In operation, a map graphic 410 is formed by selecting one of the source data elements and a corresponding target data element. Thus, as an example shown by
Turning to
As will be appreciated from the preceding discussion, systems and methods in accordance with the present invention may be used to address various situations. For example, where a data set is stored across multiple data sources, it may be merged into a new single or multiple data target. A particular implementation of the aforementioned example may include merging information related to employees in an Organization A that is spread across different databases and an XML file. Where, for example, Organization A is taken over by an Organization B which stores information about its employees is some different format, one or more embodiments of the present invention is able to extract information about the employees of Organization A and populate the extracted information in the appropriate fields of a database previously limited to employee information of Organization B. From this, a comprehensive employee report may be generated as, for example, an XML report. Alternatively, embodiments of the present invention may be used to provide a comprehensive employee report spanning Organization A and Organization B without formally merging the databases.
As another example, where a data set is stored in a data source and one or more fields of the data source contain information that is in a different format and can be parsed by some software program, one or more embodiments of the present invention may be tailored to extract information from the data source as well as identify the format of the information. A particular implementation of the aforementioned example may include taking a database with a table called “EMPLOYEES”. The table includes details (e.g., performance related data) in an XML file about each employee that is stored in a column of the table entitled “DETAILS”. One or more embodiments of the present invention may be used to generate an XML file containing all the information about all employees.
Turning to
Turning to
In
Turning to
Based on the created map graphic 510, information from source A graphic 540 and source B graphic 550 may be gathered and reassembled as data X record 572 of target data structure 570. This process can include accessing the first instance of data B record 545 including the information maintained as data B.1 element 546 of the first record, and sorting through the instances of data E.1 elements 555 from data E record 554 to find a match to the first instance of data B.1 element 546. Once a match is found between data B.1546 and data E.1555, the instance of data B.1 element 546 and data B.2 element 547 corresponding to the first instance of data B record 545; and the instance of data E.2 element 556, and data E.3 element 557 corresponding to the found instance of data E.1 element 555 are transferred to a common instance of respective sub-elements of data X record 572 in accordance with map graphic 510. This process continues with accessing the second instance of data B record and finding the next match between data B.1 element 546 and data E.1 element 555, and continues until all instances of data B.1 element 546 of data B record 545 have been considered. This process yields a unified target database.
As one of many examples, data B.1 element 546 may be EMPLOYEE_ID as used above, and data B.2 element 547 may be compensation information about the employees identified in the field EMPLOYEE_ID. Similarly, data E.1 element 555 may be EMP as used above, and data E.2 element 556 and data E.3 element 557 may be contact information, names and other identification information about the employees identified in the field EMP. Once the transfer is complete, data X record 572 includes the employee ID from data B.1 element 546, and the compensation information and identification information from the respective sub-elements of data B record 545 and data E record 554.
In conclusion, the present invention provides novel systems, methods and arrangements for exchanging data. While detailed descriptions of one or more embodiments of the invention have been given above, various alternatives, modifications, and equivalents will be apparent to those skilled in the art without varying from the spirit of the invention. Therefore, the above description should not be taken as limiting the scope of the invention, which is defined by the appended claims.
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/US05/28298 | 8/9/2005 | WO | 00 | 12/12/2005 |
