Method and system for federated querying of data sources

Abstract

A system for managing and querying a plurality of data sources, including an interface to a plurality of inter-related data sources of diverse types, each of the plurality of data sources having a schema that describes its data structure, a schema generator communicating with the interface, for generating a federated schema that describes the structures of the plurality of data sources and their inter-relationships, and for modifying the federated schema over time as the plurality of data sources undergo changes, and a query generator communicating with the schema generator, for generating a query for the federated schema. A method and a computer-readable storage medium are also described.

Description

FIELD OF THE INVENTION

The present invention relates to management and querying of enterprise data.

BACKGROUND OF THE INVENTION

Information integration applications take data that is stored in two or more data sources and build from them one large database, possibly a virtual database, containing information from all of the sources, so that the data can be queried as a unit. Thus, for example, enterprise accounting data may be stored within a relational database, and enterprise inventory may be stored within XML documents. Information integration enables an enterprise to access its various data sources from within a single data store application.

Information integration is discussed in Chap. 20 of Garcia-Molina, H., Ullman, J. D. and Widom, J., “Database Systems: The Complete Book”, Prentice-Hall, New Jersey, 2002. As pointed out in Sec. 20.1 of this reference, there are three basic modes of information integration: (i) federated databases, (ii) data warehousing, and (iii) mediation.

In a federated database architecture, data sources are independent, but one source can call on others to supply information.

In a data warehousing architecture, data from several sources is extracted and combined into a global schema. The data is then stored at the warehouse, which appears to the user like an ordinary database. Once data is in the warehouse, queries are issued by a user exactly as they would be issued to any database. However, user updates to the warehouse are generally forbidden, since they are not reflected in the underlying sources, and thus can make the warehouse inconsistent with the sources.

A data warehouse is updated periodically, by reconstructing it from current data in the data sources. Typically, a data warehouse is updated once a night, when the system can be shut down, so that queries are not issued while the warehouse is being constructed. Alternatively, the data warehouse may be incrementally updated based on changes that have been made to the data sources since the last time the warehouse was modified.

Conventional data warehouses are generally expensive and inflexible. In addition, such data warehouses generally do not provide real-time operation.

In a mediation architecture, a software component, referred to as a mediator, supports a virtual database, which a user may query as if it were physically constructed. The mediator stores no data of its own. Rather, it translates a query into one or more queries to its sources, synthesizes the answer to the query from the responses of the sources, and returns an answer to the user. A mediator supports a virtual view, or collection of views, that integrates several sources.

An example of a mediation system is the Enterprise Information Integrator (EII) of IBM Corporation, which generates a virtual warehouse. EII supports integrated querying across multiple data sources, including IBM DB2 relational databases, Microsoft SQL relational databases, and XML document databases.

All three of the approaches to information integration described above use transformers, referred to as wrappers or extractors, to transform data when it is extracted from a data source. Wrappers are used to pass ad-hoc queries to data sources, receive responses from the source, and pass information to an information integrator.

A drawback with conventional information integration is the lack of uniformity in semantics, and the lack of traceability back to individual data sources. Each database accessed by a warehouse generally has its own semantics, including inter alia names for tables and their fields, names for XML complex types and their elements, and data formats. It may happen that the same name is used in different contexts within different databases, or multiple names are used for the same construct, perhaps formatted differently for different names. Further complications that can arise with non-uniform semantics include inter alia, different inter-relationships between data constructs, different business rules relating the same data constructs, redundancies and inconsistencies.

It is thus desirable to be able to introduce a common semantic foundation for all of the data sources accessed within a data warehouse, and to provide a translation layer which enables a user to access data using queries expressed in common and meaningful semantics, and buffers the user from the individual semantics for the individual data sources.

SUMMARY OF THE DESCRIPTION

The present invention concerns use of common semantics for querying data within diverse types of data sources. The present invention provides a generic translation layer that buffers a user from the particulars of individual semantics for each data source, and enables the user to express queries in terms of common and meaningful semantics.

There is thus provided in accordance with a preferred embodiment of the present invention a system for managing and querying a plurality of data sources, including an interface to a plurality of inter-related data sources of diverse types, each of the plurality of data sources having a schema that describes its data structure, a schema generator communicating with the interface, for generating a federated schema that describes the structures of the plurality of data sources and their inter-relationships, and for modifying the federated schema over time as the plurality of data sources undergo changes, and a query generator communicating with the schema generator, for generating a query for the federated schema.

There is further provided in accordance with a preferred embodiment of the present invention a method for managing and querying a plurality of data sources, including generating a federated schema that describes the structures of a plurality of inter-related data sources and their inter-relationships, and for modifying the federated schema over time as the plurality of data sources undergo changes, wherein each of the plurality of data sources has a schema that describes its data structure, and generating a query for the federated schema.

There is yet further provided in accordance with a preferred embodiment of the present invention a computer-readable storage medium storing program code for causing a computer to perform the steps of generating a federated schema that describes the structures of a plurality of inter-related data sources and their inter-relationships, and for modifying the federated schema over time as the plurality of data sources undergo changes, wherein each of the plurality of data sources has a schema that describes its data structure, and generating a query for the federated schema.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be more fully understood and appreciated from the following detailed description, taken in conjunction with the drawings in which:

FIG. 1 is a simplified diagram of three schemas mapped into a central ontology, and a federated schema that combines the three schemas, in accordance with a preferred embodiment of the present invention;

FIG. 2 is a simplified block diagram of a system that generates the federated schema of FIG. 1, in accordance with a preferred embodiment of the present invention;

FIG. 3 is a simplified block diagram of an enterprise information system that uses the federated schema of FIG. 1 to query across multiple databases using common semantics, in accordance with a preferred embodiment of the present invention;

FIG. 4 is a detailed block diagram of the enterprise information system of FIG. 3, in accordance with a preferred embodiment of the present invention; and

FIGS. 5A, 5B and 5C are illustrations of a user interface for semantic discovery and query, in accordance with a preferred embodiment of the present invention.

DETAILED DESCRIPTION

Enterprise data is typically distributed over multiple databases, referred to as data assets. Data assets can be of various types, including inter alia relational database tables, XML databases, entity-relationship (ER) databases and Cobol databases. Each data asset stores data according to a specific data structure format, referred to as a schema or as asset metadata. Asset metadata serves as descriptors, explaining how to interpret data stored within the asset. Without knowledge of asset metadata, data stored within an asset is generally unintelligible. Thus, for example, relational database tables store data according to a relational database schema, XML documents store data according to an XML schema, ER databases store data according to an ER logical model, and Cobol databases store data according to a Cobol Copybook. Each such schema is asset metadata, which explains how to interpret data stored within the asset.

Asset metadata is typically comprised of basic data structures, or constructs. For example, relational database schema are comprised of fields and tables, XML schema are comprised of simple types and complex types, ER logical models are comprised of entities and relationships, and Cobol Copybooks are comprised of elementary items and group items.

In a preferred embodiment of the present invention, semantics are provided to enterprise data through (i) a common ontology model, referred to also as an information model; and (ii) mappings of enterprise asset metadata into the ontology model. An ontology model is comprised of classes and properties, and is particularly useful for representing data in a semantically meaningful way. Mappings are associations of constructs of a first asset metadata with constructs of a second asset metadata in a consistent way. Mappings of asset metadata into the ontology model serve as dictionaries through which constructs of the asset metadata can be semantically understood.

Reference is now made to FIG. 1, which is a simplified diagram of three schemas, for three enterprise data assets, which are mapped into a central ontology, and a federated schema that combines the three schemas, in accordance with a preferred embodiment of the present invention. Shown in FIG. 1 are three conventional data schemas, an IBM DB2 schema 110 for a relational database, a Microsoft SQL schema 120 for a relational database, and an XML schema 130 for one or more XML documents. Each of schemas 110, 120 and 130 is individually mapped into a common ontology model 140.

Ontology model 140 preferably encapsulates substantially all of the constructs from schemas 110, 120 and 130, including inter alia database tables and their fields and their interrelationships through foreign keys, and XML complex types and their elements and the type inter-relationships, as well as business rules that relate table fields to one another and XML elements to one another. Ontology model 140 serves not only to model enterprise data assets, but also provides common semantics for referencing the data assets.

The individual mappings of schemas 110, 120 and 130 into ontology model 140 can be inverted, and used to map ontology model 140 into a single federated schema 150. Federated schema 150 effectively combines the three individual schemas 110, 120 and 130. Moreover, federated schema inherits the common semantics of ontology model 140.

Reference is now made to FIG. 2, which is a simplified block diagram of a system that generates the federated schema of FIG. 1, in accordance with a preferred embodiment of the present invention. Shown in FIG. 2 are the three schemas 110, 120 and 130 from FIG. 1, the ontology model 140 of FIG. 1 and the federated schema 150 from FIG. 1. Also shown in FIG. 2 is a schema-to-ontology generator 210, which generates ontology model 140 from the three schemas 110, 120 and 130, using mappings of the three schemas into the ontology model. Preferably, schema-to-ontology generator 210 acts as a reverse-engineer, and creates ontology classes and properties that correspond to the constructs of schemas 110, 120 and 130 and their inter-relationships. Preferably, schema-to-ontology generator 210 also generates business rules that inter-relate properties of ontology classes, corresponding to the business rules that apply to schemas 110, 120 and 130.

It may be appreciated by those skilled in the art that ontology model 140 may be alternatively generated without the use of schema-to-ontology generator 210. Thus, ontology model 140 may alternatively be an industry standard “off-the-shelf” model, or a custom ontology model generated with the use of an ontology modeling tool.

After ontology model 140 is generated, an ontology-to-schema generator 220 creates federated schema 150. Federated schema may be a relational database schema, or an XML schema or another schema. Preferably, ontology-to-schema generator 220 creates relational database tables and fields, or XML complex types and elements, which correspond respectively to the classes of ontology model 140 and their properties.

Reference is now made to FIG. 3, which is a simplified block diagram of an enterprise information system that uses the federated schema of FIG. 1 to query across multiple databases using common semantics, in accordance with a preferred embodiment of the present invention. Shown in FIG. 3 are three enterprise data assets; namely, a DB2 relational database 310, an MS SQL relational database 320, and a repository of XML documents 330. Each data asset includes a corresponding wrapper 340, which serves to identify the data asset. Preferably, wrappers include meta-data that identifies the server or servers that house the data asset, access control information including users and passwords, and the constructs that comprise the data assets.

Also shown in FIG. 3 is a federated database 350, such as IBM's DB2 Information Integrator, which supports access to the three databases 310, 320 and 330. Using federated database 350, the databases 310, 320 and 330 can be queried. However, in order to use federated database 350, a user must be intimately familiar with the semantics of the individual databases 310, 320 and 330.

By using federated schema 150 in accordance with the present invention, a generic translation layer can be generated, which buffers the user from the specific semantics of the individual databases 310, 320 and 330, and enables him to express database searches using the semantics of ontology model 140. Specifically, as shown in FIG. 3, the user can query the databases using a query engine 360, which conforms to ontology model 140. Query engine 360 generates and processes queries expressed generically in terms of ontology model 140 and, using the translation layer provided by federated schema 150, activates federated database 350 to query across the three databases 310, 320 and 330.

Reference is now made to FIG. 4, which is a detailed block diagram of the enterprise information system of FIG. 3, in accordance with a preferred embodiment of the present invention.

Shown in FIG. 4 is an information integrator 410. Information integrator 410 retrieves data from a plurality of data sources 420, using wrappers 430 to identify the locations and structures of data sources 420. Generally, each data source 420 conforms to a different naming system, making it difficult to work with more than a single data source at a time.

Also shown in FIG. 4 is a design environment 450, referred to as the Unicorn Workbench™. Design environment 450 provides a user workflow for generating an ontology model 460 and for mapping one or more data schemas 470 into ontology model 460 using semantic mappings 480. Data schemas 470 provide the necessary meta-data, or IT catalogue, for using the various data sources 420. As shown in FIG. 4, ontology model 460 preferably uses meaningful semantics, referred to as a common business language, thus making it easy and efficient to work with the many data sources 420 simultaneously.

After data schemas 470 have been mapped to ontology model 460 by Unicorn Workbench 450, the ontology model and semantic mappings are stored on a server 490, which provides a web portal into the enterprise data. A user of the portal formulates queries and generates views using the common business language, which are automatically converted to appropriate information integrator 450 names and executed within information integrator 450, and the results are converted back into the common business language—all transparent to the user. Preferably, view definitions are formulated in terms of a standard query language, such as SQL or XQuery.

The user also uses the common business language to run business applications including inter alia visualization and analysis tools. It may thus be appreciated that ontology model 460 and semantic mappings 480 obviate the need for the user to learn the intricacies of the data naming conventions for the individual data sources 420 and the inter-dependencies among their data.

Reference is now made to FIGS. 5A, 5B and 5C, which are illustrations of a user interface for semantic discovery and query, in accordance with a preferred embodiment of the present invention. As shown in FIG. 5A, a search, referred to as “Semantic Discovery,” for an ontology property named “sumsUnderManagement” of a class named “Employee,” denoted Employee.sumsUnderManagement, produces four relational database tables for checking accounts and savings accounts, each having a column named “balance.”

As shown in FIG. 5B, detailed information about the database tables is provided when a user clicks on the corresponding entry in FIG. 5A. Thus, according to a business rule named “calculateSumunderManagement,” which relates properties of class Employee, the property sumsUnderManagement is determined by summing the properties branch.savingAccounts.balance and branch.checkingAccounts.balance. Moreover, the table “CHECKINGACCT” belongs to the DB2 relational database CHECKING, and the column “balance” of table CHECKINGACCT corresponds to the ontology property “balance” of ontology class “CheckingAccount.”

Shown in FIG. 5C are query results for individual employees and the sums they manage. It may thus be appreciated that the present invention enables a user to (i) express a query that spans multiple databases residing on multiple servers, in terms of a coherent ontology property such as Employee.sumsUnderManagement; and (ii) discover the sources of the data required to determine the results of the query.

In the foregoing specification, the invention has been described with reference to specific exemplary embodiments thereof. It will, however, be evident that various modifications and changes may be made to the specific exemplary embodiments without departing from the broader spirit and scope of the invention as set forth in the appended claims. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

Claims

1. A system, implemented in a data processing system, for managing and querying a plurality of inter-related data sources, the system comprising: an interface to the plurality of inter-related data sources of diverse types, wherein each of the plurality of inter-related data sources has a corresponding schema that describes a corresponding data structure of a corresponding data source;a schema generator communicating with said interface, wherein the schema generator generates a federated schema that describes structures of the plurality of inter-related data sources and inter-relationships of the plurality of inter-related data sources, and wherein the schema generator further modifies the federated schema over time as the plurality of inter-related data sources undergo changes;a query generator communicating with said schema generator, wherein the query generator generates a query for the federated schema; anda storage device in communication with the data processing system, the storage device operable to store the query.

2. The system of claim 1 further comprising a query engine communicating with said interface, wherein the query engine executes the query to the federated schema and for returning query results.

3. The system of claim 1 wherein said query engine is a component of a business process management environment.

4. The system of claim 1 further comprising a schema-to-ontology generator communicating with said interface and with said schema generator, wherein the schema-to-ontology generator maps the plurality of inter-related data sources into a common ontology model using semantic mappings, and wherein said query generator generates the query formulated in terms of nomenclature of the common ontology model.

5. The system of claim 4 further comprising a view generator communicating with the schema-to-ontology generator, wherein the view generator generates a database view of the plurality of inter-related data sources, using the federated schema, wherein the database view is formulated in terms of nomenclature of the common ontology model, and wherein said query generator generates the query for the database view.

6. The system of claim 5 wherein the database view is formulated using SQL statements.

7. The system of claim 5 wherein the database view is formulated using XQuery statements.

8. The system of claim 5 wherein said view generator updates the database view when the common ontology model is changed.

9. The system of claim 4 further comprising a portal communicating with the schema-to-ontology generator, wherein the portal manages user access privileges to the plurality of inter-related data sources, and wherein the user access privileges are expressed in terms of nomenclature of the common ontology model.

10. The system of claim 4 further comprising a data thesaurus communicating with the schema-to-ontology generator, wherein the data thesaurus looks-up a meaning of a data construct within the plurality of inter-related data sources, wherein the meaning is relative to the common ontology model.

11. The system of claim 10 wherein said data thesaurus traces an origin of a data construct within the plurality of inter-related data sources.

12. The system of claim 10 wherein said data thesaurus traces query results back to data sources.

13. The system of claim 4 wherein said query generator comprises a web-based user interface for creating and editing queries while browsing the common ontology model.

14. The system of claim 4 wherein the common ontology model includes classes, properties of classes, inheritances among classes, and business rules relating properties of classes, and wherein said query generator formulates a semantic query from an expression language that uses the classes, the properties, the inheritances, and the business rules.

15. The system of claim 14 further comprising a view generator communicating with the schema-to-ontology generator, wherein the view generator generates a database view of the plurality of inter-related data sources, using the federated schema, wherein the database view is formulated in terms an expression language that uses the classes, the properties, the inheritances, and the business rules, and wherein said query generator generates a query for the database view.

16. The system of claim 15 wherein said view generator automatically embeds the business rules within view definitions.

17. The system of claim 15 wherein said view generator updates the database view when a relevant business rule is changed.

18. The system of claim 4 wherein the common ontology model provides common semantics for referencing data assets, and wherein the common ontology model encapsulates constructs from the plurality of inter-related schemas, and wherein the constructs comprise: database tables;corresponding fields of the database tables;interrelationships of the database tables through foreign keys;XML complex types;elements of the XML complex types;type inter-relationships of the XML complex types; andbusiness rules that relate table fields to one another and XML elements to one another.

19. The system of claim 1 wherein the plurality of inter-related data sources of diverse types include at least one Microsoft SQL database and one XML document data source.

20. The system of claim 1 wherein the plurality of inter-related data sources of diverse types include at least one Microsoft SQL database and one IBM DB2 database.

21. The system of claim 1 wherein the plurality of inter-related data sources of diverse types include at least one XML document data source and one IBM DB2 database.

22. The system of claim 1 wherein said schema generator modifies the federated schema such that at least one of the plurality of inter-related data sources appears to be removed.

23. The system of claim 1 wherein said schema generator modifies the federated schema when a new data source is added to the plurality of inter-related data sources.

24. The system of claim 23 wherein said schema generator automatically detects and scans new data sources.

25. The system of claim 1 wherein said schema generator modifies the federated schema when one of the plurality of inter-related data sources is modified.

26. The system of claim 1 wherein the federated schema is an ontology model, and wherein each corresponding schema of the plurality of inter-related data sources is mapped to the ontology model.

27. A method, implemented in a data processing system, for managing and querying a plurality of inter-related data sources, the method comprising: generating a federated schema that describes structures of the plurality of inter-related data sources and inter-relationships of the plurality of inter-related data sources;modifying the federated schema over time as the plurality of inter-related data sources undergo changes, wherein each of the plurality of inter-related data sources has a corresponding schema that describes a corresponding data structure of a corresponding data source;generating a query for the federated schema; andstoring the federated schema.

28. The method of claim 27 further comprising: executing a query to the federated schema; andreturning query results.

29. The method of claim 28 wherein said executing is performed by a software application.

30. The method of claim 29 wherein said executing is performed by a business intelligence tool.

31. The method of claim 29 wherein said executing is performed within a business process management environment.

32. The method of claim 27 further comprising mapping the plurality of inter-related data sources into a common ontology model using semantic mappings, and wherein generating the query comprises formulating the query in terms of nomenclature of the common ontology model.

33. The method of claim 32 further comprising generating a database view of the plurality of inter-related data sources, using the federated schema, wherein the database view is formulated in terms of nomenclature of the common ontology model, and wherein generating the query comprises generating the query for the database view.

34. The method of claim 33 wherein the database view is formulated using SQL statements.

35. The method of claim 33 wherein the database view is formulated using XQuery statements.

36. The method of claim 33 wherein generating the database view comprises updating the database view when the common ontology model is changed.

37. The method of claim 32 further comprising managing user access privileges to the plurality of inter-related data sources, wherein user access privileges are expressed in terms of nomenclature of the common ontology model.

38. The method of claim 32 further comprising looking up a meaning of a data construct within the plurality of inter-related data sources, wherein the meaning is relative to the common ontology model.

39. The method of claim 38 wherein looking up comprises tracing an origin of a data construct within the plurality of inter-related data sources.

40. The method of claim 38 wherein looking up comprises tracing query results back to at least one data source.

41. The method of claim 32 wherein generating the query comprises employing a web-based user interface for creating and editing queries while browsing the common ontology model.

42. The method of claim 32 wherein the common ontology model includes classes, properties of classes, inheritances among classes, and business rules relating properties of classes, and wherein generating the query comprises formulating the query from an expression language that uses the classes, the properties, the inheritances, and the business rules.

43. The method of claim 42 further comprising generating a database view of the plurality of inter-related data sources, using the federated schema, wherein the database view is formulated in terms an expression language that uses the classes, the properties, the inheritances, and the business rules, and wherein generating the query comprises generating the query for the data base view.

44. The method of claim 43 wherein generating the database view comprises automatically embedding the business rules within view definitions.

45. The method of claim 43 wherein generating the database view comprises updating the database view when a relevant business rule is changed.

46. The method of claim 27 wherein the plurality of inter-related data sources of diverse types include at least one Microsoft SQL database and one XML document data source.

47. The method of claim 27 wherein the plurality of inter-related data sources of diverse types include at least one Microsoft SQL database and one IBM DB2 database.

48. The method of claim 27 wherein the plurality of inter-related data sources of diverse types include at least one XML document data source and one IBM DB2 database.

49. The method of claim 27 wherein generating the federated schema comprises modifying the federated schema such that at least one of the plurality of inter-related data sources appears to be removed.

50. The method of claim 27 wherein generating the federated schema comprises modifying the federated schema when a new data source is added to the plurality of inter-related data sources.

51. The method of claim 50 further comprising automatically detecting and scanning new data sources.

52. The method of claim 27 wherein generating the federated schema comprises modifying the federated schema when one of the plurality of inter-related data sources is modified.

53. A computer-readable storage medium storing program code for causing a computer to perform the steps of: generating a federated schema that describes structures of the plurality of inter-related data sources and inter-relationships of the plurality of inter-related data sources;modifying the federated schema over time as the plurality of inter-related data sources undergo changes, wherein each of the plurality of inter-related data sources has a corresponding schema that describes a corresponding data structure of a corresponding data source; andgenerating a query for the federated schema.