Patent Application
20080155574
With the expansion of the internet in recent years, more and more sources of data have become available to users. These sources of data are often compiled and maintained by different entities. Therefore, the data sources may have different storage formats, as well as, different interfaces that are defined to access the data. For these reasons, enterprise systems have been limited with respect to the data that is available for access. Typically, each enterprise solution would expect a unique data structure for processing or display. Conversion programs, typically called interface modules, need to be written to allow interfacing to each individual data source. Further, these conversion programs may be specific to each application. Over time, many enterprise systems developed interface modules to popular data sources either due to consumer demand or due to strategic alliances between companies. For example, a corporation may acquire multiple technology companies and operate the companies as complimentary business units. Many data sources have different data elements for common entities such as customers, users, or products. Therefore, accessing all of the available data on an entity may be difficult when the data is spread out across a number of data sources. Providing a unified view of data elements from a variety of data sources may be useful for many applications.
However, each application may have different requirements. For some applications data access may be very time sensitive and the need to retrieve data immediately is a primary concern. Other applications may place a higher value on lower overhead operations, for example less software to install and maintain, as well as, less connection management. New data from existing data sources should be made available to the consumer or application transparently without the installation of new software. Further, the system should accommodate a variety of application requirements. For example, online web searches require frequent access to some data elements, while access to other data elements may be very infrequent. It is also important for consumers and application developers to have knowledge of all the data elements available that are associated with a particular entity.
In view of the above, it is apparent that there exists a need for an improved data access scheme for online systems.
In satisfying the above need, as well as overcoming the enumerated drawbacks and other limitations of the related art, the present invention provides a meta-data driven data access system.
The system provides access to a plurality of data sources from a calling application. The system includes a client application programming interface (“API”) and a broker server. The client API receives a request for attributes from the calling application over an application interface. The client API includes a meta-data bank, a data source adapter, and a broker source adapter. The meta-data bank includes a list of attribute structures where each attribute structure is associated with an adapter by the meta-data bank. Accordingly, the client API accesses the attribute structures in the meta-data bank corresponding to the requested attribute data and retrieves attribute data through the associated adapter. As such, the client API retrieves attribute data from the data source adapter installed with the client API. If the attribute structure is not contained within the meta-data bank of the client API, the client API requests the attribute structure from the broker server through the broker source adapter.
The broker server also includes a meta-data bank and source adapters. Accordingly, the broker server accesses the meta-data bank on the broker server to identify the adapters associated with each attribute structure. The broker server then retrieves attribute data from the adapters associated with each attribute structure. Both the client API and the broker server may develop execution plans identifying the order that the attribute data will be retrieved. The execution plan is particularly helpful with regard to compute adapters or format adapters that generate attribute data based on other attribute data. In addition, the client API and broker server may store the execution plan to enhance the efficiency when retrieving data for frequently requested groups of attribute data.
Further objects, features and advantages of this invention will become readily apparent to persons skilled in the art after a review of the following description, with reference to the drawings and claims that are appended to and form a part of this specification.
Referring now to
A user interacts with a browser 18, for example on a personal computer. The browser 18 produces an HTTP request that may be communicated, for example over the internet, to the calling application 20 as denoted by line 19. The calling application 20 may perform various functions to generate a display for the browser 18 including, for example, generating and formatting content by accessing data, performing calculations on data, or other processing. As mentioned above, the calling application 20 may be an enterprise solution and, therefore, may be enhanced by accessing a wide variety of data sources. Typically, each data source would require a separate data interface to be written specifically to access each individual data source. However, the client API 12 interfaces with the calling application 20 and provides access to a wide variety of data sources through a single interface.
The client API 12 includes a meta-data bank 22, a source adapter 24, and a broker source adapter 26. The meta-data bank 22 stores attribute structures, described in more detail below, local to the client API 12. The meta-data bank 22 includes a listing of attribute structures, each attribute structure 100 corresponding to attribute data 21 that may be requested by the calling application 20. The client API 12 searches the meta-data bank 22 to determine if each corresponding attribute structure 100 is available locally. If each of the attribute structure 100 is available locally, the client API 12 will determine if a local data source adapter, such as data source adapter 24, can provide access to the appropriate data source, such as data source 25. Alternatively, the client API 12 may request the attribute data 21 from the broker server 14. The client API 12 is scalable in that one or many adapters may be installed in the client API 12 for local access. However, whether the client API 12 utilizes a local adapter or the broker server 14 to retrieve the attribute data 21 will be transparent to the calling application 20 and inherently processed by the client API 12.
Referring now to
Referring again to the ID list 104, the adapter arguments ID may be used to define variables or provide configuration information needed by the adapter to access the appropriate data source. In addition, it is noted that the same set of adapters may be re-used across multiple attributes, for example, by changing the adapter arguments for each attribute. Similarly, the data type and data representation may be used in defining and formatting the piece of data that is retrieved from the data source through the data source adapter or compute adapter.
For example, the data type may typically include integers, strings, list of integers, floats, Booleans, etc. Further, the data representation may further define the data format which may include day-month-year, month-day-year, AM/PM, 24 hour, etc. Similar to the adapter 108, the data dictionary 114 may include a foreign key to a data dictionary structure, as denoted by line 120. The data dictionary 114 may include a list 118 of the potential data values returned for an attribute, as well as, a corresponding descriptive meaning of the values. In one example, a tenure attribute may have a value of 1 that indicates 1-7 years, where a value of 2 indicates 8-15 years, and so on. As noted by column 116, a foreign key may be used to further define the structure of a data dictionary attribute in the data dictionary attribute list 118.
The attribute ID list 104 may also include dependencies of the attribute structure 100 on other attribute structures in the meta-data bank 22. These dependencies are often used by compute adapters, such as compute adapter 46 (in
Referring again to
However, if the requested attribute structure 100 is not available in the local meta-data bank 28, the broker server 14 may access a data catalog 16 that serves as a global repository for all attribute structures, as shown by meta-data bank 34. As previously discussed, application designers may be provided access to attribute information for all of the attribute structures in the meta-data bank 34 through an attribute viewer 35. The broker server 14 may communicate with the data catalog 16 through a software interface that may take one of many forms. For example, a distributed architecture interface between the broker server 14 and the data catalog 16 may be implemented over a local area network 40, such as Ethernet.
From time to time, attribute structures stored in the data catalog 16 will need to be updated. The data catalog 16 is configured to automatically update local meta-data banks 22, 28 of the broker server 14 or client API 12 according to meta-data bank 34. If the attribute structure is not available locally through the meta-data bank 28 or the data catalog 16, it is understood that another broker server may be accessed through another broker source adapter (not shown), or an attribute error may be generated and provided back to the client API 12. Similarly, if the attribute structure is available, however, the associated data source adapter, compute adapter, or data format adapter is not available, then an adapter error may be returned to the client API 12 by the broker server 14.
Once the adapter and dependencies for each attribute structure 100 have been retrieved from the meta-data bank 28, the broker server 14 may generate an execution plan 42 that defines the order in which the attribute data 21 is retrieved from the data source adapter 30, 32, calculated by the compute adapter 36, and/or formatted by the data format adapter 38. In addition, for frequently requested groups of attribute data the execution plan 42 may be saved in an execution plan cache 44 to enhance the efficiency of retrieving the attribute structures. The execution plan 42 is then executed and each attribute structure is processed by the broker server 14 according to the execution plan 42. It is well understood that the execution plan 42 may process each attribute structure sequentially, or alternatively may process certain attribute structures in parallel where appropriate based on attribute dependencies. In addition, any execution plans stored in execution plan cache 44 are automatically updated or erased when any of the meta-data banks 28, 34 are updated. When all of the attribute data requested by the client API 12 has been retrieved by the broker server 14, the results are returned to the client API 12.
Similar to the broker server 14, the client API 12 develops an execution plan 50 based on the attribute data requested from the calling application 20. If some of the attribute structures are available in the local meta-data bank 22 and the attribute data is available through local data source adapters, such as data source adapter 24, then portions of the execution plan 50 may be processed in parallel with the request to the broker server 14 for additional attribute data, as long as, no attribute dependencies prevent further execution. In addition, compute adapters, such as compute adapter 46, and data format adapters, such as data format adapter 48, may be processed later in the execution plan 50 based on attribute dependencies. Similar to the broker server 14, compute adapter 46 may calculate attribute data based on one or more inputs where the inputs may be other retrieved or calculated attribute data. The data format adapter 48 may format the data based on retrieved attribute data, for example, retrieving a date in European format (day/month/year) and formatting it in US format (month/day/year) prior to providing it to the calling application 20.
Similar to the broker server 14, the client API 12 may also save frequently used execution plans 50 in an execution plan cache 52. In addition, any execution plans stored in execution plan cache 52 are automatically updated or erased when any of the meta-data banks 22, 28, or 34 are updated. Further, the client API 12 may include cookie settings 54 that are provided to the calling application 20. The cookie settings 54 may be provided to the calling application 20 such that the attribute data, the execution plan, or both may be stored as a cookie 56 by the browser 18 on a user PC. The meta-data for an attribute may be used to determine if the attribute data should be cached in the cookie 56, as well as, determining how long the data should be valid for once cached.
Now referring to
As a person skilled in the art will readily appreciate, the above description is meant as an illustration of implementation of the principles of this invention. This description is not intended to limit the scope or application of this invention in that the invention is susceptible to modification, variation and change, without departing from the spirit of this invention, as defined in the following claims.
