The embodiments disclosed and claimed herein are related to computer systems, and more particularly, databases.
Today's businesses have sophisticated data analysis requirements. The metrics or analyses of a business's data can be difficult to obtain. To calculate a meaningful metric, business analysts often use spreadsheets to manually analyze data. Manual analysis, of course, is a tedious and time-consuming process.
Most applications fail to deliver useful metrics that provide unique insights into an organization's performance. Useful metrics highlight significant performance measures of the business. Typically, business analysts must execute multiple queries and other time-consuming manual interventions to produce these metrics. Then, despite the time-consuming effort, analysts must start the process anew to obtain follow-up information such as an explanation of a particular anomaly in a metric.
Typically, a business's data is stored on a database or on databases. These databases are operated with associated database servers, which manage the storage and retrieval of records from the databases. Analytical servers have additionally been provided to format database queries or information requests sent from a client user interface to the database server for handling. The analytical servers can be used to improve the efficiency of the database accesses and to provide metrics of interest to the user from the retrieved records from the database.
The embodiments disclosed below provide an analytical server which efficiently accesses a Relational Database Monitoring System (“RDBMS”) comprising a database and a database server. The database in this approach includes fact and dimension tables which may be, for example, configured in a star schema having a central fact table with surrounding dimension tables to form the star structure. Aggregate tables may also be provided which aggregate measures from the fact table at a higher hierarchical level than such measures are maintained in the fact table. Metadata is further stored in the database, where the metadata describes the organization of the various tables in the database, and specifically the metadata in the embodiments described below includes information about the hierarchical levels of various dimensions of the above-mentioned tables and star schema.
With further reference to the metadata stored in the database in the below-described embodiments, the analytical server described herein receives the metadata from the database and analyzes that metadata, including the hierarchical information, in order to provide relatively efficient access to the tables of the database in response to a query from a user. Such efficient access preferably supports calculation of complex metrics which might otherwise be difficult or impossible. Supported levels of stars are defined and analyzed in a sophisticated and efficient manner which facilitates the calculation of chameleon and allocated metrics. Additionally, fact level security and dimensional security are supported, as well as efficient collection and analysis of aggregate table usage statistics.
Referring now to
The database 110 is accessible by the analytical server engine 120. The analytical server engine 120 accepts requests for metric calculations from clients 135, uses the metadata structures 145 to identify the necessary fact components and the best star schema for accessing them, generates and executes structured queries in a database query language, such as Structured Query Language (SQL), performs outer joins to conform query results, calculates the desired metrics, and returns them to the clients in a structured form such as a multidimensional cubes.
The clients access the analytical server via an application programming interface (API) 140, through which metrics can be requested, possibly constrained on dimensional values. The query and metric calculation results are transmitted through the interface as objects. The client need not have knowledge of how the metric is calculated.
The database 110 includes a collection of fact tables 125a and dimension tables 125b organized in multiple star schemas 125. Exemplary star schemas are described in Ralph Kimball, T
The metadata structures 145 provides information for the analytical server 120 to determine how to access the database 110 for the values required to construct requested metrics and defines more abstract constructs, such as particular metrics which can be computed from one or more facts in the database 110. As will be described below, the metadata structures 145 include structures for hierarchies, stars, measure indicators, and metric indicators.
I. Metadata Structures
A. Hierarchies
Referring now to
Each level 210 is associated with a level name 210a, level number 210b, and column name 210c. The level names describe the grouping of the elements. In the exemplary case described in
For a dimension table 125b to be associated with a hierarchy 205, the dimension table 125b must contain the column names 210c specified for the hierarchy 205 for the levels 1 . . . n. Multiple dimension tables 125b may be associated with the same hierarchy 205 and support it to different levels. For example, a dimension table 125b for Time might contain columns only for Year, Quarter, and Month, and therefore provided a supported level of “3,” while a more complete dimension table might contain columns for all levels down to “day,” and therefore offer a supported level of “5.”
B. Stars
As shown in
The star 300 comprising the fact table 125a and the surrounding dimension tables 125b can be used to apply selection constraints and specify aggregate groupings when retrieving the fact values. A number of different stars can be identified in a database.
A star metadata structure 305 such as shown in
The supported levels of the specific aggregate table 130 in the database are represented in a star metadata structure 305 by an array 310 of integers in which elements 315 of the array 310 represent the supported hierarchical levels defined in a predetermined order. Additionally, an initialization process might ensure that the supported levels 210 are valid in all stars 300, thereby eliminating the need for checking the column names during the star selection process.
Still referring to
The star metadata structure 305 may also include a flag 313 indicating the availability of the star 300. Where the star is properly maintained or refreshed by some other mechanism, the flag 313 can be set to indicate whether the star is available. The foregoing flag 313 can be examined during star selection.
The stars 300 are collected into groups called stargroups. Aggregate tables 130 are built for frequently accessed data, in a manner that reduces table size, join complexity, a query time. For example, sales figures might be accumulated at the “day” level in one aggregate table 130, and summarized more highly to the “month” level in some other aggregate table 130. The stargroup used for accessing sales figures might contain two stars 300, possibly using exactly the same dimension tables 125b but each pointing to different aggregate tables 130. The star 300 using the monthly aggregate fact table 130 would be assigned a higher aggregate rank, or in other words would contain measures at a higher hierarchical level, and would be preferred when values were not required at a finer grain than month.
C. Measure Indicator
Referring now to
The measure indicator 405 includes an identifier 410, which identifies the facts within the database 110 that are being referred to. Also included in the measure indicator is a flag 411 which indicates whether or not the measure is additive. The measure indicator 405 also contains a query language snippet 412. To support aggregate navigation, the snippet 412 is defined using a syntax which allows substitution of specific fact table 125a names and dimension table 125b names. For example, a non-SQL character is used to delimit a substitutable form which is to be replaced by the name of the fact table in the associated star, prior to executing a query.
Additionally, different stargroups may require that the snippet 412 be written differently. Accordingly, a measure may contain a plurality of snippets 412, each associated with an indicator 415 indicating a particular stargroup. Verification that the columns specified in the snippets 412 actually appear in the aggregate tables 130 defined by each star 300 in the associated stargroup can be done during an initialization process, thereby limiting column name lookups.
A measure also may be semi-additive which means additive in some dimensions and not additive in others. Inventory levels, for example, may be additive in the product dimension but non-additive in the time dimension. To accommodate queries of measure across different dimensions and to properly account for the varying additive/non-additive properties of measures in different dimensions, a second “additivity” flag may be provided at 416, which might for example, be associated with the snippet/stargroup 412/415 pairing. In this approach, the measure 405 could be indicated as “non-additive” but could then be separately indicated as “additive” for a particular snippet by flag 416. Conversely, such semi-additivity could be indicated by a positive “additivity” indicators by particular snippet in flag 416. The use of these additivity flags will be discussed in the section below relating to non-additive measures and metrics.
Use of multiple snippets 412 for different stargroups are advantageous for calculation of chameleon metrics. Chameleon metrics represent a general concept, the exact definition or calculation of which is dependent on the dimension or level. For example, a cost metric when viewed by the product dimension, may measure production or part cost. However, when viewed by dimensions other than product, the cost include the total product cost across all parts, freight, taxes, and other top-level costs.
Chameleon metrics are constructed by taking advantage of the provision for multiple snippet 412/stargroup pairs in the underlying measure definitions. Using the Geography vs. Product forecast example, a measure is defined which uses two different stargroups. The snippet 412 associated with the first stargroup can cause the measure to be calculated in accordance with a first definition while the snippet 412 associated with the second definition cause the measure to be calculated in accordance with a second definition.
Fact-based security can also be provided, for example, by providing additional security hierarchy fields 418, 419 within the measure indicator 405. By defining for particular measures a security hierarchy, it is possible to grant access to particular users or clients according to levels of fact-based data by defining security hierarchy levels on a measure-by-measure basis. For ultimate flexibility, the security hierarchy is defined in the measure indicator both at the broad level in field 418 and at the specific snippet level 419. The definition at these different levels allows the facts to be accessed according to the measure's use within the star structure 300 or aggregate table 130 being accessed rather than just having a broad prohibition of accessing certain types of data by certain users or clients.
D. Metric Indicators
Referring now to
II. Metric Calculation
A. Aggregate Navigation
Referring now to
At step 622, the analytical server 120 selects a particular measure and associated stargroup. Within the aggregate stargroup, the analytical server 120 selects (step 625) the star 300 associated with the most highly aggregated aggregate table 130 and determines whether the aggregate table 130 supports each constrained dimension at the level required. The foregoing is measured by comparing (step 630) the level that each dimension is constrained in the metric request with the array 310 describing the supported levels 210 of the dimensions. Wherein the array 310 indicates that each constrained dimension is supported at the same or higher level, the aggregated table 130 associated with the star 300 is selected (step 630).
Wherein one or more constrained dimensions are not supported, or supported at a lower level, the aggregate table 130 associated with the star 300 is rejected (step 635), and a determination is made whether any remaining stars 300 are present in the stargroup. Wherein a remaining star 300 exists in the stargroup, the star 300 associated with the next most highly aggregated table 130 is selected (step 650) and steps 630-650 are repeated. Wherein there are no remaining stars 300, no aggregate table 130 is selected (step 655) for the particular measure and calculation of the measure must be conducted by querying the database 110. Steps 622-655 are repeated for each measure required for the requested metric(s).
After selecting the aggregate table 130, the analytical server 120 generates and conducts the queries for each measure on the selected aggregate table 130 (step 660). The queries are generated by substituting the fact 125a and dimension table 125b names where indicated in the snippets 412 associated with the selected star 300. After generating the queries, the analytical server 120 calculates the measures (step 665), calculates the metrics (step 670), and forwards the result to the client (step 675), thereby completing calculation of the metric.
The foregoing approach also permits maintenance of statistics which indicate the usage levels of each star 300. For example, statistics can monitor events such as when a star 300 is considered for selection and rejected, a star 300 is selected for use, and when a star 300 is actually used in a query. The required and supported hierarchical levels can also be recorded, thereby permitting examination of usage levels. From the foregoing information, it can be determined in a given circumstance that an additional level of detail should be added to the aggregate table 130 because a majority of requests required the additional hierarchical level. Additionally, a determination can be made that the aggregate table 130 can be consolidated without major effect on overall performance because a majority of requests require one less level of aggregation.
B. Carpooling
Certain queries can be conducted using a common aggregate table 130. For example, certain aggregate tables 130 can include multiple aggregated facts. Wherein multiple queries request different measures, but with identical constraints, the aggregated facts can be combined into a single structured query, such as a SELECT statement in SQL. Alternatively, where in multiple queries, all but one constraint are identical, and the different constraint is constrained at the same level, the queries can also be combined.
The foregoing is known in the art as carpooling. The analytical server 120 can advantageously preprocess the requisite queries, possibly allowing a number of queries to be combined into a single query, resulting in relational database 110 access efficiencies.
Referring now to
At step 705, the analytical server 120 determines the aggregate table 130 from which to calculate each measure. At step 707, the analytical server determines which of a plurality of queries can be combined when accessing the database 110. In order to combine queries, the same fact table 125a is common between the queries to be combined, and there will be commonality to at least some of the dimension tables 125b between the queries as well. The queries can be combined for a single star or among a number of stars 300, so long as there is the requisite commonality among the fact and data tables 125a-b. The determination of step 707 involves a determination of the hierarchical levels involved in the plurality of queries, and it is possible that even if a requested metric or metrics requires the same measures but at differing hierarchical levels, it may be possible to consolidate these into a single query of the database 110 and then extract the desired information needed for the different metric requests. For example, if some metric is broken down over the last six months and also for the corresponding six months in the previous year. The underlying measure for both requests can be obtained in a single query, simply by placing all desired month numbers in the “IN (l, m, n)” constraint, and selectively processing the results.
At step 710, the analytical server 120 carpools combinable queries to reduce the number of queries actually made of the database 110 through the RDBMS 105. After carpooling the queries, the analytical server 120 generates the structured query commands for each of the database queries (step 715) and forwards (step 720) the structured database query commands to the RDBMS 105.
C. Non-Additive Metric Calculation
It is noted that it is often desirable to display metrics broken down across dimensional levels, and simultaneously display a roll-up or total. Provided all the measures that have been broken down across dimensional levels are additive, the requester of the metric can simply total the returned results. However, this is incorrect wherein certain measure components of the metrics are non-additive. Correct totals can only be obtained if the requester has knowledge of which measures are non-additive and asks for the non-additive measures separately.
By using the additive/non-additive fields 411,416 described with respect to
Referring now to
At step 820, a separate totals query is generated for each non-additive measure. The query is launched using the stars as described above, and it is noted that the totals query typically requires a shallower hierarchical level on at least one dimension. Accordingly, the totals query may actually be obtained using a more highly aggregated table. Finally, at step 825, the metric is calculated and the process is terminated. In the foregoing manner, complex metrics composed of any combination of additive and non-additive measures can be calculated correctly and efficiently, without requiring any knowledge or action on the part of the requestor.
As an example, using the additive/non-additive fields 411,416, the analytical server 120 knows which measures are non-additive. Accordingly, the handling of the non-additive measures can be handled transparently, without making the non-additive attribute visible to the requester. This minimizes the chances of displaying invalid results.
To accomplish the non-additive roll-up, the analytical server 120 extends the metric result object to contain an additional multidimensional array of totals. So for example if there were a requested metric relating to monthly sales of product by business unit, the request would be for a three-dimensional cube (business unit×month×metric values). If the sales were not additive across the product dimension a separate two-dimensional array would be requested and returned (as just month×metric values), with the values representing the totals across all business unit. Alternatively, the original three-dimensional cube might simply reserve one extra element in the first dimension to contain the totals.
In the simple case where it turns out all component measures are additive, the analytical server 120 issues queries at the detail level only (business unit), and performs simple sums to calculate the totals. The individual measures are summed, and then the metric level calculations are performed using these sums. The important thing in the specific example is to average the totals rather than totaling averages.
When a measure is non-additive, the analytical server 120 instead generates and issues two separate queries, the extra query being for the total level (omitting the SELECT item and GROUP BY for Business Unit). In this way, complex metrics composed of any combination of additive and non-additive measures can be calculated correctly and efficiently, without requiring any knowledge or action on the part of the requester. The additive/non-additive fields 411, 416 (see
As an additional benefit, there may be cases where no star is available at a certain hierarchical level, in which case the analytical server 120 may attempt to obtain the measures and calculate the metric at the total level only (even in the case where the measures are all additive). This can be done, for example, when the intention is to compare two metrics, such as sales vs. forecast, as when sales can be broken down by industry, customer, etc. but forecast is only available by product. In this case, forecast could still be compared to total sales across all industries or customers. To best support this capability, the server further extends the result object to provide indicators distinguishing such indicators as “all zero results”, “no data found”, “detail level not supported”, and so forth.
D. Cross Star Joins
Many metrics must be calculated using measures obtained from different stars. For example, a metric which measures the days that inventory is on hand might be calculated by measuring the current inventory by the sales per day. Wherein one star measures sales and another star measures inventory, calculation of the days that inventory is on hand requires calculation of measures from both the sales star and the inventory star.
The analytical server 120 accesses the measures separately from each star 300, and then performs the equivalent of an outer join on the results. The different sets of results along the hierarchical level supplied in the request and retrieved by the queries are carefully “lined up”, thereby allowing the server 120 to encapsulate this knowledge and processing, and make sophisticated metrics available to the requester. The maintenance of the hierarchical levels in the metadata structures 145 associated with the various stars 300 and associated aggregate tables 130 expedites the efficient “lining up” process by allowing the analytical server 120 to quickly assess which of the available structures are available and to find the coarsest available aggregate level for those structures.
E. Invariant Metrics
Certain measures or metrics are “invariant” by dimension. For example, to calculate the metric sales per sales rep, a measure must exist for the denominator which gives the number of sales reps. Furthermore, it may be useful to look at the sales per rep metric broken down by product business unit, family, or item. If the number of sales reps is maintained in a sales forecast star, it can be accessed only by sales geography and time. However, since all reps sell all products, the measure reporting number of reps does not change whether we are looking at the business unit, family, or item level, the number of sales reps is invariant along the product dimension. Therefore, the sales forecast star is degenerate along the product dimension. The analytical server is equipped with knowledge of measures which are invariant with respect to certain dimensions. Providing this knowledge to the analytical server allows a single value to be obtained as the invariant measure in the metric calculation, regardless of the level of the dimension to which the measure is invariant.
F. Allocation Metrics
An allocation metric is a metric containing a measure that is not defined at the lowest dimension level, but which is useful and desirable to allocate a value for the metric at the lowest dimension using another measure which is definable at the lowest dimension. For example, Sales Forecast numbers may be available by Geography, Sales Rep, and Time, but not by Product Business Unit. However, suppose that Sales for the previous year are available by Product Business Unit and that it is a reasonable assumption that the breakdown of Sales by Product Business Unit will be similar to the breakdown of Sales Forecast by Product Business Unit. In such a case, the Sales Forecast by Product Business Unit can be calculated by the foregoing expression:
Allocated Forecast for Product(A)=Total Forecast*Sales for Product(A)/Total Sales
In the foregoing case, the measure “Forecast Sales” is the base measure while “Sales Last Year” is known as the control measure. Additionally, it should be noted that while Sales for Product(A) is at the same level as the request, i.e., both are at the Product Business Units, Total Forecast and Total Sales are at different levels, or “allocated levels”.
Referring now to
At step 905, the required levels for the request are determined. In the exemplary case, the required levels are “210.” At step 910, a determination is made whether a star exists with the required levels. As an example, the star metadata structure 305 shown in
In the exemplary case, the lowest ranked star is ranked as “303” which fails on the Product dimension. At step 925, the allocation levels are determined by taking the minimums of the required levels for the request and the levels of the star selected during step 920. The allocation levels are “200” in the exemplary case.
During step 930, an attempt is made to find a star which supports the allocation levels in the base measure, e.g., the sales forecast in the present exemplary case. During step 935, an attempt is made to find a star in the control measure (the Sales Last Year) which support the required levels for the request (“210”). Wherein a star for the base measure is found in step 930 and a star for the control measure is found in step 935, the allocated measure is calculated (step 940), thereby completing calculation of the metric. Wherein a star is not found in either steps 930 or 935, the allocated measure cannot be calculated and calculation of the metric is terminated.
III. Security
A. Dimension Level Security
Data security is provided on both a dimension level and a fact level. Each authorized user of the database can be associated with a particular security level which restricts the levels of each hierarchy which the user is permitted access. For example, regional sales managers can be permitted to only view sales at the regional level and not be authorized access to sales data at the national or worldwide level. Additionally, the users can be restricted access to a particular value of a hierarchical level. For example, a regional sales manager might be permitted to only view sales data from their region.
The dimension level security is provided by defining security groups which specify that all metric requests have to be performed as if the required level of a certain hierarchy is at least some predetermined level. The request is rejected outright if any of the levels are lower than the security levels. The security definitions can also contain rules which force certain constraints. The force constraints are dynamically substituted to a given request.
B. Fact Level Security
It may also be desirable to prevent users from viewing specific metrics. An additional two level hierarchy is defined, wherein level zero is indicative that the data should not be visible, while level one is indicative that the data should be made visible. The supported level for the added hierarchy is set at zero for each restricted metric and one for each unrestricted metric. Users who are restricted are placed in a security group that only permits access to level one of the hierarchy. Therefore, when a restricted user makes a query for the restricted metric, the security definition imposes a dimensional constraint of one for the additional hierarchy. During aggregate navigation, each of the stars will be rejected because the stars only support a level zero aggregation. For users who are permitted to access the restrict metric, the zero level dimensional constraint is imposed. However, each of the stars are support the zero level aggregation.
Although preferred embodiments have been illustrated in the accompanying drawings and described above, the claims shall not limited to the embodiments disclosed, but are capable of numerous rearrangements, modifications, and substitutions without departing from the spirit of the invention as set forth and defined by the following claims and equivalents thereof.
This application is a continuation of U.S. patent application Ser. No. 10/731,773, filed Dec. 8, 2003 now U.S. Pat. No. 7,031,953, pending, which is a continuation of U.S. patent application Ser. No. 09/837,114, filed Apr. 17, 2001, now U.S. Pat. No. 6,662,174, issued Dec. 9, 2003, which claims the benefit of U.S. Provisional Patent Application No. 60/197,894, filed Apr. 17, 2000, and U.S. Provisional Patent Application No. 60/199,975, filed Apr. 27, 2000. All referenced applications/patents are commonly assigned with the present application and are incorporated herein by reference in their entirety for all purposes.
Number | Name | Date | Kind |
---|---|---|---|
5355474 | Thuraisngham et al. | Oct 1994 | A |
5475826 | Fischer | Dec 1995 | A |
5864857 | Ohata et al. | Jan 1999 | A |
5884299 | Ramesh et al. | Mar 1999 | A |
5899992 | Iyer et al. | May 1999 | A |
5905985 | Malloy et al. | May 1999 | A |
5926818 | Malloy | Jul 1999 | A |
5943671 | Kleewein et al. | Aug 1999 | A |
5944825 | Bellemore et al. | Aug 1999 | A |
5983227 | Nazem et al. | Nov 1999 | A |
5991754 | Raitto et al. | Nov 1999 | A |
5991756 | Wu | Nov 1999 | A |
6032144 | Srivastava et al. | Feb 2000 | A |
6038550 | Rosenwald | Mar 2000 | A |
6078926 | Jensen et al. | Jun 2000 | A |
6122636 | Malloy et al. | Sep 2000 | A |
6161103 | Rauer et al. | Dec 2000 | A |
6163774 | Lore et al. | Dec 2000 | A |
6189004 | Rassen et al. | Feb 2001 | B1 |
6199063 | Colby et al. | Mar 2001 | B1 |
6205469 | Graham | Mar 2001 | B1 |
6212515 | Rogers | Apr 2001 | B1 |
6212524 | Weissman et al. | Apr 2001 | B1 |
6282544 | Tse et al. | Aug 2001 | B1 |
6327574 | Kramer et al. | Dec 2001 | B1 |
6366922 | Althoff | Apr 2002 | B1 |
6374263 | Bunger et al. | Apr 2002 | B1 |
6377993 | Brandt et al. | Apr 2002 | B1 |
6385201 | Iwata | May 2002 | B1 |
6385604 | Bakalash et al. | May 2002 | B1 |
6434557 | Egilsson et al. | Aug 2002 | B1 |
6438537 | Netz et al. | Aug 2002 | B1 |
6438538 | Goldring | Aug 2002 | B1 |
6442560 | Berger et al. | Aug 2002 | B1 |
6446059 | Berger et al. | Sep 2002 | B1 |
6446063 | Chen et al. | Sep 2002 | B1 |
6477525 | Bello et al. | Nov 2002 | B1 |
6480836 | Colby et al. | Nov 2002 | B1 |
6484179 | Roccaforte | Nov 2002 | B1 |
6487546 | Witkowski | Nov 2002 | B1 |
6493699 | Colby et al. | Dec 2002 | B2 |
6505205 | Kothuri et al. | Jan 2003 | B1 |
6542895 | DeKimpe et al. | Apr 2003 | B1 |
6556983 | Altschuler et al. | Apr 2003 | B1 |
6662174 | Shah et al. | Dec 2003 | B2 |
6714979 | Brandt et al. | Mar 2004 | B1 |
6718312 | McAfee et al. | Apr 2004 | B1 |
6778709 | Taubman | Aug 2004 | B1 |
20020029207 | Bakalash et al. | Mar 2002 | A1 |
Number | Date | Country |
---|---|---|
WO 9957658 | Nov 1999 | WO |
WO 0020982 | Apr 2000 | WO |
Number | Date | Country | |
---|---|---|---|
20060129564 A1 | Jun 2006 | US |
Number | Date | Country | |
---|---|---|---|
60197894 | Apr 2000 | US | |
60199975 | Apr 2000 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 10731773 | Dec 2003 | US |
Child | 11347468 | US | |
Parent | 09837114 | Apr 2001 | US |
Child | 10731773 | US |