Minimizing staleness in real-time data warehouses

Abstract

Data tables in data warehouses are updated to minimize staleness and stretch of the data tables. New data is received from external sources and, in response, update requests are generated. Accumulated update requests may be batched. Data tables may be weighted to affect the order in which update requests are serviced.

Description

BACKGROUND

1. Field of the Disclosure

The present disclosure relates to updating data warehouses.

2. Description of the Related Art

Data warehouses store data tables that contain data received from external sources. As an example, the data may relate to network performance parameters.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a data warehouse server that receives data and updates data tables in accordance with disclosed embodiments;

FIG. 2 depicts elements of a method for updating data tables in a data warehouse in accordance with disclosed embodiments;

FIG. 3 illustrates selected elements of a data processing system provisioned as a data warehouse server for updating data tables in accordance with disclosed embodiments;

FIG. 4 a is a graph of staleness values related to data tables in a data warehouse;

FIG. 4 b is a further graph of staleness values related to data tables in a data warehouse; and

FIG. 5 illustrates algorithms related to minimizing staleness values for data tables.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

In a particular embodiment, a disclosed method updates data tables stored in a data warehouse. The data warehouse may be a real-time data warehouse. The method includes receiving data for updating the data tables, generating update requests responsive to the receiving, calculating a staleness for a portion of the data tables, and scheduling data table updates on a plurality of processors based at least in part on the calculated staleness and the update requests. The method further includes transforming the data tables based on the scheduled data table updates to include a portion of the received data.

Generally, the staleness is indicative of an amount of time elapsed since the previous update of the data tables. Update requests may be assumed non-preemptible and accumulated update requests are batched together. The method may further include determining a stretch value for the update request, wherein the stretch value is indicative of the maximum ratio between the duration of time an update waits until it is finished being processed and the length of the update.

Further embodiments relate to a server for managing a data warehouse. The server includes a memory for storing the data warehouse, which includes a plurality of data tables. An interface receives data for updating the data tables and a processor for calculating a staleness for a portion of the data tables responsive to receiving data on the interface. Further instructions are for weighting a portion of the calculated stalenesses and scheduling data table updates for completion by a plurality of processors based at least in part on the weighted stalenesses. Accumulated update requests from the generated update requests are batched together.

To provide further understanding of disclosed systems, data warehouses and aspects related to updating data tables are discussed. Data warehouses integrate information from multiple operational databases to enable complex business analyses. In traditional applications, warehouses are updated periodically (e.g., every night) and data analysis is done off-line. In contrast, real-time warehouses continually load incoming data feeds for applications that perform time-critical analyses. For instance, a large Internet Service Provider (ISP) may collect streams of network configuration, performance, and alarm data. New data must be loaded in a timely manner and correlated against historical data to quickly identify network anomalies, denial-of-service attacks, and inconsistencies among protocol layers. Similarly, on-line stock trading applications may discover profit opportunities by comparing recent transactions against historical trends. Finally, banks may be interested in analyzing streams of credit card transactions in real-time to protect customers against identity theft.

The effectiveness of a real-time warehouse depends on its ability to make newly arrived data available for querying. Disclosed embodiments relate to algorithms for scheduling updates in a real-time data warehouse in a way that 1) minimizes data staleness and 2) under certain conditions, ensures that the “stretch” (delay) of each update task is bounded. In some cases, disclosed systems seek to schedule the updating of data tables to occur within a constant factor of an optimal solution for minimizing staleness and stretch.

Data warehouses maintain sets of data tables that may receive updates in an online fashion. The number of external sources may be large. The arrival of a new set of data records may generate an update request to append the new data to the corresponding table(s). If multiple update requests have accumulated for a given table, the update requests are batched together before being loaded. Update requests may be long-running and are typically non-preemptible, which suggests that it may be difficult to suspend a data load, especially if it involves a complex extract transform-load process. There may be a number p processors available for performing update requests. At any time t, if a table has been updated with data up to time r (i.e., the most recent update request arrived at time r), its staleness is t−r.

Given the above constraints, some embodied systems solve the problem of non-preemptively scheduling the update requests on p processors in a way that minimizes the total staleness of all the tables over time. If some tables are more important than others, scheduling may occur to prioritize updates to important tables and thereby minimize “priority-weighted” staleness.

Some disclosed systems use scheduling algorithms to minimize staleness and weighted staleness of the data in a real-time warehouse, and to bound the maximum stretch that any individual update may experience.

On-line non-preemptive algorithms that are not voluntarily idle can achieve an almost optimal bound on total staleness. Total weighted staleness may be bounded in a semi-offline model if tables can be clustered into a “small” number of groups such that the update frequencies within each group vary by at most a constant factor.

In the following description, details are set forth by way of example to facilitate discussion of the disclosed subject matter. It should be apparent to a person of ordinary skill in the art, however, that the disclosed embodiments are exemplary and not exhaustive of all possible embodiments. Throughout this disclosure, a hyphenated form of a reference numeral refers to a specific instance of an element and the un-hyphenated form of the reference numeral refers to the element generically or collectively. Thus, for example, widget 12-1 refers to an instance of a widget class, which may be referred to collectively as widgets 12 and any one of which may be referred to generically as a widget 12.

FIG. 1 illustrates system 100 that includes data warehouse server 118, which as shown maintains a real-time data warehouse. Data warehouse server 118 receives data (over interface 120 through network 106 from server 127) from multiple external sources including data processing system 125, storage server 121, and mail server 123. The received data is stored as new data 114 by processor 110, which also generates update requests 116. Data warehouse server 118 accesses a computer readable medium (not depicted) embedded with computer instructions for managing data tables 108. A particular embodiment includes instructions that maintain data tables 108 as a data warehouse and receive requests with new data 114 for updating a portion of data tables 108. Further instructions generate update requests 116 that correspond to the received data. Staleness values are calculated for individual data tables of data tables 108. For example, staleness values can be calculated for data tables 108-2 and 108-1. The calculated stalenesses can be ranked or compared to a threshold, as examples. Further instructions schedule updating of data tables 108 with updates 102. The scheduling of updating data tables 108 may be based on the ranked stalenesses.

With the addition of updates 102, portions of data tables 108 are transformed based on the scheduling and the update requests 116. Update requests 116, in some embodiments, are for appending new data 114 to data tables 108 as updates 102.

In some embodiments, a stretch value for one or more of data tables 108 is determined and ranking data tables 108 is based at least in part on the calculated stretch. In some embodiments, the calculated stretch value the maximum ratio between the duration of time an update (e.g., an update based on update request 116-3) waits until it is finished being processed and the length of the update. Data warehouse server 118 may batch together accumulated portions of the generated update requests 116. New data 114 is distributed to data tables 108 as updates 102 by the scheduling processors 109 to minimize staleness. As shown, there are a number p processors, which is indicated by scheduling processor 109-p.

In some embodiments, update requests 116 are assumed non-preemptible. A portion of data tables 108 may be analyzed for a staleness value indicative of an amount of time elapsed since a previous update of the portion of data tables. A first portion of data tables 108 (e.g., data table 108-1) may be weighted higher than a second portion of the data tables (e.g., data table 108-2), and the scheduling processors 109 can schedule updates to these data tables responsive to the weighting results.

Illustrated in FIG. 2 is a method 200 is for updating data tables (e.g., data tables 108 in FIG. 1). Data is received (block 201) for updating the data tables. Responsive to receiving (block 201) the data, update requests are generated (block 203). The generated update requests may be non-preemptible. A staleness value is calculated (block 205) for a portion of the data tables. Optionally, the staleness values are weighted (block 207) and a stretch value is calculated (block 209) for data tables. Weighting may occur by multiplying a first data table staleness by a first weight and multiplying a second data table staleness by a second weight. The stretch value can be calculated which indicates the maximum ratio between the duration of time an update waits until it is finished being processed and the length of the update. A determination is made (block 211) whether there are accumulated update requests. If there are accumulated update requests, the accumulated update requests are batched (block 213). Data table updates are scheduled (block 215) for the update requests (as shown, whether batched or not) based at least in part on the calculated stalenesses. The updates may be scheduled to occur at variable intervals.

Referring now to FIG. 3, data processing system 321 is provisioned as a server for managing a data warehouse. As shown, the server includes a computer readable media 311 for storing the data warehouse 301 which has a plurality of data tables 313. The server further has an input/output interface 315 for receiving data for updating the data tables and a processor 317 that is enabled by computer readable instructions stored in computer readable media 311. Processor 317 is coupled via shared bus 323 to memory 319, input/output interface 315, and computer readable media 311. It will be noted that memory 319 is a form of computer readable media 311. In operation, responsive to input/output interface 315 receiving new data, staleness calculation module 303 calculates a staleness for a portion of data tables 313. Staleness weighting module 304 optionally weights a portion of the calculated stalenesses. Staleness ranking module 307 ranks the stalenesses. Update scheduling module 309 schedules data table updates for completion by a plurality of processors based at least in part on the weighted stalenesses. Update request generation module 302 generates update requests for newly received data. In some embodiments, update requests may be batched together by batching update requests module 305. Stretch calculations may be performed for a portion of the data tables 313 by stretch calculation module 306. Accordingly, update scheduling module 309 may schedule data table updates based at least in part on the stretch value. The calculated stretch value, in some embodiments, represents the maximum ratio between the duration of time an update waits until it is finished being processed and the length of the update.

In disclosed methods including method 200, updating the data tables may include appending new data to corresponding data tables. The staleness can be indicative of an amount of time elapsed since a previous update of the data tables. Some embodied methods include scheduling data table updates on p processors based at least in part on the calculated staleness and the update requests. Updating the data tables with the new data transforms the data tables based on the scheduled data table updates to include a portion of received data. Disclosed methods may include weighting a first portion of the data tables higher than a second portion of the data tables, wherein the scheduling is at least in part based upon the weighting.

Further embodiments relate to computer instructions stored on a computer readable medium for managing a plurality of data tables in a data warehouse. The computer instructions enable a processor to maintain a plurality of data tables in the data warehouse, receive data for updating a portion of the plurality of data tables, and generate update requests corresponding to the received data. For individual data tables, a staleness for the data table is calculated and ranked. Updating the data tables is scheduled based on the ranked stalenesses, and the data table is transformed (e.g., appended with new data) based on the updating. A stretch value may be calculated for a portion of individual data tables. The stretch value is indicative the maximum ratio between the duration of time an update waits until it is finished being processed and the length of the update. Further instructions allow for accumulated update requests to be batched and processed together. A portion of update requests may be non-preemptible.

Calculations and other aspects of updating data warehouses are touched on for a better understanding of disclosed systems. Suppose a data warehouse consists of t tables and p processors, and that p≦t. Each table i receives update requests at times r_i1<r_i2< . . . <r_i,ki, where r_i0=0<r_i1. An update request at time r_ij contains data generated between times r_i,j−1 and r_ij. The length of this update is defined as r_ij−r_i,j−1. Associated with each table i is a real α_i≦1 such that processing an update of length L takes time at most α_iL. The constants α_i need not be the same. For example, some data feeds may produce more data records per unit time, meaning that updates will take longer to load. At any point in time, an idle processor may decide which table it wants to update, provided that at least one update request for this table is pending. At time τ, table i may be picked, and the most recently loaded update may arrive at time r_ij. A processor would need to non-preemptively perform all the update requests for table i that have arrived between time r_ij+1 and τ, and there could be one or more pending requests. These pending update requests may be referred to as a “batch” with its length defined as the sum of the lengths of the pending update requests. In practice, batching may be more efficient than separate execution of each pending update. When the entire batch has been loaded, the processor can choose the next table to update.

At any time τ, the staleness of table i is defined τ−r, where r is the arrival time of the most recent update request rij that has been loaded. FIG. 4(a) illustrates the behavior of the staleness function of table i over time. The total staleness for this table is simply the area under the staleness curve. Suppose that table i is initialized at time ri₀=0. Let rsij and rf_ij denote the times that update r_ij starts and finishes processing, respectively. As can be seen, staleness accrues linearly until the first update is loaded at time rf_i1. At this time, staleness does not drop to zero; instead, it drops to rf_i1−r_i1. In contrast, FIG. 4(b) shows the staleness of table i assuming that the first two updates were batched. In this case, rs_i2=rs_i1 and rf_i2=rf_i1; conceptually, both update requests start and finish execution at the same times. Observe that staleness accrues linearly until the entire batch has been loaded. Clearly, total staleness is higher in FIG. 4(b) because the first update has been delayed.

The flow time of a task can be defined as the difference between its completion time and release time, and the stretch of a task is the ratio of its processing time to the flow time. Stretch measures the delay of the task relative to its processing time. These definitions may be slightly modified for various update tasks disclosed herein. For example, the flow time of an update request arriving at time r_ij may be redefined as the “standard” flow time plus the length of the update, i.e., rs_ij−r_i,j−1. Furthermore, the stretch of said update request may be redefined as the “standard” stretch plus the length of the update, i.e.:

$\frac{\left({\mathrm{rf}}_{\mathrm{ij}}-{\mathrm{rs}}_{\mathrm{ij}}\right)+\left(r_{\mathrm{ij}}-r_{i,j-1}\right)}{{\mathrm{rf}}_{\mathrm{ij}}-r_{i,j-1}}.$

Given the above definitions, the total staleness of table i in some time interval is upper-bounded by the sum of squares of the flow times (using the modified definition of flow time) of update requests in that interval. There are no known competitive algorithms for minimizing the L2 norm of “standard” flow time; however, any non-preemptive algorithm that is not voluntarily idle may nearly achieve the optimistic lower bound on total staleness.

There are competitive algorithms for minimizing the L₂ norm of flow times of all the update requests, using the modified definition of flow times defined previously. An algorithm is so-called “opportunistic” if it leaves no processor idle while a performable batch exists.

For any fixed β and δ such that 0<β,δ<1, C_β,δ=√δ(1−β)/√6 may be defined. Given a number p of processors and t of tables, α:=(p/t) min{C_β,δ,¼} may be defined. Then, provided that each α_i<α, the competitive ratio of any opportunistic algorithm is at most

${\left(1+\alpha \right)}^2\frac{1}{{\beta }^4}\frac{1}{1-\delta }.$

Any choice of constant parameters β and δ results in a constant competitive ratio. Note that as β→1 and δ→0, α approaches 0 and hence the competitive ratio approaches 1.

The penalty (i.e., sum of squares of flow times) of a given algorithm may be compared to a simple lower bound. Let A be the set of all updates. Independent of their batching and scheduling, each update i of length a_i needs to pay a minimum penalty of a_i². This follows from the convexity of the square function. If a set of updates are batched together, it may be necessary to pay no less than the sum of the squares of the updates' lengths. Therefore,

$\begin{array}{cc}\mathrm{LOW}:=\sum _{i\in A}{a}_i^2\le \mathrm{OPT}.& \left(1\right)\end{array}$

Looking at the cost a particular solution is paying, let B be the set of batches in the solution. For a batch B_iεB, let J_i be the first update in the batch, having length c_i. This batch is not applied until, for example, d_i time units have passed since the release of J_i. The interval of size d_i starting from the release of update J_i is called the “delay interval” of the batch B_i, and d_i is called the “delay” for this batch. As for the length of the batch, denoted by b_i, the following applies:c_i≦b_i≦c_i+d_i.  (2)

For the penalty of this batch, denoted by ρ_i, the following applies:

$\begin{array}{cc}{\rho }_i={\left[\left(c_i+d_i\right)+\alpha b_i\right]}^2,\mathrm{by}\mathrm{the}\mathrm{definition}\mathrm{of}\mathrm{penalty},& \left(3\right)\\ \le {\left(1+\alpha \right)}^2{\left(c_i+d_i\right)}^2\mathrm{by}\left(2\right).& \left(4\right)\end{array}$

Considering the case of one processor and two tables, if the updates of one table receive a large delay, it may indicate that the processor was busy applying updates from the other table (because it may be desirable for an applied algorithm to avoid remaining idle if it can perform something). Therefore, these jobs which are blocking the updates from the first table can pay (using their own sum-of-squares budget originally coming from the lower bound on OPT) for the penalty of a disclosed solution. It may be problematic if updates from the other tables (which are responsible for the payment) might be very small pieces whose sum of squares is not large enough to make up for the delay experienced (consider that the sum of their unsquared values is comparable to the delay amount). There may only be two batches, one of which is preferably large, occurring while a job of table 1 is being delayed. Another caveat is that the budget—the lower bound (LOW)—is Σ_iεAa_i², rather than Σ_iεBb_i², which may be much larger. If each of these batches is not much larger than its first piece (i.e., b_i=Θ(c_i)), then by losing a constant factor, the length of the batch can be ignored, and job sizes may be adjusted. Otherwise, this batch has a large delay and some other batches should be responsible for this large delay. It may be preferable to have those other batches pay for the current batch's delay. This indirection might have more than one level, but may not be circular.

Each job iεA has a budget of a_i² units. A batch B_iεB by (4), may need to secure a budget which is proportional to (c_i+d_i)². These conditions may be relaxed slightly in the following: A so-called “charging scheme” specifies what fraction of its budget each job pays to a certain batch. Let a batch B_i be “tardy” if c_i<β(c_i+d_i) (where β comes from the statement above); otherwise it is “punctual.” Let us denote these sets by B_t and B_p respectively. More formally, a charging scheme is a matrix (v_ij) of nonnegative values, where v_ij shows the extent of dependence of batch i on the budget available to batch j, with the following two properties.

1. For any batch B_iεB,

$\begin{array}{cc}{\left(c_i+d_i\right)}^2\le \sum _{j\in B_p}{v}_{\mathrm{ij}}{b}_j^2,\mathrm{and}& \left(5\right)\end{array}$

2. There exists a constant λ>0 such that, for any punctual batch B_j,

$\begin{array}{cc}\sum _{i\in B}{v}_{\mathrm{ij}}\le \lambda .& \left(6\right)\end{array}$

The existence of a charging scheme with parameters β and λ gives a competitive ratio of at most

${\left(1+\alpha \right)}^2\frac{1}{{\beta }^2}\lambda$

for an opportunistic algorithm.

Let:

$\begin{array}{cc}{\rho }_i\le {\left(1+\alpha \right)}^2{\left(c_i+d_i\right)}^2\mathrm{by}\left(4\right);& \left(7\right)\\ \le {\left(1+\alpha \right)}^2\sum _{j\in B_p}{v}_{\mathrm{ij}}{b}_j^2\mathrm{by}\left(5\right);& \left(8\right)\\ \le {\left(1+\alpha \right)}^2\sum _{j\in B_p}{v}_{\mathrm{ij}}\frac{1}{{\beta }^2}{c}_j^2\mathrm{by}\left(2\right)\mathrm{and}\mathrm{definition}\mathrm{of}\mathrm{punctuality}.& \left(9\right)\end{array}$

Hence, the total cost of a solution is

$\begin{array}{cc}\sum _{i\in B}{\rho }_i\le {\left(1+\alpha \right)}^2\frac{1}{{\beta }^2}\sum _{j\in B_p}{c}_j^2\sum _{i\in B}{v}_{\mathrm{ij}}\mathrm{by}\left(9\right)& \left(10\right)\\ \le {\left(1+\alpha \right)}^2\frac{1}{{\beta }^2}\lambda \sum _{j\in B_p}{c}_j^2\mathrm{by}\left(6\right)& \left(11\right)\\ \le {\left(1+\alpha \right)}^2\frac{1}{{\beta }^2}\lambda ·\mathrm{LOW}\mathrm{by}\left(1\right)& \left(12\right)\\ \le {\left(1+\alpha \right)}^2\frac{1}{{\beta }^2}\lambda ·\mathrm{OPT}\mathrm{by}\left(1\right).& \left(13\right)\end{array}$

The “execution interval of a batch B” may be defined as the time interval during which the batch B is being processed. Accordingly, its length is α_i times the length of B, if the update belongs to table i.

Batch B blocks batch B′ if B's execution interval has intersection of positive length with the delay interval of B′. Note that many batches can block a given batch B′. A charging scheme with the desired properties may be introduced. This is done by defining how v_ij values are computed. If a batch B_i is punctual, this may be relatively straightforward: all v_ij values are zero except for v_ii=1/β². Take a tardy batch B_i. In this case d_i is large compared to c_i. During the interval of length d_i, during which J_i is waiting (i.e., the delay interval of batch B_i), all p processors should be busy. Let [r, r′] denote this interval. The total time is pd_i. A relaxed version of this bound may be used to draw the conclusion (e.g., established by equation (6)).

A weighted directed graph with one node for each batch may be built. Punctual batches may be denoted as sinks (i.e., denoted as having no outarcs). Any tardy batch has arcs to all the batches blocking it, and there is at least one, since it has positive d_i. Even though punctual batches may be blocked by other batches, they have no outarcs.

The result is a so-called “directed acyclic graph” (DAG), because along any directed path in the graph, the starting (execution) times of batches are decreasing. The weight w_e on any such arc e is the fraction, between 0 and 1, of the blocking batch which is inside the delay interval [r, r′] of the blocked batch). Also, there is a parameter γ,

$\begin{array}{cc}\gamma :=\frac{6t{\alpha }^2}{p^2}{\left(\frac{1}{1-\beta }\right)}^2.& \left(14\right)\end{array}$

Then, for any two batches i and j, v_ij is defined as

$\begin{array}{cc}{v}_{\mathrm{ij}}=\frac{1}{{\beta }^2}\sum _{p\in P_{\mathrm{ij}}}\prod _{e\in p}\left(\gamma w_e^2\right),& \left(15\right)\end{array}$

where P_ij denotes the set of directed paths from i to j. The dependence along any path is the square of the product of weights on the path multiplied by γ to the power of the length of the path. This definition includes as a special case the definition of the v_ij's for punctual batches i, since there is a path of length zero between any batch i and itself (giving v_ii=1/β²) and no path from batch i to batch j for any j≠i (giving v_ij=0 if j≠i).

Such a charging scheme may satisfy the desired properties as shown: The cost paid for each batch should be accounted for using the budget it secures, as required above in (5).

For any batch B_iεB,

${\left(c_i+d_i\right)}^2\le \sum _{j\in B_p}{v}_{\mathrm{ij}}{b}_j^2,$

If B₁, . . . , B_k are the children of B₀, having weights w₁, . . . , w_k, respectively, in a run of any opportunistic algorithm, then

$\sum _{j^{\prime }=1}^k{\left(w_{0j^{\prime }}{b}_{j^{\prime }}\right)}^2\ge \frac{p^2{d}_0^2}{6t{\alpha }^2}.$

By the definition of the w_e's, the construction of the graph, the fact that B₁, B₂, . . . , B_k are all the batches blocking B₀, and the fact that the k blocking batches are run on p processors in a delay interval of length d₀ (so that their actual lengths must sum to at least 1/α times as much), it can be expressed that:

$\begin{array}{cc}\sum _{j^{\prime }=1}^k{\left(w_{0_{j^{\prime }}}{b}_{j^{\prime }}\right)}^2={\mathrm{pd}}_0·1/\alpha .& \left(16\right)\end{array}$

All but 3t of the batches may be removed, such that the sum of sizes of the remaining batches is at least ¾ times as large. Let [r, r′] be the delay interval corresponding to batch B₀. There may be one batch per processor whose process starts before r and does not finish until after r. At most p batches may be kept, and in addition, the first at-most-two batches for each table that intersect with this interval may be kept. The contribution of the other batches, however many they might be, may be small. Consider the third (and higher) batches performed in this interval corresponding to a particular table. Their original tasks start no earlier than r and their release times do not exceed r′. The former is true, since otherwise, such pieces would be part of the first or second batch of this particular table; call them B₁ and B₂. However, suppose there exists an update J that starts before r and is not included in B₁ or B₂. As it is not included in B₁ it preferably would have been released after the start of B₁. The batch B₂ cannot include any update released before r. So if it does not contain J, it should be empty, which is a contradiction. Hence, the total length of these batches (third and later) is no more than d₀, as they only include jobs whose start and end times are inside the delay interval [r, r′]. Nowtd₀≦pd₀/(4α), by definition of α.  (17)

In conjunction with (16), the total (unsquared) length of the remaining at-most-3t batches is at least pd₀/α−(¼)pd₀/α=(¾)pd₀/α. Considering that generally:

$\begin{array}{cc}\sum _{i=1}^N{x}_i^2\ge \frac{{\left(\sum _{i=1}^N{x}_i\right)}^2}{N}& \left(18\right)\end{array}$

it may be inferred that the sum of squares of the at-most-3t leftover tasks is at least (¾pd₀α)²/(3t), which exceeds p²d₀²/(6tα²).

To show that each batch receives a sufficient budget, let the “depth” of a node be the maximum number of arcs on a path from that node to a node of outdegree 0. The punctual nodes are the only nodes of outdegree 0. Induction on the depth of nodes may be used to prove, for any node B_i of depth at most Δ, that

$\left(c_i+d_i\right)\le \sum _{j\in B_p}{v}_{\mathrm{ij}}{b}_j^2$

For sinks, i.e., nodes of outdegree 0, the claim is apparent, since

$\begin{array}{cc}\begin{array}{cc}{\left(c_i+d_i\right)}^2\le \frac{1}{{\beta }^2}{c}_i^2& \mathrm{by}\mathrm{definition}\mathrm{of}\mathrm{punctuality}\end{array}& \left(19\right)\\ \begin{array}{cc}\le \frac{1}{{\beta }^2}{b}_i^2& \mathrm{by}\left(2\right)\end{array}& \left(20\right)\\ \begin{array}{cc}=v_{\mathrm{ii}}{b}_i^2\mathrm{and}& \mathrm{by}\mathrm{definition}\mathrm{of}{v}_{\mathrm{ii}}\end{array}& \left(21\right)\\ \begin{array}{cc}\begin{array}{cc}=\sum _{j\in B_p}& v_{\mathrm{ij}}{b}_j^2\end{array}& \mathrm{because}{v}_{\mathrm{ij}}=0\mathrm{if}j\ne i.\end{array}& \left(22\right)\end{array}$

Take a tardy batch B₀ of depth Δ whose immediate children are B₁, . . . , B_k. For any child B_i of B₀, whose depth has to be less than Δ, there is the following:

$\begin{array}{cc}{b}_i^2\le {\left(c_i+d_i\right)}^2& \left(23\right)\\ \le \sum _{j\in B_p}{v}_{\mathrm{ij}}{b}_j^2& \left(24\right)\end{array}$

Now we prove the inductive assertion as follows.

$\begin{array}{cc}{\left(c_0+d_0\right)}^2\le {\left(\frac{1}{1-\beta }\right)}^2{d}_0^2\mathrm{by}\mathrm{the}\mathrm{definition}\mathrm{of}\mathrm{tardiness}& \left(25\right)\\ =\gamma \frac{p^2{d}_0^2}{6t{\alpha }^2}\mathrm{by}\mathrm{the}\mathrm{choice}\mathrm{of}\lambda & \left(26\right)\\ \le \gamma \sum _{j^{\prime }=1}^k{\left(w_{0j^{\prime }}{b}_j\right)}^2\mathrm{from}\mathrm{above};\mathrm{and}& \left(27\right)\\ \le \sum _{j^{\prime }=1}^k\gamma w_{0^{j^{\prime }}}^2\sum _{j\in B_p}{v}_{j^{\prime }j}{b}_j^2\mathrm{by}\left(23\right)\mathrm{and}\left(24\right)\mathrm{above}.& \left(28\right)\end{array}$

Therefore:

$\begin{array}{cc}\le \sum _{j\in B_p}{v}_{0j}{b}_j^2& \left(29\right)\end{array}$ from (15) above, and because for jεB_p, the first arc of the paths can be factored out to get

$\begin{array}{cc}{v}_{\mathrm{oj}}=\sum _{j^{\prime }=1}^k\left(\gamma w_{0^{j^{\prime }}}^2\right)v_{j^{\prime }j};& \left(30\right)\end{array}$

More precisely, let P_e,j, for an arc e=(u, v) and a node j, be the set of all directed paths from u to j whose second node is v. Then,

$\begin{array}{cc}{v}_{\mathrm{oj}}=\frac{1}{{\beta }^2}\sum _{p\in P_{0j}}\coprod _{e\in p}\left(\gamma w_e^2\right)\mathrm{by}\left(15\right)& \left(31\right)\\ =\sum _{j^{\prime }=1}^k\frac{1}{{\beta }^2}\sum _{p\in P_{j^{\prime }j}}\coprod _{e\in p}\left(\gamma w_e^2\right)\mathrm{since}{P}_{\mathrm{oj}}=\stackrel{k}{\bigcup _{j^{\prime }=1}}{P}_{{\mathrm{ej}}^{\prime },j}\mathrm{and}{P}_{e,j}\bigcap P_{e^{\prime },j}=\varnothing \mathrm{for}e\ne e^{\prime }& \left(32\right)\\ =\sum _{j^{\prime }=1}^k\gamma w_{e_{j^{\prime }}}^2\frac{1}{{\beta }^2}\sum _{p\in P_{j^{\prime }j}}\coprod _{e\in p}\left(\gamma w_e^2\right)\mathrm{and}& \left(33\right)\\ =\sum _{j^{\prime }=1}^k\left(\gamma w_{0j^{\prime }}^2\right)v_{j^{\prime }j}\mathrm{by}\left(15\right).& \left(34\right)\end{array}$

The second property of a charging scheme says that the budget available to a batch should not be overused.

For any batch B_j,

$\sum _{i\in B}{v}_{\mathrm{ij}}\le \lambda :=\frac{1}{{\beta }^2\left(1-t\gamma \right)}$

and tγ<1.

The delay intervals corresponding to batches of a single table may be disjoint, as shown: The delay interval of a batch B_iεB starts at the end of J_i. Suppose this interval intersects one of B_j,j≠i, from the same table. Without loss of generality, assume that J_j starts at least as late as J_i. Thus, as J_i and J_j intersect, J_j should have been released before the delay interval of B_i ends. This is in contradiction with the definition of batching, as it implies J_j should be included in batch B_i.

To demonstrate the second property of the charging scheme, let the height of a node be the maximum number of arcs on a path from any node to that node. Induction on the height of nodes can be used to demonstrate for any node B_j of height H,

$\sum _{i\in B}{v}_{\mathrm{ij}}\le \lambda$ It may be noted that:

$\begin{array}{cc}t\gamma =\frac{6t^2{\alpha }^2}{p^2}{\left(\frac{1}{1-\beta }\right)}^2\mathrm{by}\left(14\right)\mathrm{above},\mathrm{and}& \left(35\right)\\ \le \delta <1\mathrm{by}\mathrm{definition}\mathrm{of}\alpha ,\delta \mathrm{above}.& \left(36\right)\end{array}$

For a batch B_j at height zero (a source, i.e., a node of indegree 0), the definition of v_ij, which involves a sum over all i→j paths, would be 0 unless i=j, in which case v_ij=1/β². Now the claim that λ≧1/β² follows from the definition of λ and the fact that tγ<1.

As above, the last arc of the path can be factored out, except for the zero-length trivial path. Consider B₀, whose immediate ancestors are B₁, . . . , B_k with arcs e_i=(B₁, B₀), . . . , e_k=(B_k, B₀), respectively. These incoming arcs may come from batches corresponding to different tables. However, it may be shown that the sum Σ_i=1^kw_ei of the weights of these arcs is at most t. More precisely, it may be shown that the contribution from any table is no more than one. Consider that w_ei denotes the fraction of batch B₀ which is in the delay interval of batch B_i. As the delay intervals of these batches are disjoint, as above in some cases, their total weight cannot be more than one and hence the total sum over all tables cannot exceed t.

Further, for any e, it may be shown that w_e<1. So

$\begin{array}{cc}\sum _{i=1}^k{w}_{\mathrm{ei}}^2\le t.& \left(37\right)\end{array}$

As the height of any ancestor B_i of B₀ is strictly less than H, the inductive hypothesis ensures that the total load Σ_jεBvji on B_i is no more than λ. The total load on B₀ is:

$\begin{array}{cc}\sum _{i\in B}{v}_{i0}=\frac{1}{{\beta }^2}+\sum _{i\in B,i\ne 0}\sum _{i^{\prime }=1}^k\gamma w_{i^{\prime }0}^2{v}_{{\mathrm{ii}}^{\prime }}& \left(38\right)\end{array}$

by definition of v_ij in (15) above, noting that v₀₀=1/β² and the fact that for any i≠0, any path from B_i to B₀ visits another batch B_i′ which is an immediate ancestor of B₀,

$\begin{array}{cc}=\frac{1}{{\beta }^2}+\gamma \sum _{i^{\prime }=1}^k{w}_{i^{\prime }0}^2\left(\sum _{i\in B}{v}_{{\mathrm{ii}}^{\prime }}\right),\mathrm{by}\left(38\right)& \left(39\right)\end{array}$ Now

$\sum _{i\in B}{v}_{{\mathrm{ii}}^{\prime }}\le \lambda$ by the inductive hypothesis applied to i′ and (39)

$\sum _{i^{\prime }=1}^k{w}_{i^{\prime },0}^2\le t$ by (37), then

$\begin{array}{cc}\sum _{i\in B}{v}_{i0}\le \frac{1}{{\beta }^2}+\gamma t\lambda \mathrm{by}\left(37\right)\mathrm{and}\mathrm{the}\mathrm{inductive}\mathrm{hypothesis}& \left(40\right)\\ =\lambda ,\mathrm{by}\mathrm{the}\mathrm{choice}\mathrm{of}\lambda ,& \left(41\right)\end{array}$ as desired. The matrix (v_ij) is a charging scheme, as shown above.

Above the staleness measure is twice between the penalty measure and the lower bound LOW. Since the main theorem shows that these two outer values are close, staleness should also be close to the lower bound.

It can be argued that LOW is also a lower bound on staleness. Staleness is an integration on how out-of-date each table is. Tables can be considered separately. For each particular table, one can look at portions corresponding to different updates. If an update starts at time r and ends (i.e., is released) at r′, then at point r≦x≦r′, the staleness is no less than x. Thus, the total integration is at least ½Σ_iεAa_i². Staleness in most or all cases cannot be larger than half the penalty paid. For each specific table, the time frame is partitioned into intervals, marked by the times when a batch's performance is finished. The integration diagram for each of these updates consists of a trapezoid. It can be denoted by y the staleness value at time r. The staleness at r′ is then y+r′−r. Total staleness for this update is

$\begin{array}{cc}{\rho }^{*}=\left(r^{\prime }-r\right)\frac{y+y+r^{\prime }-r}{2}& \left(42\right)\\ \le \frac{{\left(r^{\prime }-r+y\right)}^2}{2},& \left(43\right)\end{array}$ as y≧0 and AB≦(A+B/2)² for A,B≧0,=ρ,  (44)

where ρ is the penalty for this batch according to our objective.

There is no known online algorithm which is competitive with respect to stretch. With regard to this, suppose there is one processor and two tables. A large update of size S1 arrives on the first table. At some point, it needs to be applied. As soon as this is done, a very small update of size S₂ appears on the second table. Since preemption is not allowed, the small update needs to wait for the large update to finish. The stretch would be at least αS₁/S₂. But if there was advanced knowledge of this, the larger job could be delayed until completion of the smaller update.

Even if there is an offline periodic input, the situation might be less than optimal. In some cases, stretch can be large. Again, if there are two tables and one processor, one table may have a big periodic update of size S₁. The other table may have small periodic updates of size S₂. At some point, an update from table one should be performed. Some updates from table two may arrive during this time and need to wait for the large update to finish. So their stretch is at least a(S₁−S₂)/S₂.

The above examples all work with one processor, but similar constructions show that with p processors, the stretch can be as large as desired because it is not bounded. To do this, p+1 tables and p processors are needed. The i^th table has a period which is much larger than the (i+1)^th one. The argument above was a special case for p=1.

The identity of a condition that allows stretch to be bounded is sought. Suppose each table has updates of about the same length (i.e., it is semi-periodic). In other words, the updates from each table have size in [A, cA] for some constant c. Any constant c would work (yet give a different bound finally), but c=2 is picked for ease of exposition. Further assume that tables can be divided into a few (g, to be precise) groups, such that periods of updates in each group is about the same thing (the same condition for the update lengths being in [A, 2A]). Then at least as many processors are needed as compared to the number of groups. Otherwise, there can be examples to produce arbitrarily large stretch values. Additionally, a reasonable assumption can be made that p is much larger than g. Each group is assigned some processors, in an amount proportional to their load. That is, the number of processors given to each group is proportional to the number of tables in the group. The algorithm is given in FIG. 5. After the assignment of processors to groups, each group runs a specific opportunistic algorithm. This algorithm, at each point when a processor becomes idle, picks the batch corresponding to the oldest update.

At that point, each group forms an independent instance. Let us assume that for the t′ tables in a specific group with p′ processors, the upper bound is α≦p′/8t′ on each α_i.

It can be shown that if all the updates of a group have sizes between A and 2A, and α≦p′/8t′, stretch is bounded by 3. Taking any one task, it can be shown it cannot wait for long, and thus its stretch is small. Note that stretch also considers the effect of batching this task with some other tasks of the same table. To this end, the execution of tasks is divided into several sections. Each section is either tight or loose. A tight section is a maximal time interval in which all the processors are busy. Loose is defined in the example as not tight.

Jobs can be ordered according to their release times, and ties may be arbitrarily decided. Let ω_i denote the wait time (from release time to start of processing) of the i^th job (say, J_i). Let θ_k be the length of the k^th tight section (call it S_k). Recursive bounds can be established on values ω_i and θ_k, and then induction may be used to prove they cannot be too large. There is inter-relationship between them, but the dependence is not circular. Generally, θ_k depends on ω_i for jobs which are released before Sk starts. On the other hand, ω_i depends on ω_i′ for i′<i and θ_k for S_k in which J_i is released.

Let i_k be the index of the last job released before S_k starts. The topological order of recursive dependence is then as follows: the ω_i's are sorted according to i, and θ_k is placed between ω_ik and ω_1+ik. The recursive formulas developed below relate the value of each variable to those to its left, and hence, circular dependence is avoided.

To derive a bound on θ_k, one can look more closely at the batches processed inside S_k. Let r and r′ be the start and end time of the section S_k. These batches correspond to updates which are released before r′. Let the so-called “load” at time r be the total amount of updates (released or not) until time r that has not yet been processed. Part of a batch that is released might have been processed (although its effect would not have appeared in the system yet). After half of processing time is passed, it can be considered that half of the batch has been processed. Updates which have not been released, but have a start time before r, may be considered to be part of the load (not all of it, but only the portion before r). The contribution to load by any single table is at most 2A+max_i≦ikω_i. There are at least three cases to consider. First, if no batch of the table is completely available at time r, the contribution X≦2A; that is, there can only be one update which has not yet been released. Second, if a batch is waiting until time r, then X≦2A+max_i≦ikω_i, since the length of the batch is the actual contribution. However, if a batch is running at time r, let z be the time at which its processing started. In most or all cases, z≦r and the processing of the batch continues up to at least time r. The load is

$\begin{array}{cc}X\le \left(r-z\right)+\left(2A+\underset{i\le i_k}{\max }{\omega }_i\right)-\left(r-z\right)/\alpha & \left(45\right)\end{array}$ where the first term corresponds to a (possibly not yet released) batch which is being formed while the other batch is running, the second term bounds the length of the running batch and the last term takes out the amount of load that is processed during the period from z to r. Noting that α≦1, equation (45) gives X≦2A+max_i≦ikω_i.

Hence, the total load at time r is at most t(2A+max_i≦ikω_i). Yet, there can be an additional load of tθ_k which corresponds to the updates inside S_k. Thus, all the batches to be processed during S_k can be processed to get:

$\begin{array}{cc}2At+t\underset{i\le i_k}{\max }{\omega }_i+t{\theta }_k\ge {\theta }_{k}p/\alpha .& \left(46\right)\end{array}$

Rearranging yields:

$\begin{array}{cc}{\theta }_k\le \frac{\left(2A+{\max }_{i\le i_k}+{\omega }_i\right)t}{\frac{p}{\alpha }-t},& \left(47\right)\end{array}$

Inequalities for ω_i may be considered. Without loss of generality, consideration is made of the waiting time for the first update of a batch. It may be noted that they have the largest wait time among all the updates from the same batch. If a task has to wait, it should have one of two reasons. Either all the processors are busy; or another batch corresponding to this table is currently running Consider two cases:

The first case is when J_i is released in the loose section before S_k. If ω_i>0, it is waiting for another batch from the same table. The length of the batch is at most max_i′<iω_i′+2A. If as soon as this batch is processed at time τ, J_i processing may begin. Any job with higher priority than J_i should have been released before it is. When J_i is released, all these other higher-priority jobs are either running, or waiting for one batch of their own table. So, their count cannot be more than p−2 (there is one processor working on the table corresponding to J_i and one for each of these higher-priority jobs, and at least one processor is idle). In other words, there are at most p−2 tables which might have higher priority than J_i's table at time τ. Thus, J_i cannot be further blocked at time τ. Hence,

$\begin{array}{cc}{\omega }_i\le \alpha \left(\underset{i^{\prime }<i}{\max }{\omega }_{i^{\prime }}+2A\right).& \left(48\right)\end{array}$

The other case is when J_i is released inside the tight section S_k. If J_i is processed after ω>θ_k time units pass since the start of S_k, a batch from the same table has to be under processing at the moment S_k finishes; otherwise, J_i would start at that point. However, processing of this batch must have started before J_i was released; or else, J_i has to be part of it. Moreover, similarly to the argument for the first case, it can be shown that as soon as the processing of the blocking batch from the same table is done, the batch corresponding to job J_i will start to be processed. More precisely, there can be at most p−1 other batches with higher priorities than J_i. So they cannot block J_i at time τ when the lock on its table is released. Hence,

$\begin{array}{cc}{\omega }_i\le \max \left\{{\theta }_k,\alpha \left(\underset{i^{\prime }<i}{\max }{\omega }_{i^{\prime }}+2A\right)\right\},& \left(49\right)\end{array}$ since, it either waits for the tight section to end, or for a batch of its own table whose length cannot be more than max_i′<iω_i′+2A.

One can pick of ω*=θ*=A/3 and use induction to show that ∀i:ω_i≦ω* and ∀k:θ_k≦θ*, in part because α≦⅛ and p/α≧8t. Note that the right-hand side of Equations (47), (49) and (48) would be no more than θ*=ω* if one replaces these values for the w_i and θ_k values in the formula. It only remains to observe that the dependence is indeed acyclic, which is clear by the ordering and by the fact that each formula uses the values to its left in the ordering.

The length of a batch is at most A′=A/3≦7/3A, where the first term comes from length of the first job in the batch (A≦A′≦2A), and the second term comes from the bound on its wait time. The resulting maximum stretch for any update piece would be bounded by 7/3(1+α)<3.

The algorithm preferably keeps the stretch low. The algorithm in FIG. 5 can keep the stretch below 3 if

$\alpha \le \frac{p-g}{8t}.$ If the sum of (p−g)|T_i|/t for different groups is p−g, then Σ_i[(p−g)|T_i|/t≦p. Thus, one would use, at most, as many processors as were available. Then, in each group p′≧(p−g)t′/t. So the following applies: α≦p′/8t′. The arguments above, regarding if all of the updates of a group have sizes between A and 2A, apply to show that stretch is bounded by 3.

So-called “weighted staleness” can also be considered. That is, each table has a weight w_i that should be multiplied by the overall staleness of that table. This takes into account the priority of different tables. In an online setting, no known algorithm can be competitive with respect to weighted staleness. As soon as a job from the low priority table is scheduled, a job appears from the high priority table which will then cost too much.

In the semi-periodic instance, weighted staleness of the algorithm in FIG. 5 is no more than nine times that of OPT. If w_i is a weights for staleness definition, Σ_iεAw_ia_i² is a lower bound on the weighted staleness. As stretch is less than 3, the weighted staleness cannot be larger than 3²=9 times that of the optimum.

To the maximum extent allowed by law, the scope of the present disclosure is to be determined by the broadest permissible interpretation of the following claims and their equivalents, and shall not be restricted or limited to the specific embodiments described in the foregoing detailed description.

Claims

1. A method of updating data tables stored in a data warehouse, the method comprising: storing, in memory, a plurality of data tables;detecting, by a processor, incoming data for updating the plurality of data tables;generating, by the processor, an update request associated with each data table in the plurality of data tables;determining a calculated staleness for a portion of the plurality of data tables;scheduling updates to the portion of the plurality of data tables based on the calculated staleness;determining a stretch value for each one of the data tables in the portion of the plurality of data tables, the stretch value indicative of a maximum ratio between a duration of time a corresponding one of the updates waits until processing is finished and a length of the corresponding one of the updates, wherein scheduling data table updates is based at least in part on the stretch value; anddistributing the updates among a plurality of processors to minimize the calculated staleness;transforming the portion of the plurality of data tables to include a different portion of the incoming data based on the scheduling.

2. The method of claim 1, further comprising determining a previous update of the portion of the plurality of data tables.

3. The method of claim 1, further comprising determining the update request is non-preemptible.

4. The method of claim 1, further comprising batching an accumulation of update requests.

5. The method of claim 1, further comprising performing the updates in a real-time data.

6. The method of claim 5, further comprising responsive to receiving a user request for the portion of the plurality of data tables.

7. The method of claim 1, further comprising weighting a first portion of the plurality of data tables higher than a second portion of the plurality of data tables.

8. The method of claim 1, further comprising determining the calculated staleness is a priority weighted staleness.

9. The method of claim 8, further comprising multiplying a first data table staleness by a first weight and multiplying a second data table staleness by a second weight.

10. The method of claim 1, further comprising appending new data to one of the plurality of data tables.

11. The method of claim 1, further comprising performing the updates at variable intervals.

12. A non-transitory computer readable medium storing computer instructions that when executed cause a processor to perform a method for managing a plurality of data tables in a data warehouse, the method comprising: maintaining the plurality of data tables in the data warehouse;receiving requests to update, with incoming data, a portion of the plurality of data tables;generating update requests corresponding to the requests to update;determining calculated stalenesses for individual data tables of the portion of the plurality of data tables;ranking the calculated stalenesses;scheduling updates to the portion of the plurality of data tables based on the calculated stalenesses;determining a stretch value for each one of the individual data tables in the portion of the plurality of data tables, the stretch value indicative of a maximum ratio between a duration of time a corresponding one of the updates waits until processing is finished and a length of the corresponding one of the updates, wherein scheduling data table updates is based at least in part on the stretch value;distributing the updates among a plurality of processors to minimize the calculated stalenesses; andtransforming the portion of the plurality of data tables to include a different portion of the incoming data based on scheduling of the updates and the update requests.

13. The non-transitory computer readable medium of claim 12, further comprising weighting a first portion of the plurality of data tables higher than a second portion of the plurality of data tables, and wherein the scheduling is at least in part responsive to a result of the weighting.

14. The non-transitory computer readable medium of claim 12, further comprising appending the incoming data to the portion of the plurality of data tables.

15. A server for managing a data warehouse, the server comprising: a processor; anda memory storing instructions that when executed cause the processor to perform operations, the operations comprising:receiving incoming data for updating a plurality of data tables;an interface for receiving incoming data for updating the plurality of data tablesdetermining calculated stalenesses for a portion of the plurality of data tables;weighting a portion of the calculated stalenesses;scheduling updates to the portion of the plurality of data tables based on the calculated stalenesses;determining a stretch value for each one of the individual data tables in the portion of the plurality of data tables, the stretch value indicative of a maximum ratio between a duration of time a corresponding one of the updates waits until processing is finished and a length of the corresponding one of the updates, wherein scheduling data table updates is based at least in part on the stretch value; anddistributing the updates among a plurality of processors to minimize the calculated stalenesses.

16. The server of claim 9, wherein the operations further comprise batching the updates.