Patent Grant
7873689
The invention relates generally to information processing systems and, more particularly, to query processing in a distributed streaming environment.
In distributed data streaming applications, rapid update streams originating at tens or hundreds of remote sites are continuously transmitted to a central processing system for online querying and analysis. Examples include monitoring of service provider network traffic statistics, telecommunication call detail records, Web usage logs, financial stock tickers, retail chain transactions, weather data, sensor data, and so on.
An important consideration in the above-mentioned monitoring applications is the communication overhead imposed by the distributed query processing architecture on the underlying network. Specifically, transmitting every update stream to a central site for processing can lead to inordinate amounts of message traffic, and thus have a crippling effect on the communication infrastructure as well as the central site processor.
For many distributed stream-oriented applications, exact answers are not required and approximations with guarantees on the amount of error suffice. The tradeoff between answer accuracy and communication overhead for specific classes of continuous queries over distributed update streams has been studied recently.
One approach considers aggregation queries that compute sums and averages of dynamically changing numeric values spread over multiple sources. In this approach, each site is assigned an interval of a certain width such that the sum of site interval widths is less than the application's total error tolerance. Thus, as long as the numeric value at each site stays within the interval for the site, no messages need to be sent by the sites in order to satisfy the application's accuracy requirements. However, in case the value at a site drifts outside the site's interval, the site is required to transmit the value to the central site and make appropriate adjustments to its interval.
Another approach focuses on the problem of continually tracking top-k values in distributed data streams; the developed techniques ensure the continuing validity of the current top-k set (at the central site) by installing arithmetic constraints at each site.
Unfortunately, most existing approaches for processing data streams are primarily concerned with exploring space-accuracy tradeoffs (mostly for single streams) rather than communication-accuracy tradeoffs in a distributed streams setting.
The invention addressed various deficiencies in the prior art by providing methodologies pertaining to the problem of approximately answering set-expression cardinality queries over distributed streams originating at tens or hundreds of remote sites.
Various embodiments provide a method and system for answering set-expression cardinality queries while lowering data communication costs by utilizing a coordinator site to provide global knowledge of the distribution of certain frequently occurring stream elements to significantly reduce the transmission of element state information to the central site and, optionally, capturing the semantics of the input set expression in a Boolean logic formula and using models of the formula to determine whether an element state change at a remote site can affect the set expression result.
The inventive methodologies estimate set-expression cardinality with guaranteed accuracy at a central processing site, while keeping data communication costs between the remote sites and the central processor at a minimum. The inventive solutions exploit global knowledge of the distribution of frequent elements as well as the semantics of set expressions to reduce data transmission overhead while preserving user-specified error guarantees. The inventive methodologies and protocols efficiently propagate global frequency information across sites, and provide a logic-based formulation for identifying the element state changes (at a remote site) that can affect the set expression result (at the central site). The methodologies are effective in reducing the volume of message traffic.
Specifically, in one embodiment of the invention, a distributed framework for processing set-expression cardinality queries is provided wherein: each update stream site being allocated an error budget adapted to determine when the update stream site communicates stream state information to a central site; each update stream site associates a charge with every stream element that is inserted or deleted since stream state was previously transmitted to a central site; and each update stream site transmits current stream state information when the sum of its respective element charges exceeds its respective error budget.
The teachings of the present invention can be readily understood by considering the following detailed description in conjunction with the accompanying drawings, in which:
To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures.
The invention will be described within the context of distributed stream-oriented applications in which a relatively large number of remote sites provide update streams to a central site for processing.
The present invention is adapted to providing a distributed framework for processing set-expression cardinality queries. Each site or stream source is allocated an error budget which governs when the site communicates stream state information to the central processing site (e.g., for estimating set-expression cardinality). Each remote site associates a charge with every stream element that is inserted or deleted since stream state was last transmitted to the central site. Only when the sum of element charges at a site exceeds the site's error budget does the site communicate the current stream state information to the central site. The framework allows for flexibility in how elements are assigned charges as well as how site error budgets are allocated. Specifically, methods for computing charges are only required to satisfy certain basic properties needed for correctness in terms of providing the stipulated error guarantees. Methods that return smaller charges for elements are more desirable since they result in lower communication overhead.
In another embodiment of the invention, additional techniques that incorporate global knowledge to reduce communication are provided. In many distributed streaming environments, the frequency distribution of stream elements will be skewed with certain elements occurring more frequently than others. For example, in the flows collected from an ISP's border routers, IP addresses corresponding to popular Web sites like Yahoo, Google, Amazon and the like will be contained in a disproportionately large number of flows. If such a frequently occurring element (e.g., an IP address or other stream element) is inserted into a stream at a site (where it does not appear previously), there is no need to charge for it since the element must already be present at the central site and, therefore, the insert has no effect on the set-expression cardinality at the central site. Similarly, the charge for the deletion of a frequent element can be distributed across all the sites where the element occurs since the element would need to be deleted at all these sites to truly go away. Thus, global knowledge of frequent stream elements can lead to lower overall communication costs due to reduced element charges at each site. In one embodiment, protocols for disseminating this global information to the various sites while incurring minimal message overhead are provided.
In another embodiment of the invention, additional techniques that exploit set-expression semantics to reduce communication are provided. In this embodiment, the semantics of set expressions are exploited to obtain further reductions in element charges. For example, in the expression S∪T, if an element e is already present in stream S, then inserts and deletes of e from T have no effect on the set-expression result. Therefore, there is not need to charge for them. In this embodiment, a logic-based approach is provided where the conditions for a change in a set-expression result are captured in a Boolean formula. Models for the Boolean formula then represent scenarios for result changes and are used to compute element charges. Finally, in order to address the (provably required) large time complexity of model enumeration, the methodologies develop an efficient heuristic for computing element charges whose running time is polynomial in the number of streams.
It is noted that while the primary focus of the invention is estimating set-expression cardinality, the described techniques are quite powerful, and can also be used to approximate set expression results (i.e., sets of data elements) at a central site. This can be used by the coordinator to run other potentially complex queries on top of it, which could be more useful than just cardinality queries. For example, in the below DDoS scenario, the results could be filtered to identify malicious hosts, or in the below Akamai example, to identify users corresponding to certain traffic patterns.
It will be appreciated by those skilled in the art that the invention has broad applicability to distributed stream-oriented applications and query processing. For example, in wireless sensor networks (e.g., for environmental monitoring, inventory tracking, etc.), sensors tend to have a very limited battery life, and radio communication is much more expensive in terms of power consumption compared to processing. To ensure longer lifetimes for sensor nodes, it is critical to reduce the amount of data transmitted, even if that implies additional processing at the sensor nodes. Generally speaking, the invention is adapted to enable sufficiently accurate approximations to queries while avoiding excessive communication overhead within the distributed query processing architecture.
Another application is the problem of detecting distributed denial-of-service (DDoS) attacks by analyzing network flow information collected from an ISP's border routers. In a typical DDoS attack scenario, hundreds of compromised “zombie” hosts flood a specific victim destination with large numbers of seemingly legitimate packets. Furthermore, in order to elude source identification, attackers typically forge, or “spoof”, the IP source address of each packet they send with a randomly-chosen address. Consequently, one approach for detecting DDoS attacks is to look for sudden spikes in the number of distinct IP source addresses observed in the flows across the ISP's border routers. The DDoS monitoring application does not require IP source address counts to be tracked with complete precision. Approximate counts can be equally effective for the purpose of discerning DDoS activity as long as errors are small enough so as to not mask abrupt changes in the true counts. Thus, depending on the accuracy requirements of the DDoS application, routers only need to transmit a subset of flow records to the central monitoring site.
As another example, consider a Web content delivery service such as that provided by Akamai (www.akamai.com). In this case, Web sites are replicated at a large number of geographically distributed servers, and users requesting access to a Web site are automatically redirected to the geographically closest server, or the least loaded server. Here, one might often be interested in tracking (approximately) the number of (distinct) users accessing a Web site (across all servers), the number of users who visit both a Web site A and Web site B, or the number of users who visit Web site A but not B. These statistics can be useful for determining the servers at which to replicate Web sites, deciding which advertisements to display at each Web site, and so on.
The problem of counting the number of distinct IP source addresses or web-site users, as discussed above, are special cases of the more general set-expression cardinality estimation problem, which is discussed in more detail herein. In this more general problem, of interest is estimating the number of distinct values in the result of an arbitrary set expression over distributed data streams. For example, in the DDoS scenario, in one embodiment is employed the set difference cardinality query |S−T| to detect significant traffic deviations. Specifically, S is the IP source address set for a sliding window spanning the past week (until now) and T is the set of IP source addresses from the week prior to that (e.g., two weeks ago). Similarly, in the Web example, if S and T are the sets of users who visit Web sites A and B, respectively, then the set intersection query |S∩T| yields the number of users who access both sites A and B.
System Model
This section describes a distributed update-stream processing architecture and formally defines the set-expression cardinality estimation problem addressed in this paper.
The model 100 of
Each remote site exchanges messages only with the coordinator, providing it with state information for streams at the site. Note that this distributed communication model is representative of a large class of real-life applications including network monitoring where a central Network Operations Center (NOC) is responsible for processing network traffic statistics collected at the switches and routers distributed across the network.
Specifically,
The processor 420 cooperates with conventional support circuitry 430 such as power supplies, clock circuits, cache memory and the like as well as circuits that assist in executing the software routine stored in the memory 440. As such, it is contemplated that some of the steps discussed herein as software processes may be implemented within hardware, for example as circuitry that cooperates with the processor 420 to perform various steps. The processing architecture 400 also contains input/output (I/O) circuitry 410 which forms an interface between the various functional elements communicating with the architecture 400.
The architecture 400 may be advantageously employed within the context of a network management system (NMS), an element management system (EMS) or any other network management system. More specifically, the architecture 400 may be utilized within the context of a coordinator site (110) and/or a remote site (120) within the model 100 of
The invention may be implemented as a computer program product wherein computer instructions, when processed by a computer, adapt the operation of the computer such that the methods, data structures and/or techniques of the present invention are invoked or otherwise provided. Instructions for invoking the inventive methods may be stored in fixed or removable media, transmitted via a data stream in a broadcast media, and/or stored within a working memory within a computing device operating according to the instructions.
Memory 440 is depicted as including, in one embodiment, coordinator site processing routines such as a transmission control routine 442, a set expression cardinality algorithm 444 and other programs and data 446. Memory 440 is depicted as including, in one embodiment, remote site processing routines 448. The various routines and related methodologies will be discussed below.
It will be appreciated by those skilled in the art and informed by the teachings of the present invention that while the memory 440 includes a plurality of data structures, algorithms and storage regions, there is no requirement within the context of the present invention that a single memory device as depicted be utilized within the context of the update-stream processing architecture. Specifically, any combination of internal, external and/or associated memory may be utilized to store the software instructions necessary to provide the various functions. Thus, while the architecture depicts the memory as an integral portion of a relatively unified structure, the memory 440 may in fact be distributed internal or external to the update-stream processing architecture 400.
Referring back to FIG. 1., at each remote site j, the n update streams render n distinct multi-sets S0,j, . . . , Sn-1,j of elements from the integer domain [M]={0, . . . , M−1}. It is noted that other non-integer domains may also be utilized within the context of the present invention. Generally speaking, any finite domain may be used.
Each stream update at remote site j is a triple of the form <i,e,±ν> where i identifies the multi-set Si,j being updated, e∈[M] is the specific data element whose frequency changes, and ±ν is the net change in the frequency of e in Si,j, i.e., “+ν” (“−ν”) denotes ν insertions (resp., deletions) of e. It is assumed that all deletions in the update streams are legal; that is, an update <i,e,−ν> can only be issued if the net frequency of e in Si,j is at least ν. Note that delete operations help to substantially enrich the streaming model; for example, with deletions, the methodologies can easily handle sliding window queries by simply issuing a delete operation for each expired stream update that is no longer in the window of interest. For each i=0, . . . , n−1, let Si=∪jSi,j. Thus, Si reflects the global state of the ith update stream, while each multi-set Si,j captures the local state of stream i at site j. For purposes of simplicity, the terms Si and Si,j will be referred to as streams, though actual meaning of these terms is more precisely given as the current states of the underlying streams.
The invention is adapted to the problem of answering set-expression cardinality queries over the underlying collection of distributed update streams. Specifically, given a set expression E over streams S0, . . . , Sn-1 (with the standard set operators ∪, ∩ and − as connectives), the methodologies seek to estimate |E|, the number of distinct elements in E. For example, |S0∩S1| is the number of distinct elements in the intersection of streams S0 and S1. If for m=2 remote sites, S0,1={a}, S0,2={a,b}, S1,1={b} and S1,2={c}, then S0={a,b} and S1={b,c}. Thus, E=S0∩S1={b} and |E|=1.
The problem of estimating |E| at the coordinator is complicated because the substreams Si,j that comprise each distributed stream Si are distributed across the remote sites. Accurately tracking |E| by having remote sites continuously ship every stream update to the coordinator is clearly impractical for high data rate streams. Consequently, in order to reduce the burden on the communication infrastructure, the methodologies allow |E| to be approximated, but enforce a bound on the error in the final estimate. Specifically, for a pre-specified error tolerance ε, the methodologies seek to compute an estimate {circumflex over (X)} for X=|E| (at the coordinator) such that X−ε≦{circumflex over (X)}≦X+ε. The ε error parameter provides system designers with a useful knob that enables them to trade accuracy for efficiency. Essentially, the larger the error tolerance of an application, the smaller the communication overhead required to ensure that the estimate {circumflex over (X)} meets the ε accuracy guarantee.
1. Estimating Single Stream Cardinality.
A distributed algorithm for the case when the expression E whose cardinality the methodologies wish to estimate is a single stream Si (which is the union of substreams Si,j at remote sites) will first be described. Thus, the methodologies are basically looking to estimate the number of distinct elements in stream Si. The scheme for the distinct elements estimation problem illustrates the key concepts underlying the approach as well as the overall structure of the distributed solutions. In the next section, the methodologies will generalize the solution for a single stream to handle arbitrary set expressions.
The objective of the distributed algorithm is to be able to continuously estimate |E| at the coordinator site with ε accuracy. To achieve this, the methodologies distribute the error tolerance of ε among the m remote sites. The methodologies denote the error budget allocated to site j by εj; thus, Σjεj=ε. While there are multiple ways to allocate error budgets to sites, a simple approach is to allocate these proportional to the stream update rates at the sites. Another approach is to distribute the error budget uniformly across the remote sites. In an alternate embodiment, any scheme may be used as long as the following invariant is satisfied: Σjεj=ε and for all j, εj≧0.
The error parameter εj essentially dictates when site j sends the current states of substreams Si,j at site j to the coordinator. The methodologies denote by Ŝi,j, the most recent state of substream Si,j communicated (by site j) to the coordinator. In addition to Si,j, site j also stores in its local memory, the transmitted states Ŝi,j for substreams at the site. For each stream Si, the coordinator constructs the global state Ŝi by taking the union of all the local substream states Ŝi,j received from the remote sites. Thus, Ŝi=ÅjŜi,j. Now let Ê be the result of evaluating expression E on the states Ŝi instead of Si. The coordinator estimates the cardinality of set expression E as |Ê|.
Generally speaking, if remote sites have limited memory, then the methodologies are optionally modified to store a compact sketch synopsis for each substream (instead of the complete substream state). Techniques associated with the compact sketch synopsis are described in more detail in U.S. patent application entitled METHOD FOR DISTINCT COUNT ESTIMATION OVER JOINS OF CONTINUOUS UPDATE STREAMS, Ser. No. 10/957,185 which is incorporated herein by reference in its entirety. In this embodiment each remote site keeps track of its respective substream states. Briefly, the JD Sketch techniques are adapted to maintaining a summary of a first continuous stream of tuple updates by hashing tuples received from the first continuous data stream according to at least one initial attribute, and for each bucket of the at least one initial attribute, generating a corresponding set of 2-level hash sketches according to at least one other attribute. The JD Sketch synopsis data structure is not a standard hash table. It may be conceptualized as a hash table having buckets which additionally include second level hash structures adapted to capture information pertaining to the set of elements of a stream. That is, only a summary of the set of information mapping to each hash table bucket is actually kept in the bucket. This summary hash table bucket summary may comprise, illustratively, a set of 2-level hash sketch synopses. Thus, each hash table bucket stores a corresponding set of hash tables of those data elements that map or “collide” into the hash table bucket.
Other synopsis structures may be used to maintain a summary of update stream data prior to transmission of that data from a remote site, such as 2-level or multiple level hash structures.
Generally speaking, any means of maintaining a summary or reduced size representation of the underlying update stream may be advantageously employed within the context of the present invention. In one embodiment, a delete resistant sampling scheme (such as the JD Sketch scheme) is used. A delete resistant sampling scheme comprises any sampling scheme which takes as an input a stream of inserts and deletes of elements and outputs a distinct sample (of a given size) of the stream which is identical (with high probability) to the sample that would be obtained if the deleted elements were never inserted to begin with.
In order to guarantee that the estimate |Ê| is correct, the methodologies ensure that |E|−ε≦|Ê|.≦|E|+ε. A simple approach for ensuring this for E=Si is as follows. At each remote site j, if either of |Si,j−Ŝi,j| or |Ŝi,j−Si,j| exceeds εj, then site j sends the most recent state Si,j to the coordinator. One can easily show that this simple scheme guarantees that at all times, |E−Ê|≦ε and |Ê−E|≦ε, and is thus correct. For instance, consider an element e in E−Ê. The element must belong to Si,j−Ŝi,j at some site j, and since |Si,j−Ŝi,j|≦εj, it must be counted against the error budget εj at site j. As a result, since Σjεj=ε, the methodologies get that |E−Ê|≦ε. Further, since |E|−|Ê|≦|E−Ê|, the methodologies obtain that |E|−|Ê|≦ε. Similarly, it is possible to show that |Ê|−|E|≦ε, and thus the estimate |Ê| is within ε error of |E|.
The above scheme associates a charge φj(e) with each element e at every remote site j, and if the total of these charges exceed ej, then the remote site communicates state information to the coordinator. More formally, let φj+(e)=1 if e∈(Si,j−Ŝi,j), φj−(e)=1 if e∈(Ŝi,j−Si,j), and φj+(e)=φj−(e)=0, otherwise. As a result, Σeφj+(e)=|Si,j−Ŝi,j| and Σeφj−(e)=|Ŝi,j−Si,j|. Thus, there is a message exchange between site j and the coordinator if either Σeφj+(e)>εj or Σeφj−(e)>εj.
In the above scheme, element charges are computed based entirely on the local state information available at each site. The methodologies next show that by exploiting global knowledge about element e, the methodologies can reduce the charge φj(e) for e, and as a consequence, the overall message traffic between remote sites and the coordinator.
A key observation made by the present inventors is that in many stream-oriented domains, there will be a certain subset of globally “popular” elements. For instance, in an IP network monitoring scenario, destination IP addresses corresponding to popular Web sites like Yahoo, Amazon, Google etc. will frequently appear in the flow records collected from network routers. An important characteristic of each such globally popular element is that, at any given point in time, it will appear in substreams at multiple sites although the exact sites that contain the element may vary over time.
Assuming, for a popular element e, each remote site (approximately) knows the number of substream states Ŝi,j that contain e. Specifically, for stream Si, let θi(e)≧1 be a lower bound on the number of sites for which e appears in the Ŝi,j states communicated to the coordinator. Then, even if element e is newly inserted into Si,j at site j (that is, e∈(Si,j−Ŝi,j)), the methodologies should not charge for it since e is already in Ŝi and, thus, cannot possibly be in Si−Ŝi. Similarly, if e is deleted from Si,j at site j (that is, e∈(Ŝi,j−Si,j)), then in order for e to be deleted from Si and thus be in Ŝ1−Si,e, must be deleted from Si,j at least θi(e) sites. Thus, it suffices to charge φj−(e)=1/θi(e) (instead of 1) for the local delete of e at each site j. This way, if local deletions at the ≧θi(e) sites cause e to be globally deleted (that is, e∈(Ŝi−Si)), then the cumulative charge Σjφj−(e) for e across the sites is at least 1. As a result, since Σeφj−(e)≦εj at each site j, this total charge of 1 is counted against the various εjs, and correctness is not compromised.
2. Distributed Algorithm.
The details of a distributed scheme for producing a correct cardinality estimate |Ê| at the coordinator will now be described. For each element e∈Ŝi, the coordinator maintains a count Ci(e) of the number of remote sites whose states Ŝi,j contain the element e. Elements whose counts Ci(e) exceed a threshold τ are considered to be frequent, and added to a frequent element set Fi for stream Si. The coordinator also uses the count Ci(e) for each element e∈Fi to compute a lower bound threshold θi(e) such that the invariant Ci(e)≧θi(e) always holds. It continuously communicates changes in the frequent element sets Fi and the threshold values θi(e) to the remote sites so that these can be used to compute element charges φj(e) at the sites (as described in the previous subsection). Thus, in order to keep the message overhead under control, the coordinator does not send exact element counts Ci(e) to remote sites, but rather disseminates the thresholds, as described in the paragraph below. Each remote site j keeps track of the sum of local element charges Σeφj(e) in variable Φj. Further, when Φj becomes greater than εj, it sends the deltas Δi+=Si,j−Ŝi,j and Δi−=Ŝi,j−Si,j that capture the local state changes for substream Si,j since site j last transmitted state information to the coordinator. (Note that the deltas are sets and not multi-sets).
Coordinator Site Actions.
The correctness of the distributed scheme is noted by the fact that for each element e∈Fi, the threshold value θi(e) is always a lower bound on the number of sites j for whom e is in the local state Ŝi,j sent to the coordinator. Thus, the scheme for modifying Fi and θi(e) needs to preserve the invariant Ci(e)≧θi(e) while controlling the number of messages between the coordinator and remote sites. Clearly, to maintain the invariant, the coordinator needs to send messages to all sites every time the count Ci(e) drops below the current threshold θi(e) for an element e∈Fi. Consequently, in order to prevent minor fluctuations in the value of Ci(e) from generating excessive amounts of control message traffic, one strategy is to try and keep a sufficient gap between Ci(e) and θi(e). Thus, for instance, if Ci(e) becomes less than θi(e), then the methodologies simply halve the value of θi(e) (or reduce by some amount other than a factor of 2). Similarly, the methodologies double (or increases by some amount other than a factor of 2) the value of θi(e) only when Ci(e) exceeds 4θi(e), and (conservatively) consider an element to be frequent only if Ci(e) exceeds 2τ. It is noted that other schemes can be used to propagate global knowledge of element counts, as long as the invariant Ci(e)≧θi(e) is satisfied. For efficiency purpose, it may be desirable to choose schemes that minimize the number of control messages used to propagate the global count information.
An additional mechanism used in optional embodiment of the invention has been found by the inventors to be effective for keeping the volume of control messages low. Specifically, to double θi(e) only after the count Ci(e) is somewhat stable (that is, has stayed above 3θi(e) for a certain time period after crossing 4θi(e)). Using this strategy, the number of control messages is relatively insensitive to the value of the threshold parameter τ. Finally, observe that while increasing θi(e) is not required for preserving the invariant Ci(e)≧θi(e), larger θi(e) values are key to reducing the charges φj(e) that sites incur for elements.
Remote Site Actions.
The procedure COMPUTECHARGE in
Correctness Argument.
For ease of exposition, in the arguments pertaining to the correctness of the distributed scheme, it is assumes that all message transmissions and the actions they trigger are performed instantaneously. While not a perfect assumption, the scheme can be extended to simulate such an instantaneous execution (at a logical level) by having sites send special acknowledgements for messages once all the actions triggered by the messages have completed. Additional details are provided in U.S. patent application entitled METHOD FOR DISTINCT COUNT ESTIMATION OVER JOINS OF CONTINUOUS UPDATE STREAMS, Ser. No. 10/957,185 which is incorporated herein by reference in its entirety.
The charges φj+(e) and φj−(e) computed by COMPUTECHARGE can be shown to satisfy the following two invariants:
For each e∈E−Ê, Σjφj+(e)≧1 (equation 1)
For each e∈Ê−E, Σjφj−(e)≧1 (equation 2)
Thus, the distributed scheme is correct because it can be shown that Equation (1) implies that |E|−ε≦|Ê| and Equation (2) implies that |Ê|≦|E|+ε. Moreover, as discussed earlier, the space usage of the distributed algorithm can be reduced by storing a compact sketch synopsis for each substream Si,j instead of the entire substream state. This scheme provides probabilistic as opposed to deterministic error guarantees. For instance, it can maintain a (delete-resistant) distinct sample for each substream, and use the substream samples in place of the substream states in our distributed scheme.
Estimating Cardinality of Arbitrary Set Expressions
The previously described methodologies will now be applied to the problem of estimating (to within e absolute error) the cardinality of an arbitrary set expression E involving the distributed update streams S0, . . . , Sn-1. The distributed scheme for general set expressions is identical to the scheme for single streams except for the charging procedure COMPUTECHARGE. Thus, as previously noted, for each stream Si, the coordinator maintains the states Si, the frequent sets Fi, and the threshold values θi(e) for the number of sites j whose shipped state Ŝi,j contains element e. The cardinality estimate at the coordinator is |Ê|, where Ê is the result of evaluating expression E using Ŝi instead of Si. The coordinator processes the deltas from a remote site for an arbitrary stream Si as described in procedure COORDINATOR (see
In one embodiment of the charging procedure for the single stream case, the methodologies charged 1 for inserts and deletes of elements e∉Fi, and if e∈Fi, inserts were free (i.e., not charged) and deletes were charged 1/θi(e). However, when E contains multiple streams, computing the charge φj(e) for an element e is more involved since e may be concurrently inserted/deleted from more than one substream Si,j at site j. In one embodiment, this complication is overcome by setting the charges φj+(e)=φj−(e)=1 if, for any of the substreams Si,j, either e∈(Si,j−Ŝi,j) or e∈(Ŝi,j−Si,j). However, while this scheme is correct it may be too conservative in some applications, and may end up overcharging in some situations. This, in turn, could lead to frequent state transmission messages from remote sites to the coordinator. It is noted that any scheme satisfying the invariants of equations 1 and 2 can be used to determine element charges. However, for efficiency purposes (i.e., to minimize communication) it is desired to use a scheme that allocates as low a charge as possible to inserts and deletes of elements.
Consider distributed streams S1, S2, and S3, and let expression E=S1∩(S2−S3). For element e at site j, let e∈Ŝ3,j and e∈S3,j. Clearly, e∈Ŝ3 and e∈S3, and thus e∉Ê and e∉E. As a result, even if e∈(Ŝ1,j−S1,j) or e∈(S2j−Ŝ2,j), the methodologies should not charge for element e at site j since e cannot possibly be in either E−Ê or Ê−E; thus, based on the semantics of expression E, setting the charges φj+(e)=φj−(e)=0 will still ensure correctness.
As will be discussed below, for an arbitrary set expression E, the methodologies focus on the problem of computing the minimum possible charges φj+(e) and φj−(e) for a fixed element e at site j by leveraging the semantics of expression E. The various charging schemes ensure that charges φj+(e) and φj−(e) satisfy Equations (1) and (2), and thus provide an accuracy guarantee of ε for the final estimate |Ê|.
The first charging method is denoted as a Model-Based Charging Scheme, and is based on enumerating models for a Boolean formula corresponding to expression E. Finding the optimal charge under the scheme can be shown to be NP Hard and, thus, has an exponential time complexity. The methodologies develop a heuristic that at the expense of overcharging in some situations described later below is able to eliminate model enumeration altogether, and bring down the time complexity so that it is polynomial in the number of streams.
The methodologies developed by the inventors provide that a stream Si has a local state change at site j if either e∈(Si,j−Ŝi,j) or e∈(Si,j−Ŝi,j). Similarly, the methodologies provide that a stream Si has a global state change if either e∈(Si−Ŝi) or e∈(Si−Ŝi).
A Model-Based Charging Scheme
The charging procedure first constructs a Boolean formula Ψj that captures the semantics of expression E and local stream constraints at each site j. It then defines the charge φj(e) at site j in terms of the charges for models M that satisfy Ψj.
In one embodiment to construct the Boolean formula, for each stream Si, let pi and {circumflex over (p)}i be Boolean variables with semantics e∈Si and e∈Ŝi, respectively. The methodologies construct two Boolean formulae Ψj+ and Ψj− over the variables pi and {circumflex over (p)}i. Intuitively, Ψj+ and Ψj− specify the conditions that stream states Si and Ŝi must satisfy for e∈(E−Ê) and e∈(Ê−E), respectively. The formulae also capture constraints on Si and Ŝi due to local knowledge at site j of the substream states Si,j, Ŝi,j, and threshold values θi. For example, if e∈Si,j, then it must be the case that e∈Si (since Si=∪jSi,j), and thus, variable pi must be true.
The formulae Ψj+ and Ψj− are built using the following three formulae: (1) an Expression formula FE representing the logic of expression E, (2) State formulae Ĝj,Gj that model the local knowledge that site j has about stream states Si and Ŝi, and (3) a Threshold formula H that captures the constraints due to the thresholds θi for each stream Si.
Expression Formula.
The expression formula FE is constructed recursively as follows:
1. For every stream in Si in E, the methodologies replace its occurrence by the Boolean variable pi;
2. The expression E1∪E2 is translated as FEFE
3. The expression E1∩E2 is translated as FEFE
4. The expression E1−E2 is translated as FE(
FE
For example, the set expression E=S1∩(S2−S3) is translated into the Boolean formula FE=pi(p2
p3). It is then seen that element e∈E if FE is true for the stream states Si. For instance, e∈S1∩(S2−S3) if e∈S1
(e∈S2
e∉S3). Formula {circumflex over (F)}E is constructed similarly, except that variables pi are replaced by {circumflex over (p)}i.
State Formula.
The state formulae Gj and Ĝj are conjunctions of a subset of the Boolean variables pi and {circumflex over (p)}i, respectively. In one embodiment, if e∈Si,j, then variable pi is added to Gj. Thus, Gj captures the constraints on streams Si for whom the methodologies can infer that e∈Si based on local information that e∈Si,j at site j. Similarly, the methodologies construct Ĝj by adding variable {circumflex over (p)}i to it if e∈Ŝi,j. Note that Gj and Ĝj may be different for the various remote sites depending on the substream states at each site.
Threshold Formula.
The threshold formula H only applies to Boolean variables {circumflex over (p)}i. If e∈Fi for stream Si, then the methodologies add variable {circumflex over (p)}i to H. Thus, H captures the constraints on stream states Ŝi for whom the methodologies can deduce that e∈Ŝi from the frequent element sets. Note that formula H is identical at all sites since Fi is the same at all sites. That is, the above protocol generally is adapted to the condition in which the set of frequent elements Fi is identical across all sites. However, in an alternate protocol the set of frequent elements Fi,j need not be identical across all sites. In this alternate protocol, the coordinator needs to maintain the invariant Ci(e)≧θi,j(e) for all j.
The formulae Ψj+ and Ψj− at site j are constructed as follows.
Ψj+=({circumflex over (F)}E
FE)
(Ĝj
Gj
H)
Ψj−=({circumflex over (F)}EFE)
(Ĝj
Gj
H)
The formulae Ψj+ and Ψj− comprise two parts. The first part, involving FE and {circumflex over (F)}E, captures the conditions for one of e∈(E−Ê) or e∈(Ê−E) to hold. The second part (ĜjGj
H) specifies the constraints on stream states Ŝi and Si due to local knowledge at site j of substream states and frequent element sets. Thus, for the Boolean formula Ψj+, it follows that e∈(E−Ê) if Ψj+ is true for stream states Ŝi,Si. Consequently, if Ψj+ is unsatisfiable, then it is impossible that e∈(E−Ê), and so the methodologies can set φj+(e)=0. Similarly, if Ψj− is unsatisfiable, then charge φj− (e)=0.
Revisiting Example 1, where E=S1∩(S2−S3), and element e∈Ŝ3,j and e∈S3,j, the methodologies get that Ψj+=({circumflex over (p)}1
{circumflex over (p)}2
{circumflex over (p)}3)
(p1
p2
p3)
({circumflex over (p)}3
p3). It is noted that Ψj+ is unsatisfiable (due to
p3
p3), and thus, charge φj+(e)=0. In the following discussion, the methodologies show how models for Ψj can be used to compute the charges φj+(e) when Ψj is satisfiable.
Computing Charges Using Formula.
As an overview, consider the problem of computing the charge φj+(e). For an arbitrary Boolean formula over {circumflex over (p)}i,pi, the methodologies define a model to be an arbitrary subset of ∪i{pi,{circumflex over (p)}i}. Each model M assigns truth values to variables pi,{circumflex over (p)}i with variable pi({circumflex over (p)}i) being assigned true if pi∈M (resp., {circumflex over (p)}i∈M); otherwise, pi(resp., {circumflex over (p)}i I) is assigned false. The methodologies say that model M satisfies a Boolean formula if the formula evaluates to true for the truth assignment specified by M. For example, model {{circumflex over (p)}1,p2} satisfies the formula {circumflex over (p)}1p2, but the model {{circumflex over (p)}1} does not. Now, each model M represents a specific scenario for states Ŝi,Si. Essentially, e∈Si(e∈Ŝi) if pi∈M (resp., {circumflex over (p)}i∈M). If e∈(E−Ê) for stream states Si,Ŝi, then the model corresponding to these states must satisfy Ψj+. Further, every model M that satisfies Ψj+ represents (from the local viewpoint of site j) a possible scenario for states Ŝi,Si that is consistent with local substream states at site j, and in which e∈(E−Ê).
This model-based approach assigns a charge φj(M) to each model M that satisfies Ψj+ at site j. Furthermore, since as far as site j is concerned, any of these models can potentially occur and cause e∈(E−Ê), the methodologies set charge φj+(e) as follows.
φj+(e)=max{φj(M): Model M satisfies Ψj+} (equation 3)
As previously noted, if e∈(E−Ê), then Σjφj+(e)≧1. Thus, by choosing the charge φj(M) for each model M such that Σjφj+(M)≧1 if M were to occur, the methodologies can ensure that Σjφj+(e)≧1 if e∈(E−Ê) due to some model M that satisfies Ψj+.
To compute the charge φj(M) for a model M that satisfies Ψj+ let P be the set of streams Si such that exactly one of pi or {circumflex over (p)}i belongs to M, i.e., either {pi,{circumflex over (p)}i}∩M={pi} or {pi,{circumflex over (p)}i}∩M={{circumflex over (p)}i}. Thus, P is the set of streams that experience a global state change in model M. In an exemplary model-based scheme, site j selects a single “culprit” stream Si from P using a selection mechanism that satisfies the following property:
UNIFORM CULPRIT SELECTION PROPERTY: Given a model M and a set P of streams with global state changes in M, every site selects the same culprit stream Si∈P for M.
As further discussed below, a specific culprit selection scheme satisfying the above property that attempts to minimize the magnitude of the charge φj+(e) at site j will be provided. For the selected culprit stream Si, let charge φ(Si) be defined as follows:
The reciprocal of this charge 1/φ(Si) is the lower bound on the number of sites where stream Si must have local state changes for it to have a global state change. For instance, if e∈Fi, then for e to be in (Ŝi−Si), e must be in (Ŝi,j−Si,j) for at least 1/φ(Si)=θi(e) sites. The methodologies define the charge φj+(M) for model M in terms of the charge φ(Si) for the culprit stream Si, as follows:
Thus, the methodologies are able to ensure that if model M indeed does occur, then since the culprit stream Si has a global state change in M, at least the 1/φ(Si) sites j at which Si has local state changes, choose φj(M)=φ(Si) and thus, Σjφj(M)≧(1/φ(Si))φ(Si)≧1.
The correctness of the charging scheme follows from Lemma 1 below.
Lemma 1: Let charge φj+(e) be computed as described in Equations (3), (4) and (5), and the culprit stream Si for each model M selected using a scheme that satisfies the uniform culprit selection property. If e∈(E−Ê), then Σjφj+(e)≧1. (An analogous lemma holds for φj−(e))□
Culprit Selection. For a model M, in one embodiment a culprit selection scheme is implemented as follows: lexicographically order streams Si∈P based on charge, index pairs <φ(Si),i>, and choose the smallest stream in the lexicographic ordering as the culprit. In other words, the culprit stream is the stream with the minimum charge φ(Si), with ties being broken in favor of the stream with the smallest index. Since the charge φ(Si) for stream Si is the same across all the sites, the simple culprit selection scheme satisfies the uniform culprit selection property. Thus, due to Lemma 1, the charging procedure is correct. Also, the charging procedure selects the stream with the smallest charge as the culprit for model M, it minimizes the maximum charge incurred for M across the sites.
Consider distributed streams S1, S2 and S3, and let expression E=S1∩(S2−S3). At some site j, let the substream states be as shown in the table below.
Thus, element e is in all substream states except for Ŝ1,j and S3,j. Also, let e∈F3, e∉F1, e∉F2 and θ3(e)=4; the meaning here is that e is contained in at least 4 substream states for S3 transmitted to the coordinator. It follows that φ(S1)=φ(S2)=1 and φ(S3)=1/4. Also, the formula Ψj+ for E at site j is given as:
({circumflex over (p)}1
{circumflex over (p)}2
{circumflex over (p)}3)
(p1
p2
p3)
(p1
{circumflex over (p)}2
{circumflex over (p)}3)
Thus, for any model M that satisfies Ψj+, it must be the case that {{circumflex over (p)}3,p3}. As a result, S3∈P and since the charge φ(S3) for S3 is the smallest, it is chosen as the culprit for all models. Consequently, since S3 has a local state change at site j, φj(M)=φ(S3)=1/4 for all models M that satisfy Ψj+, and thus, the charge φj+(e)=1/4. Furthermore, since Ψj− is unsatisfiable, charge φj−(e)=0.
If the stream S3 does not have a local state change at site j (i.e., e is neither in Ŝ3,j nor in S3,j), then e∈F3, Ψj+ will remain the same as before and S3 will still be chosen as the culprit stream for all models M that satisfy Ψj+. However, since S3 does not have a local state change at site j, φj(M) will be 0 for the models, and thus charge φj+(e)=0.
Computational Complexity.
In order to determine the complexity of the model-based approach, the following decision problem for φj+(e) is considered.
PROBLEM (MAXIMUM CHARGE MODEL): Given expression E, site j, element e, and constant k, does there exist a model M that satisfies Ψj+ and for which φj(M)≧k?
The following theorem can be proved using a reduction from 3-SAT: The MAXIMUM CHARGE MODEL problem is NP-complete. Therefore, it follows that since φj+(e) is the maximum charge for models M that satisfy Ψj+, computing φj+(e) is intractable.
Heuristic for Charge Computation.
The model-based charging procedure enumerates all models M in the worst case, and thus, has a worst-case time complexity of O(22n). While this may be reasonable for small values of n (e.g., 3 or 4 streams), the model enumeration-based approach will clearly not scale when set expressions involve a moderately large number of streams, a scenario likely in practice. (e.g. in the Akamai case). In this section, the methodologies present a heuristic solution for computing the charges φj+(e) and φj−(e) for an element e at site j. The heuristic procedure has a time complexity that is polynomial in the number of streams n, and computes identical charge values as the model-based approach as long as every stream appears at most once in the expression E. However, the heuristic may overcharge for element e in certain cases when there are duplicate occurrences of streams in expression E.
Overview.
The model-based charging procedure computes φj+(e) as the maximum stream charge φ(Si) such that (1) Si has a local state change at site j, and (2) Si is the culprit stream for some model M that satisfies Ψj+ (Recall that the culprit stream Si for model M is the stream with the smallest charge, index pair <φ(Si),i> from among streams with a global state change in M.) Thus, for a stream Si, if the methodologies can develop a test for quickly determining if Si is the culprit stream for some model that satisfies Ψj+, then the methodologies can speed up the computation of charge φj+(e). This is a key idea underlying the heuristic.
Let T denote the expression tree for E with leaves and internal nodes corresponding to streams and set operators in E, respectively. For each node V of T, let E(V) be the subexpression for the subtree rooted at node V, and {circumflex over (F)}E(V) and FE(V) be the formulae for E(V) as defined in Section 4.1.1. For example, in the expression tree for E=S1∩(S2−S3), the subexpression for the subtree rooted at V=“−” is E(V)=S2−S3, and FE(V)=p2p3. Now, in order to quickly test if a stream Si is the culprit stream for some model satisfying Ψj+, the heuristic keeps track of culprit streams (for models) at each node of the expression tree using the notion of charge triples. Formally, suppose that M is a model that satisfies the local constraints (Gj
Ĝj
H) at site j. At node V in T, the methodologies define the charge triple for model M, denoted by t(M,V), as the triple (a,b,x) with the following values:
If M satisfies {circumflex over (F)}E(V), then bit a=1; otherwise, a=0.
Similarly, if M satisfies FE(V), then bit b=1; otherwise b=0.
If none of the streams in V's subtree have a global state change in model M, then x=∞. (The charge, index pair <φ(S∞),∞> is considered to be greater than <φ(Si),i> for all streams Si) Otherwise, x is the index of the culprit stream for M in V's subtree; that is, x=i, where Si is the stream with the smallest charge, index pair <φ(Si),i> from among streams (in V's subtree) with a global state change in M.
For example, consider a model M that satisfies {circumflex over (F)}E(V)
FE,V (in addition to local constraints). Then, if the culprit stream Si for M in V's subtree is defined, the charge triple t(M,V) for M at node V is (0,1,i); otherwise, t(M,V)=(0,1∞). The charging heuristic computes, in a bottom-up fashion, a set C of charge triples for each node V of T. Furthermore, it ensures that for every model M that satisfies (Gj
Ĝj
H), the computed set C for node V contains the triple t(M,V). Here, it is important to note that the size of C (in the worst case) is linear in the number of streams n—this is because there are at most O(n) distinct charge triples t(M,V) (one for each combination of a,b and x).
Now, consider the charge triple set C for the root V of T. Clearly, since E(V)=E, if a model M satisfies Ψj+=({circumflex over (F)}E
FE)
(Ĝj
Gj
H) and has culprit stream Si, then triple t(M,V)=(0,1,i) must be in C. Thus, the methodologies can quickly determine if a stream Si is the culprit stream for some model satisfying Ψj+ by checking if C contains the triple (0,1,i). Hence, by selecting φj+(e) to be the maximum stream charge φ(Si) such that (1) Si has a local state change at site j, and (2) triple (0,1,i)∈C, the methodologies can ensure that Model M satisfies Ψj+} and thus, due to Lemma 1, the charging heuristic is correct.
The following example illustrates the execution of a bottom up charge triple computation (a more detailed explanation is provided below with respect to
Consider the distributed scenario described in Example 2 involving streams S1, S2 and S3, and expression E=S1∩(S2−S3). Suppose that element e is in all substream states except for Ŝ1,j and S3,j, and also e∈F3 and θ3(e)=4. Thus, φ(S1)=φ(S2)=1 and φ(S3)=1/4.
The charge triple set for each leaf Si is first initialized to contain t(M,Si) for models M that satisfy local constraints. For example, since e is in S1,j but not in Ŝ1,j, it follows that p1∈Gj and thus for models M that satisfy (GjĜj
H), pi∈M but {circumflex over (p)}1 may or may not be in M; so the charge triple set for S1 contains the triples (1,1,∞) (for models that contain {circumflex over (p)}1) and (0,1,1) (for models that do not contain {circumflex over (p)}1).
Next, the charge triple (a,b,x) for each internal node V is computed by combining pairs of triples (a1,b1,x1) and (a2,b2,x2) from V's two children. Suppose that op is the Boolean operation corresponding to the set operation for V, the Boolean operations for ∪,∩ and − are ,
and
, respectively. Then a=a1 op a2,b=b1 op b2, and x is set to one of x1 or x2, whichever has the smaller charge, index pair <φ(Sx,i),xi> For example, the charge triples for node “−” of T are generated by combining triples for nodes S2 and S3. Triples (1,1,∞) and (1,1,∞) when combined result in the triple (0,0,∞) (since 1
1=0). Similarly, combining triples (1,1,∞) and (1,0,3) results in the triple (1,0,3) (since 1
0=1, and <φ(S3),3> is less than <φ(S∞),∞>). Finally, the sets for Si and “−” are combined to obtain the charge triple set C for the root node “∩”, which is then used by the charging heuristic to compute the charges φj+(e) and φj−(e). Since C contains the triple (0,1,3) and S3 has a local state change at site j, charge φj+(e)=φ(S3)=1/4. Further, since C does not contain a triple of the form (1,0,x),φj−(e)=0.
Correctness Argument. The following lemma establishes the correctness of] the charging heuristic.
Lemma 2: Consider a model M that satisfies local constraints (GjĜj
H) at site j. Then, for an arbitrary node V in T, charge triple t(M,V) is in the set of charge triples for V computed by the heuristic.
Computational Complexity. The maximum size of a charge triple set for a node is O(n), and thus, the worst-case time complexity of the charging heuristic can be shown to be O(n2s), where s is the size of set expression E. By using the following pruning optimization, the maximum size of a charge triple set for a node can be reduced to O(1), and thus the time complexity of our charging heuristic can be reduced to O(s):
At each node having charge triples of the form (a,b,x), at most two distinct charge triples are maintained for each of the four possible combinations of values a and b can take:
The following lemma implies that when E contains no duplicate streams, the heuristic returns the same charge values as the model based approach.
Lemma 3: Let E be a set expression in which each stream appears at most once. For an arbitrary node V in T, charge triplet is in the set of charge triples for V computed by the heuristic if and only if t=t(M,V) for some model M satisfying (GjĜj
H) at site j.
The following discussion is directed to more detailed descriptions of alternate embodiments of the invention.
Charging Schemes
Correctness Invariants
The charging schemes presented herein (such as the procedure C
For each e∈E−Ê, Σjφj+(e)≧1 (equation 6)
For each e∈Ê−E, Σjφj−(e)≧1 (equation 7)
These invariants are necessary for correctness of any distributed charging scheme. In this section, it will be shown that maintaining these invariants are also sufficient to ensure correctness of the charging schemes. In particular, Equation (7) implies that |Ê|≦|E|+ε and (analogously) Equation (6) implies that |E|−ε≦|Ê|. Suppose Equation (7) holds. Then, we have
In an analogous manner, Equation (6) implies that |E|−ε≦|Ê|.
Pseudo Code for Model-Based Charging Algorithm
Pseudo Code of Procedure for Tree-Based Charging Heuristic
e ∈ Fi)
e ∈ Si, j
e ∈ Fi
e ∉ Si, j
e ∉ Fi
e ∉ Si, j
e ∈ Fi
e ∉ Si, j
e ∉ Fi
e ∈ Si, j
e ∉ Fi
e ∉ Si, j
Ĝj
H) can be grouped under one of four categories depending on whether or not {circumflex over (p)}i∈M, pi∈M. The models M in each category map to the same charge triple (a,b,x), where a=1 iff {circumflex over (p)}i∈M and b=1 iff pi∈M. Furthermore, if a=b, then since Si does not have a global state change in M, x=∞. On the other hand, if a≠b, then x=i since Si is the (only) culprit stream for M. For example, suppose that e∈Ŝi,j, e∉Si,j and e∈Fi. Then, {circumflex over (p)}i∈Ĝj, {circumflex over (p)}i∈H, and for the models M that satisfy (Gj
Ĝj
H), we have that {circumflex over (p)}i∈M, while pi may or may not belong to M. Thus, for models M that contain pi, the charge triple is (1,1,∞), and for those that do not, the charge triple is (1,0,i).
Next, consider the case when V is an internal node for one of the set operators in {∪,∩,−}. Let C1 and C2 be the charge triple sets for V's two children V1 and V2. Then, for a model M satisfying (GjĜj
H), the charge triple t(M,V) in the charge triple set C at node V can be obtained by combining the charge triples t (M,V1)∈C1 and t (M,V2)∈C2 for M at nodes V1 and V2, respectively. To see this, suppose that t(M,V1)=(a1,b1,x1) and t(M,V2)=(a2,b2,x2). Also, let op=bool_op(V), where bool_op(∪)=V, bool_op(∩)=
and bool_op(−)=
are Boolean operators corresponding to the set operators.
From the construction of FE(V), it is known that FE(V)=FE(V
While the foregoing is directed to various embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof. As such, the appropriate scope of the invention is to be determined according to the claims, which follow.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5926816 | Bauer et al. | Jul 1999 | A |
| 6188695 | Przybysz | Feb 2001 | B1 |
| 6449618 | Blott et al. | Sep 2002 | B1 |
| 6460010 | Hanes et al. | Oct 2002 | B1 |
| 6738789 | Multer et al. | May 2004 | B2 |
| 6757696 | Multer et al. | Jun 2004 | B2 |
| 6820034 | Hanes et al. | Nov 2004 | B2 |
| 6826523 | Guy et al. | Nov 2004 | B1 |
| 7299219 | Green et al. | Nov 2007 | B2 |
| 7305483 | Yao et al. | Dec 2007 | B2 |
| 7395075 | Posner et al. | Jul 2008 | B2 |
| 7401002 | Hanes et al. | Jul 2008 | B2 |
| 20020055972 | Weinman, Jr. | May 2002 | A1 |
| 20060085592 | Ganguly et al. | Apr 2006 | A1 |
| 20080163189 | Chen et al. | Jul 2008 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20060149744 A1 | Jul 2006 | US |
