Patent Application
20110255424
The present invention relates to a data processing method and system for monitoring traffic on a computer network, and more particularly to a technique for efficiently summarizing Internet traffic patterns.
Efficient processing of massive databases is a challenging and important task in applications such as observational sciences, product marketing and monitoring large systems. A data set being processed by in a massive database may be distributed across several network devices, each receiving a portion of the data as a stream. The devices locally process their data, producing a small sketch, which can then be transmitted to other devices for further processing. Known techniques focus on producing sketches of minimal size for various problems, thereby providing space-efficient processing, while failing to adequately provide time-efficient processing. Thus, there exists a need to overcome at least one of the preceding deficiencies and limitations of the related art.
In one or more embodiments, the present invention provides a computer-implemented method of monitoring network traffic by estimating an L1 difference. The method comprises:
receiving a first sketch s(x) from a first network device and a second sketch from a second network device, wherein the first sketch describes a first set of pairs of data (x, ft(x)), wherein x indicates a source and a destination of a first plurality of data packets that include input x, transmitted via the first network device, wherein ft(x) indicates a first amount of information in the first plurality of data packets transmitted via the first network device from the source to the destination in a time period t, wherein the second sketch s(y) describes a second set of pairs of data (y, ft′(y)), wherein y indicates the source and the destination of a second plurality of data packets that include input y transmitted via the second network device, wherein ft′(y) indicates a second amount of information in the second plurality of data packets transmitted via the second network device from the source to the destination in a time period t′;
a processor of a computer system estimating an L1 difference ∥ft(x)−ft′(y)∥1, wherein the estimating the L1 difference includes determining an L0 value of a data stream resulting from the first network device inserting elements of a first set of distinct items as insertions into the data stream and from the second network device inserting elements of a second set as deletions from the data stream, wherein the first set includes distinct items derived from xi, wherein the second set of distinct items includes distinct items derived from yi, and wherein the determining the L0 value includes processing updates to ranges of the data stream; and
determining a pattern of the network traffic based on the L1 difference.
A system and program product corresponding to the above-summarized method are also described and claimed herein.
Embodiments of the present invention provide space- and time-efficient sketches for the determination of an L1 distance for summarizing contents of databases distributed over a network and for Internet traffic monitoring. Furthermore, embodiments of the present invention allow network devices to compute their sketches in only O(n log2(nM)) time.
The problem of producing space-efficient and time-efficient sketches for the L1 distance (i.e., the Manhattan distance) has applications related to summarizing contents of databases distributed over a network, and is especially useful for monitoring computer network traffic (e.g., monitoring Internet traffic). Embodiments of the present invention describe a method and system that utilizes a sketching procedure that provides nearly-optimal sized sketches for L1 in nearly linear time. Specifically, suppose a first computing device (e.g., a first router) receives a vector x ∈ {−M, . . . , M}n of data, a second computing device (e.g., a second router) receives a vector y ∈ {−M, . . . , M}n of data, and the first and second computing devices share randomness. As used herein, computing devices that share randomness are computing devices that have access to a shared random key, such as a cryptographic key or a public book of random numbers. Hereinafter, the first and second computing devices are also referred to as “Alice” and “Bob,” respectively. Alice and Bob also receive a parameter 0<ε≦1. Alice outputs a sketch s(x), and Bob outputs a sketch s(y), such that a third party (e.g., a computer system) can later quickly recover (1±ε)∥x−y∥1, i.e., a value Z ∈ [(1−ε)∥x−y∥1, (1+ε)∥x−y∥1], given only s(x) and s(y). As used herein, ƒx−y∥1−Σi=1n|xi−yi| and denotes the L1-norm of the vector x−y (i.e., denotes the L1-difference of x and y). The third party succeeds with probability at least ⅔ over the randomness of Alice and Bob.
In embodiments of the present invention, Alice and Bob run a sketching procedure to compute the sketches s(x) and s(y) in O(n log2(nM)) processing time, which is independent of ε. The size of each of the sketches s(x) and s(y) is a nearly-optimal O(ε−2 log(1/ε)log(nM)) bits. Thus, the time for a third party to recover a (1±ε)-approximation to the L1-difference, given the sketches, is nearly linear in the sketch size. Furthermore, the sketching procedure disclosed herein may be implemented as a one-pass streaming algorithm over an adversarial ordering of the coordinates of x, y (i.e., the sketching procedure works even if the entries of x, y are given in arbitrary order in a data stream). Hereinafter, the sketching procedure is also referred to as a streaming algorithm.
A known sketching procedure for estimating the L1-difference is described in Feigenbaum et al., An approximate L1-difference algorithm for massive data streams, SIAM Journal on Computing 32(1):131-151, 2002, achieves an optimal sketch length of O(ε−2 log(nM)) bits, but has a running time of O(n log(nM)/ε2), which unlike the present invention, is dependent upon ε.
Although systems and methods are discussed herein relative to network traffic monitoring, it will be apparent to those skilled in the art that the systems and methods may be modified so that databases generate sketches of their data, and the central processing facility receives the sketches from the databases and given the received sketches, determines whether or not the databases include the same data.
Router 104, router 106 and any other router (not shown) each include a software-based sketch generator 110 that generates a sketch (i.e., a summary or description) of information of a flow (a.k.a. flow information), where the flow includes a vector of data received by the router that includes the sketch generator. The characteristics of a sketch generated by sketch generator 110 are presented below relative to the discussion of
Computer system 102 includes the following software-based components: a sketch collector 112, an L1-difference estimator 114, and a network traffic monitor 116. Sketch collector 112 receives sketches from routers 104, 106 via network 108. L1-difference estimator 114 estimates an L1-difference between one vector of data received by router 104 and another vector of data received by router 106. To estimate the L1-difference, L1-difference estimator 114 uses one sketch generated by router 104 and received by sketch collector 112 and another sketch generated by router 106 and received by sketch collector 112, as described below relative to
Details of the functionality of computer system 102 and its components, and the functionality of routers 104, 106, and their respective components are included in the discussions presented below relative to
In one embodiment, the aforementioned Alice and Bob are implemented as routers 104 and 106, respectively, and the aforementioned third party that recovers (1±ε)∥x−y∥1, given only sketches s(x) and s(y) respectively generated by Alice and Bob is implemented as computer system 102.
In step 206, a second router (e.g., router 106 (see
In one embodiment, the first router and the second router are the same router and time period t and t′ are different time periods. In another embodiment, the first router and the second router are different routers and time periods t and t′ are the same time period.
In step 210, sketch collector 112 (see
In step 214, network traffic monitor 116 running in computer system 102 determines differences in traffic patterns on network 108 (see
The aforementioned set of pairs (x, ft(x)) may be very large for the given time period t, and the method described herein provides a very fast (i.e., in near linear time) procedure for sweeping through the set and producing a very short (i.e., logarithmic size in the original description of the set) description Dt of the set of pairs.
Since storing the complete set of pairs is expensive in terms of computer data storage resources required, router 104 in
The ability of sketch generator 110 (see
In one embodiment of the method of summarizing Internet traffic patterns, the L1-difference problem is embedded into L0, the number of non-zero coordinates of the underlying vector (i.e., x−y) presented as data stream. Suppose for simplicity xi, yi≧0 for all i ∈ [n]. In this embodiment, Alice treats her input x, as a set of distinct items M(i−1) +1, . . . , M(i−1)+xi, while Bob treats his input yi as a set of distinct items M(i−1)+1, . . . , M(i−1)+yi. The size of the set-difference of these two sets of distinct items is |xi−yi|. Thus, if Alice inserts all of the set elements corresponding to her coordinates as insertions into an L0-algorithm, while Bob inserts all of the set elements corresponding to his coordinates as deletions in the L0-algorithm, the L0-value in the resulting stream equals ∥x−y∥1. In known techniques, the problem with directly reducing to L0 is that, while the space of the resulting L1-difference algorithm is an optimal O(ε−2 log(nM)), the processing time can be as large as O(nM) because each set element must be inserted into the L0-algorithm. The present invention overcomes this problem by providing a range-efficient L0 algorithm that allows updates to ranges at a time, which works for streams coming out of the reduction to L0 by exploiting the structure of ranges that are updated. All updated ranges are of length at most M and start at an index of the form M(i−1)+1. Furthermore, the range-efficient L0 algorithm described herein allows deletions, a feature not found in known L0 algorithms.
At a high level, the sketching procedure presented herein works by sub-sampling by powers of 2 the universe [nM] arising out of the reduction to L0. At each level in the sub-sampling, the sketching procedure keeps a corresponding data structure (a.k.a. summary data structure) of size O(ε−2 log(1/ε)) to summarize the items that are sub-sampled at the corresponding level. The sketching procedure also maintains another data structure on the side to handle the case when L0 is small, and in parallel, the sketching procedure obtains a constant-factor approximation R of the L1-difference. At the end of the data stream, the sketching procedure provides an estimate of the L1-difference based on the summary data structure that summarizes the items at a level at which the expected number of universe elements sub-sampled is 1/ε2, where the level is determined based on R being known. The sub-sampling of the data stream may be implemented using pairwise-independent hash functions, which allows the use a subroutine for quickly counting the number of universe elements that are sub-sampled at each of the log(nM) levels. Given these counts, each of the aforementioned summary data structures may be updated efficiently.
The summary data structure at a given level may be viewed as maintaining (x′−y′)H, where H is the parity-check matrix of a linear error-correcting code, and x′, y′ are the vectors derived from x, y by sub-sampling at that level. When it is ensured that x′, y′ differ on few coordinates, x′−y′ may be treated as a corruption of the encoding of the 0 codeword followed by an attempt to decode to recover the “error” x′−y′. The decoding succeeds as long as the minimum distance of the code is sufficiently high.
Known techniques use the parity-check matrix of a Reed-Solomon code of minimum distance O(k), given that that x′, y′ differ on at most k coordinates. Decoding may then be done by a known recovery procedure in time O(k2+kpoly log(k) log(n)). In one or more embodiments of the present invention, k=ε−2; therefore, the method disclosed herein does not use the known recovery procedure because it would be too slow. To avoid this slowness problem, the sketching procedure disclosed herein hashes the indices of x′, y′ into O(ε−2/log(1/ε)) buckets with an O(log(1/ε))-wise independent hash function, and then keeps in each bucket the product of the difference vector, restricted to the indices mapped to that bucket, with the parity check matrix. With constant probability, no bucket receives more than O(log(1/ε)) indices where x′, y′ differ. Thus, the sketching procedure disclosed herein may use a Reed-Solomon code with minimum distance only O(log(1/ε)), thereby making the decoding sufficiently fast.
In the Detailed Description of the Invention section, all space bounds are in bits, and all logarithms are base 2, unless explicitly stated otherwise. Running times are measured as the number of standard machine word operations (i.e., integer arithmetic, bit-wise operations, and bitshifts). Each machine word is assumed to be Ω(log(nM)) bits so that each vector can be indexed and arithmetic can be done on vector entries in constant time. Also, for integer A, [A] denotes the set {1, . . . , A}.
The model in which the sketching procedure runs is formally defined as follows. Alice receives x ∈ {−M, . . . , M}n , and Bob receives y ∈ {−M, . . . , M}n. Both computing devices (i.e., Alice and Bob) have access to a shared source of randomness and must, respectively, output bit-strings s(x) and s(y). The requirement is that a third party (e.g., computer system 102 in
Since the sketching procedure is presented below as a streaming algorithm, streaming notation is introduced in this section. Consider a vector f=(f1, f2, . . . , fn) that is updated in a stream as described in this paragraph. The data stream has exactly 2n updates (i1, v1), . . . , (i2n, v2n) ∈ [n]×{−M, . . . , M}. Each update (i, v) corresponds to the action fi←fi+v. For each j ∈ [n], there are exactly two stream updates (i, v) with i=j. If these two stream updates are (iz1, vz1), (iz2, vz2), then at most one of vz1, vz2 is negative, and at most one of vz1, vz2 is positive. The nonnegative update corresponds to adding xi to fi, and the nonpositive update corresponds to subtracting yi from fi. Again, the assumption is that xi, yi≧0. There is no restriction on the possible values for z1 and z2. That is, the sketching procedure described herein functions correctly even if the stream presents an adversarial permutation of the 2n coordinates x1, . . . , xn, y1, . . . , yn. At the end of the stream ∥∫∥1=∥x−y∥1, so the streaming algorithm must approximate ∥∫∥1. For Alice and Bob to use the streaming algorithm for sketching, Alice runs the streaming algorithm with updates (i, xi) for each 1≦i≦n to produce a sketch, and Bob separately runs the streaming algorithm (using the same random bits) with updates (i, yi) to produce a sketch. The sketches produced are simply the contents of the streaming algorithm's memory at the end of the data stream. It is a consequence of how the streaming algorithm works that these sketches can be combined by a third party (e.g., computer system 102 in
In step 302, define t and s as follows: set t=2┌log k┐+12 and set s=2t+┌log k┐. Select a random h1: [n]→[┌k/log k┐] from a t-wise independent hash family and select a random h2: [n]→[p−1] from a pairwise independent family. The value p is calculated prior to step 302 and satisfies Equation 1:
C≦p≦2C, C=4·(5 ┌log k┐+24)2·┌k/log k┐+1 (1)
Prior to step 302, the following items are calculated: a generator g for the multiplicative group *p and logarithm tables T1, T2 such that T1[i]=gi mod p and T2[x]=d log(x), where 0≦i≦p−2 and 1≦x≦p−1. As used herein, d log(x) is the discrete logarithm of x (i.e., the i ∈ GF(p) such that gi≡x mod p). The choice of k will be relatively small in the final algorithm, so this step will be efficient. As used herein, GF(p) may be replaced with the set {0, 1, 2, 3, . . . , p−1}. In general, GF(x) for an integer x denotes the finite field (i.e., Galois field) containing x elements.
In step 304, for each j ∈ [┌k/log┐], maintain 2s counters X1j, . . . , X2sj modulo p, where the counters are initialized to 0.
In step 306, in response to identifying stream update (i, v), increment Xzh
In step 308, in response to detecting the stream's end, for each j ∈ [┌k/log k┐], attempt to recover the non-zero entries of an s-sparse vector fj=((fj)x∈ satisfying Σx∈
((fj)x)xz=Xzj for each z ∈ [2s] using Theorem 1.
In step 310, define σ: GF(p)→ to be such that σ(α) equals α if α<p/2, and equals α−p otherwise. Furthermore, in step 310, output Σj=1┌k/log k┐Σ(f j)x≠0|σ((fj)x)|. The subroutine in
The subroutine T
Theorem 1. Let p be prime and r=(rx)x∈ have at most s non-zero entries (2s+1<p). Given Σx∈
rxxi for i ∈ [2s], there is an algorithm to recover {(x, rx)|rx≠0} which uses O(s2+s(log s) (log log s) (log p)) field operations over GF(p).
The proof of correctness of the T
Lemma 2. Let Xi ∈ [0,1], 1≦i≦n, be t-wise independent for t≧4 an even integer,
Theorem 3. Ignoring the space to store the hash functions h1, h2 and tables T1, T2, the T
Proof of Theorem 3: Aside from storing h1, h2, T1, T2, the number of counters is 2s ┌k/log k┐=O(k), each of size O(log p)=O(log k) bits, totaling O(k log k) bits. The space to store h1 is O((log k) (log n)), and the space to store h2 is O(log n). The tables T1, T2 each have p−1=O(k log k) entries, each requiring O(log p)=O(log k) bits. Processing a stream update requires evaluating h1, h2, taking O(log k) time and O(1) time, respectively. There must be an update of 2s=O(log k) counters (see step 306). Each counter update can be done in constant time with the help of a table lookup since (h2(i))z=gz·d log(h
Correctness is analyzed as follows. Define I={i ∈ [n]: fi≠0 at the stream's end}. Note |I|≦L1≦k. For j ∈ [┌k log k┐], define the random variable Zj=|h1−1(j)∩I|. Two events Q and Q′ are defined below.
Let Q be the event that Zj≦s=2t+┌log k┐ for all j ∈ [┌k/log k┐].
Let Q′ be the event that there do not exist distinct i, i′ ∈ I with both h1(i)=h1(i′) and h2(i)=h2(i′).
It is first argued that, conditioned on both Q, Q′ holding, the output of the T(rj)xxi, where (rj)x is the frequency (modulo p) of the unique element in the set h2−1(i)∩h1−1(j)∩I (or 0 if that set h2−1(i)∩h1−1(j)∩I is empty). Conditioned on Q, every rj is s-sparse, so rj is correctly recovered in step 308 by Theorem 1 since 2s+1=5 ┌log k┌+13<100k ┌log k┐<p. Note that p is strictly greater than twice the absolute value of the largest frequency since L1<k, and thus negative frequencies are strictly above p/2 in GF(p), and positive frequencies are strictly below p/2. Thus, given that the rj are correctly recovered, a correctly recovers the actual frequencies in step 310, implying correctness of the final output of the T
The proof of Theorem 3 now proceeds to lower bound Pr[QA′]. First it is shown that Q holds with probability at least ⅞. Let Zj,i indicate h1(i)=j and then note the random variables {Zj,i}i∈I are t-wise independent and Zj=Σi∈IZj,i. Also, E[Zj]=|I|/┌k/log k┐≦log k. Setting A=2t and applying Lemma 2,
since E[Zj]≦t. A union bound implies Pr[Q]≧⅞.
The proof of Theorem 3 proceeds with an analysis of Pr[Q|Q′], as described below. Let Yi,i′ be a random variable indicating h2(i)=h2(i′) and define the random variable Y=Σ(i,i′)∈(I2),h
where the first inequality holds by pairwise independence of h2. Conditioned on Q, |h1−1(j)∩I|≦s for all j so that E[Y|Q]<⅛, implying Pr[Q′|Q]=1−Pr[Y≧1|Q]≧⅞ by Markov's Inequality.
Thus, Pr[QQ′]=Pr[Q]·Pr[Q′|Q]≧(⅞)2>¾, and Theorem 3 is proven.
Remark 1. In step 302, the subroutine twice picks a hash function h: [a]→[b] from an m-wise independent family for some integers m and a≠b (namely h1 and h2). However, known constructions have a=b, with a being a prime power. This is easily circumvented. When an h with unequal domain size a and range size b is desired, the T(h′(x) mod b)+1. The family of such h is still m-wise independent, and by choice of Λ, no range value is more than twice more likely than any other, which suffices for the present invention with a slight worsening of constant factors.
Theorem 4 analyzes the pre-processing and post-processing complexity of the T
Theorem 4. Ignoring the time needed to find the prime Λ in Remark 1, the pre-processing time of the T
Proof of Theorem 4. First, with regard to the pre-processing time of the T can be found in time poly log(C)=poly log(k). After p and g are found, filling in T1, T2 takes O(p)=O(k log k) time, which dominates the pre-processing time. The time to allocate the O(k) counters Xzj is just O(k).
The post-processing work of the T
The full algorithm in the sketching procedure requires, in part, a constant factor approximation to the L1-difference. To obtain this constant factor approximation to the L1-difference, an implementation of the algorithm in Feigenbaum et al. (see Theorem 12 in Feigenbaum et al.) is used with c a constant.
Theorem 5. There is a one-pass streaming algorithm for (1±ε)-approximating the L1-difference using O(ο−2 log(nM)) space with update time O(ε−2 log(nM)), and succeeding with probability at least 19/20.
Remark 2. The update time in Theorem 5 is O(ε−2field(log(nM))), where field(D) is the time to do arithmetic over GF(2D) (not including division). Feigenbaum et al. notes that field(D)=O(D2) naively. It suffices for the purposes of the algorithm in Feigenbaum et al. to work over GF(2D) for the smallest D≧log(nM) such that D=2·379 , in which case a highly explicit irreducible polynomial of degree D over [x] can be used to perform GF(2D) arithmetic in time O(D).
The sketching procedure also makes use of the algorithm presented in Theorem 6.
Theorem 6. Let a, b, c, d, x, r, m be integers fitting in a machine word with m>0 and a, b, c, d ∈ {0, . . . , m−1}. There is an algorithm to calculate |{i: (a·(x+i)+b mod m) ∈ [c, d], 0≦i≦r}| in time O(log(min(a, r))) using O(log(r·m)) space.
In step 402, set ε′=ε/8. In step 404, select a random hash function h: [q]→[q] from a pairwise independent family so that h(x)=ax+b mod q for some prime q ∈ [2 nM,4 nM] and a, b ∈ GF(q).
In step 406, initialize instantiations TLE1, . . . , TLE┌log((e′)
In step 408, in response to detecting a stream update (i, v), let vj be the output of the algorithm from Theorem 6 with inputs a, b as in step 404, c=cj=2└log q┘−j, d=dj=2|log q|−j+1−1, x=(i−1)M+1, r=|ν|−1, and m=q. Step 408 also includes feeding the update (i, sgn(ν)·νj) to TLEj for j=1, . . . , ┌log((ε′)2 nM)┐. Furthermore, step 408 includes generating Rj as the output of TLEj.
In step 410, which is performed in parallel with steps 406 and 408, run an instantiation TLE of the T
In step 412 of
In inquiry step 414, R′ is compared to ┌1/(ε′)2┐. If step 414 determines that R′≦┌1/(ε′)2┐, then the Yes branch of step 414 is taken and R is output in step 416. Following step 416, the process of
Theorem 7. The algorithm L1-DIFF has update time O(log(ε2 nM) log(M/ε)) and the bits of space used is O(ε−2 log(1/ε) log(ε−2 nM)). The pre-processing time used is polylog(nM)+O(ε−2 log(1/ε) log(ε−2 nM)). Time O(ε−2 log(1/ε) log log(1/ε) log log log(1/ε)) is needed for post-processing. The output is (1±ε)L1 with probability at least ⅔.
Proof of Theorem 7. The hash function h requires O(log (nM)) space. There are O(log(ε2 nM)) instantiations of the T
As for running time, in step 406 in
In pre-processing, a prime q must be selected in the desired range, which can be accomplished by picking numbers at random and testing primality. The expected time is polylog(nM). There must be a preparation of h1, h2, T1,T2 and all the T
In post-processing, the estimate R′ from step 412 (see
Correctness is analyzed as follows. Let Q be the event that R′ ∈ [L1/2, L1]. The analysis henceforth conditions on Q occurring, which happens with probability at least 19/20 by Theorem 5.
For the first case, suppose L1≦┌1/(ε′)2┐. Then, TLE computes L1 exactly with probability at least ¾ by Theorem 3, and hence overall outputs L1 exactly with probability at least (19/20)•(¾)>⅔.
Now, suppose L1>┌1/(ε′)2┐. In analyzing this case, it helps to view L1-DIFF as actually computing L0(f′)|{i: f′i≠0}|, where an nM-dimensional vector f′ is being updated as follows: when receiving an update (i, ν) in the stream, conceptually view this update as being |ν| updates ((i−1)M+1, sgn(ν)), . . . , ((i−1)M+|ν|, sgn(ν)) to the vector f′. Here, the vector f′ is initialized to {right arrow over (0)}. Note that at the stream's end, L0(f′)=∥f∥1.
Let f′j denote the vector whose ith entry, i ∈ [nM], is f′i if h(i) ∈ [cj, di], and 0 otherwise. That is, f′j receives stream updates only to items fed to TLEj. For i ∈ [nM], let Xi,j be a random variable indicating h(i) ∈ [cj, dj], and let Xj=Σf′(dj−cj+1)/q=2└log q┘−j/q so that E[Xi,j]=pj. Thus, E[Xj]=pj·L0(f′). Note that ½≦2└log q┘/q≦1. Also note the following inequalities:
By the choice of j=┌log((ε′)2)R′┐ in step 420 of
Let Q′ be the event that |Xj−E[Xj]|≦εE[Xj]. Applying Chebyshev's inequality,
The second inequality holds since h is pairwise independent and Xj is the sum of Bernoulli random variables, implying Var[Xj]=Σi Var[Xi,j]≦Σi E[Xi,j]=E[Xj]. The last ineqnality holds by choice of ε′=ε/8.
Let Q″ be the event that TLEj outputs Xj correctly. Now, conditioned on Q′, Xj≦2(1+ε)/(ε′)2≦4/(ε′)2 since ε≦1. Thus by Theorem 3, Pr[Q″|Q′]≧¾. Overall, noting Pr[Q″|QQ′]=Pr[Q″|Q′] by independence, L1 of the entire stream is computed correctly with probability at least
Pr[Q
Q′{circumflex over (0)}Q″]=PR[Q] Pr[Q′|Q]·Pr[Q″|Q
Q′]≧(19/20)·(15.16)·(¾)>⅔
Thus, Theorem 7 is proven.
The streaming algorithm disclosed herein also gives a sketching procedure because, as long as Alice and Bob share randomness, they can generate the same h, h1, h2, p, g and then separately apply the streaming algorithm to their vectors x, y. The sketch is then just the state of the streaming algorithm's data structures. Since each stream token causes only linear updates to counters, a third party can then take the counters from Bob's sketch and subtract them from Alice's, then do post-processing to recover the estimation of the L1-difference. The running time for Alice and Bob to produce their sketches is the streaming algorithm's pre-processing time, plus n times the update time. The time for the third party to obtain an approximation to ∥x−y∥1 is the time required to combine the sketches, plus the post-processing time. This discussion of the sketching procedure leads to Theorem 8, which is presented below.
Theorem 8. Sharing polylog(nM) randomness, two parties Alice and Bob, holding vectors x, y ∈ {−M, . . . , M}n, respectively, can produce O(ε−2 log(1/ε) log(ε2 nM))-bit sketches s(x), s(y) such that a third party can recover ∥x−y∥1 to within (1±ε) with probability at least ⅔ given only s(x), s(y). Each of Alice and Bob use time O(n log(ε2 nM) log(M/ε)) to produce their sketches. In O(ε−2(log(ε2 nM)+log(1/ε) log log(1/ε) log log log(1/ε))) time, the third party can recover ∥x−y∥1 to within (1±ε).
Note Alice and Bob's running time is always O(n log2(nM)) since ε≧1/√{square root over (n)}.
Memory 504 may comprise any known computer readable storage medium, which is described below. In one embodiment, cache memory elements of memory 504 provide temporary storage of at least some program code (e.g., program code 514, 516, 518) in order to reduce the number of times code must be retrieved from bulk storage while instructions of the program code are carried out. Moreover, similar to CPU 502, memory 504 may reside at a single physical location, comprising one or more types of data storage, or be distributed across a plurality of physical systems in various forms. Further, memory 504 can include data distributed across, for example, a local area network (LAN) or a wide area network (WAN).
I/O interface 506 comprises any system for exchanging information to or from an external source. I/O devices 510 comprise any known type of external device, including a display device (e.g., monitor), keyboard, mouse, printer, speakers, handheld device, facsimile, etc. Bus 508 provides a communication link between each of the components in computer system 102, and may comprise any type of transmission link, including electrical, optical, wireless, etc.
I/O interface 506 also allows computer system 102 to store and retrieve information (e.g., data or program instructions such as program code 514, 516 and 518) from an auxiliary storage device such as computer data storage unit 512 or another computer data storage unit (not shown). Computer data storage unit 512 may comprise any known computer readable storage medium, which is described below. For example, computer data storage unit 512 may be a non-volatile data storage device, such as a magnetic disk drive (i.e., hard disk drive) or an optical disc drive (e.g., a CD-ROM drive which receives a CD-ROM disk).
Memory 504 may include computer program code 514, 516 and 518 that provides the logic for collecting sketches, estimating an L1 difference, and monitoring network traffic, respectively, (e.g., the processes of
As will be appreciated by one skilled in the art, the present invention may be embodied as a system, method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “module” or “system” (e.g., system 100 in
Any combination of one or more computer readable medium(s) (e.g., memory 504 and computer data storage unit 512) may be utilized. The computer readable medium may be a computer readable storage medium or a computer readable signal medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, apparatus, device or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer-readable storage medium includes: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with a system, apparatus, or device for carrying out instructions.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with a system, apparatus, or device for carrying out instructions.
Program code (e.g., program code 514, 516 and/or 518) embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code (e.g., program code 514, 516 and/or 518) for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java®, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. Instructions of the program code may be carried out entirely on a user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server, where the aforementioned user's computer, remote computer and server may be, for example, computer system 102 or another computer system (not shown) having components analogous to the components of computer system 102 included in
Aspects of the present invention are described herein with reference to flowchart illustrations (e.g.,
These computer program instructions may also be stored in a computer readable medium (e.g., memory 504 or computer data storage unit 512) that can direct a computer (e.g., computer system 102), other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer program instructions may also be loaded onto a computer (e.g., computer system 102), other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other devices to produce a computer implemented process such that the instructions which are carried out on the computer, other programmable apparatus, or other devices provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
Any of the components of an embodiment of the present invention can be deployed, managed, serviced, etc. by a service provider that offers to deploy or integrate computing infrastructure with respect to the process of monitoring network traffic by estimating an L1 difference. Thus, an embodiment of the present invention discloses a process for supporting computer infrastructure, comprising integrating, hosting, maintaining and deploying computer-readable code (e.g., program code 514, 516 and 518) into a computer system (e.g., computer system 102), wherein the code in combination with the computer system is capable of performing a process of monitoring network traffic by estimating an L1 difference.
In another embodiment, the invention provides a business method that performs the process steps of the invention on a subscription, advertising and/or fee basis. That is, a service provider, such as a Solution Integrator, can offer to create, maintain, support, etc. a process of monitoring network traffic by estimating an L1 difference. In this case, the service provider can create, maintain, support, etc. a computer infrastructure that performs the process steps of the invention for one or more customers. In return, the service provider can receive payment from the customer(s) under a subscription and/or fee agreement, and/or the service provider can receive payment from the sale of advertising content to one or more third parties.
The flowcharts in
While embodiments of the present invention have been described herein for purposes of illustration, many modifications and changes will become apparent to those skilled in the art. Accordingly, the appended claims are intended to encompass all such modifications and changes as fall within the true spirit and scope of this invention.
