Various network measurement and inference problems can be posed as under-determined systems of equations. In undetermined systems, the number of unknowns in the system exceeds the number of equations that constrain the system. Because network inference problems belong to the general category of problems called inverse problems that are used to estimate and/or infer information about the network from observed measurements. Such inverse problems are known to be often ill-posed given that there are no unique solutions, or, the problems are sensitive to changes in observed data. One example of an under-determined problem for networks is traffic matrix estimation problem. A traffic matrix represents a measure of traffic entering a network from any ingress point and destined to any egress point. Traffic matrices provide useful information for network design, capacity planning, traffic engineering, and the like. Traffic matrices can be estimated at different levels of granularity in a network: between Points-of-Presence (PoPs), routers, links, and IP prefixes. Finer grained traffic matrices provide more insight about the behavior of the network which can be used in different applications. As such, traffic matrices are typically measured or estimated at the router-to-router level.
However, measuring and estimating traffic matrices are both challenging problems. Direct traffic measurement imposes significant costs, including measurement equipment, processing costs, and communication and storage overheads, for the monitoring of the network. Additionally, measurement systems can fail if they use an unreliable transport protocol. Further, legacy network components may not support the flow collection that measurement systems use. Additionally, many networks lack adequate resources for making the implementation of a measurement system impractical.
Certain exemplary embodiments are described in the following detailed description and in reference to the drawings, in which:
In a network with N nodes, e.g., routers, there is an N×N matrix, Zt, with entries representing origin-destination flows of the network averaged over a snapshot of time, [t, t+Δt]. Origin-destination flows represent a flow of data traffic between an origin node and a destination node. Each entry in the matrix, Zt, represents the volume of network traffic (in bytes, packets, or flows) from an origin node, i, to a destination node, j, over a time interval, [t, t+Δt]. In other words, the columns of snapshot Zt form an n=N×(N−1) vector, Xt={xtk}knk=1. The vector, Xt, is referred to herein as the traffic matrix.
The traffic matrix is related to a link load measurement vector Yt={yit}i=1m by Equation 1:
Y
t
=HX
t, (1)
where H is a routing matrix that indicates which links are used by which routes. The structure of the routing matrix depends on the topology of the network, IGP link weights, and routing techniques where the shortest path between each pair of nodes is computed. IGP link weights represent the cost of using particular communication links to forward traffic in a network.
In the routing matrix, rows may represent the links of the network. The columns may represent the origin-destination pairs. The routing matrix, H, may be a sparse matrix of ones and zeros. For example. an element of H, hij will have a value of one, if link, i, belongs to the path associated to origin-destination pair j; otherwise, hij will have a value of zero. The entries of H may have fractional values if the network supports traffic splitting.
Equation 1 is an under-determined and ill-posed, linear system of equations because the number of nodes is typically much smaller than the number of origin-destination flows. Accordingly, traffic matrix estimation is an under-determined and ill-posed linear inverse problem. In embodiments, under-determined problems, such as traffic matrix estimation, are partitioned into several smaller, simpler sub-problems. Each of these sub-problems is solved to generate multiple descriptions of the traffic matrix estimation problem. Solutions to traffic matrix estimation can be based on side information, which is prior or current knowledge about the traffic matrix estimation problem. In embodiments, side information is provided by the multiple descriptions of the traffic matrix. These multiple descriptions are fused together to provide a more accurate traffic matrix estimate than typical approaches.
The routing matrix, H, for the network 100 may be based on ISIS link weights and the Dijkstra Algorithm. The Dijkstra Algorithm is a graph search algorithm for solving a single-source shortest path problem in a graph with non-negative edge path costs.
Embodiments use various deterministic and probabilistic methods for solving each sub-problem. For example, one deterministic approach minimizes the Least Square Error using singular value decomposition (SVD). The local solutions are fused. In other words, a fusion function, e.g., f([XL]), shown in
By dividing a large under-determined linear system of equations, e.g., Equation 1, into independent sub-problems, multiple local views (containing partial entries of the traffic matrix) are generated. These local views are more coherent and may be more informative than the global estimate. Each sub-space produces a more coherent, informative description of the partial traffic matrix, resulting in a more precise estimation of the subset of traffic matrix entries. The accuracy of traffic matrix estimations may also depend on the partitioning and fusion techniques used. Embodiments may merely use link count data to partition and fuse, and not auxiliary measurements, such as SNMP data or NetFlow records. Simple Network Management Protocol (SNMP) is a protocol for managing devices on IP networks. NetFlow records provide a wide variety of information about the traffic in a given flow through the direct observation of the network traffic. Note that, in both centralized and distributed cases, if these auxiliary information are available they can be used to improve the performance of traffic matrix estimation.
By solving these sub-problems in independent sub-spaces (using existing traffic matrix estimation techniques), multiple descriptions of the same traffic matrix, Xt, are produced. These descriptions, which are considered as side information for each other, can be weighted to increase or diminish their contribution to the traffic matrix estimation, depending on the partitioning technique.
At block 306, these descriptions can be fused to enhance the accuracy of traffic matrix estimation. The multiple descriptions can be fused together to solve the original Equation 1, and provide a global view with better accuracy than current techniques. The accuracy depends on the partitioning technique and weights for fusion function used in fusing the multiple descriptions. This method 300 also decreases the computational overhead, enhancing the system's robustness against noise and failures.
The method 300 may be implemented using different data fusion structures depending on where the data is collected, how or where the estimation is performed, and how or where the fusion takes place. In embodiments described herein, traffic matrix estimation may be performed in centralized or distributed manner, depending on the location of the monitors and fusion centers. Embodiments include various algorithms for partitioning the original traffic matrix problem into subspaces. In this way, more coherent multiple descriptions can be generated, and fused together, to reduce overall estimation error.
This method provides a flexible and complementary framework that can be coupled with different existing traffic matrix estimation techniques. By intelligently dividing an under-determined linear system of equations into several reduced-order problems, multiple description of the traffic matrix can be estimated. By fusing these local descriptions, the accuracy of traffic matrix is improved, the computation overhead is reduced, and the robustness of the system against noise and failures is enhanced. This is useful for traffic engineering on large, dynamic networks, where traffic matrix entries are estimated at relatively faster time-scales than small, static networks.
In an embodiment of a system with a decentralized structure (not shown), local monitors on nodes gather a sub-set of the link load measurements (forming sub-spaces), and generate descriptions of the local traffic matrix. These local monitors communicate their local descriptions with one another, which are then combined, at each node, to form a global view of the traffic matrix locally. In embodiments with a decentralized structure, the system may not include the central fusion center 402.
Embodiments also include a heuristic approach for distributed traffic matrix estimation to determine how link counts can be clustered for solving traffic matrix estimation sub-problems at local monitors. Embodiments not only decrease the computational overhead, but enhance robustness against failures in communication networks and monitoring infrastructure.
In an example implementation, traffic matrix estimation error was reduced more than 20% while processing time was reduced by more than 90%. The performance of traffic matrix estimation is also enhanced against noise in link load measurements and sub-space erasure. In this way, embodiments are robust to failures and errors in the network measurement system. This is also true for large-scale networks, e.g., large IP backbone, distributed traffic matrix estimation, distributed data centers, or a cloud environment, where traffic engineering may be performed at much smaller time-scales. Embodiments may also be used to solve other under-determined linear systems of equations, independent of the specific sub-space's traffic matrix estimation technique. Hence, embodiments provide a flexible framework for partitioning and fusing original problem and a complementary approach that is independent of the estimation used for solving individual partitions. In this way, a variety of traffic matrix estimation methods may be used while solving individual partitions.
Embodiments provide a complementary approach for solving traffic matrix estimation, where it is assumed that an appropriate estimation method is used for local and global traffic matrix estimations, based on the input traffic matrix. For example, the least squares error (LSE) method could be used for solving individual partitions. LSE is sensitive to traffic matrix inputs with unusual origin-destination flows. As such, the least square error method may be used for traffic matrix estimation, globally and locally, if the traffic matrix does not contain unusual origin-destination flows that differ in size by a large order of magnitude.
In the original traffic matrix estimation problem, estimating Xt from link load measurements Yt, is performed using Equation 1, where Xt={xtk}nk=1, and Yt={ytk}kmk=1. Here, for simplicity, the subscript, t, is dropped. The least square error solution of this under-determined linear system of equations, i.e., the global estimate, can be computed as shown in Equation 2:
{circumflex over (X)}=H†Y=H
T(HHT)−1Y, (2)
where H† is the pseudoinverse of the routing matrix H defined as H†=HT(HHT)−1. The pseudoinverse of H† can be computed more accurately and efficiently using the Singular Value Decomposition (SVD) of H.
The SVD of H is represented as H=UΣVT where U is an m×m unitary matrix, Σ is an m×n rectangular, diagonal matrix with nonnegative-real diagonal entries, i.e., singular values, and VT is an n×n real unitary matrix. Accordingly, H† is computed as H†=VΣ†UT where Σ† is computed by replacing non-zero diagonal entries of Σ00 by its reciprocal and transposing the resulting matrix. The overall cost of the computing of the SVD of an m×n matrix is O(mn2) flops.
Assume that H is known and this problem is partitioned into L sub-problems. Also, let I denote the set of all indices of link counts (that is, I={1, 2, . . . , m} where m is the number of link counts) and Ii denotes the ith set of disjoint indices of link counts where I=Ui=1LIi and Ii= for i≠j. The set P=Ui=1LIi forms a Partition of I where I is divided into multiple non-overlapping and non-empty sub-sets, covering I. Let J denotes the set of all indicies of origin-destination flows (that is, J={1, 2, . . . , n}, where n is the number of origin-destination flows), and Ji denotes the ith set of indices of origin-destination flows where J=Ui=1LJi. However, the intersection of Ji and Jj may not be empty. It is noted that |Ii|≦m and |Ji|≦n for all i=1, . . . , L (|.| denotes the cardinality of the set). Now, let Yi: ={yk}kεI
Y
i
=H
i
X
i. (3)
The least square error solution of this under-determined sub-problem, i.e., local estimate, is computed using the pseudoinverse of the sub-routing matrix, Hi, as represented in Equation 4:
{circumflex over (X)}
i
†Y
i
=H
i
T(HiHiT)−1Yi. (4)
The pseudoinverse Hi† is also computed using the SVD of Hi. Hi is an mi×ni matrix where mi≦m and ni≦n. Therefore, the computation overhead for finding the least square error solution {circumflex over (X)}i is reduced. In fact, the computation cost in each sub-space is O(mini2) flops; therefore, by solving each sub-problem in parallel, the computation time can be further reduced.
Because each subrouting matrix, Hi, is more compact compared to the global traffic matrix estimation, Equation 4 can provide a more coherent solution in the ith sub-space than could be seen in Equation 2. Further, the condition number of each sub-routing matrix, Hi, is lower than the condition number of routing matrix, H. The condition number represents the ratio of the maximum and minimum singular values of the routing matrix, H, that is,
The condition number indicates how well-behaved is the output in proportion to small changes in the input of the function. Thus, if κi≦κ, then the least square error solution of ith sub-space is more accurate and robust.
The condition numbers of sub-routing matrices, Hi, are reduced in comparison with the condition number of the routing matrix, H. The signal-to-noise ratio of local estimates are higher than the signal-to-noise ratio of the global estimate. In fact, under specific conditions which may be achieved through partitioning, the improvement factor of local sub-spaces is lower than the original global space on average. It is noted that the lower the improvement factor is, the more the signal-to-noise ratio improvement is. Further, the amount of redundancy between descriptions, produced by observing each traffic matrix entries from different sub-spaces, is increased, which leads to generating a better estimate through the fusion process. This redundancy can be evaluated by two measures, that is, the sum of the Number of origin-destination Flows (NoD) observed by different subspaces (Rdn1), and the sum of the ratio
(Rank over origin-destination flow) which, conceptually, represents the contribution of each independent link count into the estimation of each origin-destination flow, represented in Equation 5:
Rdn1=Σi=1LNoDi & Rdn2=Σi=1LRoDi (5)
Thus, if Rdn2>RoD (where
embodiments can provide a more accurate traffic matrix estimation. Having estimates, {circumflex over (X)}i, form L sub-spaces, these local views are conformed and fused together to improve the precision of the estimated traffic matrix, {circumflex over (X)}F. The conformation and fusion processes are performed by applying appropriate weights to each local estimate, and concatenating the resulting estimates, which is represented by Equation 6:
{circumflex over (X)}
F⊕i=1LωiF{circumflex over (X)}i, (6)
where the operator, ⊕, denotes the fusion process of the partitioned problem, that is, combining only origin-destination flows observed by different sub-spaces.
By applying a fusion function, origin-destination flows observed in the local sub-spaces are combined to produce the global traffic matrix estimate {circumflex over (X)}F. The fusion functions may choose the origin-destination flow, xi, from the sub-space with highest inverse condition number, ICN, or RoD, or compute the average of the observed origin-destination flows by different sub-spaces. The averaging process, by itself, can improve the accuracy of the estimation by increasing the signal to noise ratio. It is noted that these fusion functions are also efficient because the computational overhead of these particular functions is reduced in comparison with the computation time of solving Equations 2 and 4.
The effectiveness of the partitioning technique contributes to the improvement in the performance of the traffic matrix estimation. Further, the design of the partition is an optimization problem that depends on the structure of the traffic matrix estimation. In a centralized implementation, described with reference to
The performance of the traffic matrix estimation in accordance with embodiments is further discussed with respect to an example implementation performing traffic matrix estimation for example network 100. As stated previously with respect to
In an example implementation, this problem was partitioned into L=5 sub-problems as shown in Equation 8:
where non-overlapping sets {Ii}i=15 are defined as I1={1, . . . , 7}, I2={8, . . . , 11}, I3={12, . . . , 19}, I4={20, . . . , 37}, I5={38, . . . , 50}. Table 1 indicates the mapping of the various origin-destination flows and link counts into 5 different sub-spaces, SS1-SS5, as compared to the original, i.e., whole space. In Table 1, the origin-destination relation of link counts in different sub-spaces ((i,O.D) denotes (index, Origin, Destination). Given this partition and the routing matrix, H, the corresponding sets {Ji}i=1L may be determined.
Table 2 represents characteristics of the original traffic matrix estimation problem for the network 100, and characteristics of the sub-spaces. The inverse condition number, ICN, of each sub-space is higher than the inverse condition number of the original global problem. The total redundancy using this partition is improved, that is, Rdn1=313 which is 313/182=1.72 times of the original problem and Rdn2>RoD. This table also shows that the improvement factor (IF) of each sub-space is less than one, and thus, the signal to noise ratio is improved by partitioning. Therefore, overall, it is expected that embodiments improve the performance using this partition.
Another advantage of embodiments is the improvement of the robustness of the system against noise in link load measurements, and failures in communication networks and monitoring infrastructure. Noise in link load measurements can be modeled as a White Gaussian Noise (WGN), which is added to link measurement vector Y with different signal to noise ratios. Table 3 shows that embodiments using averaging for a fusion function achieve better improvement in the presence of noise, even at low signal to noise ratios, in comparison to typical approaches for global traffic matrix estimation, such as Equation 2.
Embodiments are robust against failures in the system. However, this robustness depends on the characteristics of sub-space erased from the fusion process. For example, the example network 100 is more sensitive to the erasure of the sub-space with highest rank and largest number of observed origin-destination flows, e.g., sub-space 4 according to Table 2. The robustness against failures can be improved by increasing the number of sub-spaces. Further, in a distributed system, unequal protection strategies can be used to ensure that the descriptions from specific sub-spaces are reported correctly for fusion.
In a centralized embodiment, such as system 400A, link count measurements are communicated to the central fusion center 402, where the multiple descriptions are generated and fused together to estimate the traffic matrix. In such an embodiment, a partition is formed by choosing sub-spaces with the highest value inverse condition numbers. In this inverse condition number based partitioning, the sub-spaces (i.e., {Ii}i=1L) are sequentially chosen to build the partition P=Ui=1LIi with the highest inverse condition number.
Assume there are m link counts and the traffic matrix estimation is partitioned into L sub-problems with K link load measurements (where K∝m/L) in each sub-space. In large-scale networks where L<<m, there is a very large number of different partitiions (M) of the set I={1, . . . , m}; therefore, choosing a partition that provides efficient traffic matrix estimation is a hard problem.
Choosing L, in general, depends on the complexity of traffic matrix estimation technique and the processing gain. An initial estimate for L can be approximated as
where OG denotes the complexity of global traffic matrix estimation and OmaxL is the maximum complexity of traffic matrix estimation in sub-spaces to achieve the desired L0. By considering the complexity of SVD as O(mn2), L0 can be obtained by
where (mn2)max is the maximum complexity of sub-spaces.
In one embodiment, a partition is determined by computing the inverse condition numbers of the possible combinations of rows of H. To partition the traffic matrix, the sub-spaces with highest inverse condition numbers are selected according to Algorithm 1:
In one embodiment, a greedy algorithm is used for determining the partition. In such an embodiment, the greedy algorithm starts from the first row of H and sequentially chooses the row that maximizes the inverse condition number of the sub-matrix formed by a partition there. An example greedy algorithm is shown in Algorithm 2:
It is noted that the inverse condition number of each individual row of the routing matrix, H, is one. This continues to complete the first sub-space with K rows. After removing these K rows from the routing matrix, the algorithm repeats from the beginning. This algorithm can be combined by the method described in Algorithm 1 to successively refine the first K0(K0≦K) elements of each set (depends on m and K) or by design of the last sets of the partition P using Algorithm 1.
In another embodiment, the design of partition P is based on the QR decomposition of the routing matrix, represented as shown in Equation 9:
H=QR=Q
m×m
[R
r×r
11
R
r×(n-r)
12], (9)
with orthonormal matrix Q, upper-triangular matrix R11 and rank(H)=r(=m). For rank deficient matricies, QR decomposition with pivoting, known as QRP, provides a less expensive method for solving linear system of equations. However, QR decomposition may also be less accurate, such as, in comparison with SVD. QRP is also useful for recognizing singularities or rank deficiency. In QRP, at each step of the factorization process, the column of the unfactored part of the routing matrix with largest norm is used as the basis for that step. Accordingly, HPν is factorized as HPν=QR where Pν is a permutation matrix, indicating the pivoting process. This pivoting strategy attempts to produce a well-conditioned R11. Accordingly, the diagonal elements of |R| occur in decreasing order, revealing the linear independence among the rows of the routing matrix. An example QRP algorithm is shown in Algorithm 3:
In QRP based partitioning algorithm, as shown in Algorithm 3, the diagonal entries of |R| (occurring in decreasing order) are used to design an initial partition, P0. For this purpose, a set of diagonal elements of matrix |R| is used to construct a sub-space. Each set includes indicies of K successive diagonal entries of matrix |R|. The partition, P0, is defined as P0=Ui=1LIi where Ii for initial partition P0 can be set as Ii={(i=1)K+1, . . . , iK}. Partition P0 is modified by extending or shrinking the boundaries of sets {Ii}i=1L, to improve performance.
In one embodiment, the routing matrix H is full row-rank. Therefore, rows corresponding to very small values of diag(|R|) are removed. This scenario typically results when the routing matrix is not sparse. In such a scenario, the structure of the network and routing protocol (represented by the routing matrix) can be used to improve performance via column-wise partitioning. In column-wise partitioning, the cross-correlation between origin-destination flows are used for partitioning.
In a distributed environment, such as system 400B, link measurements are distributed between local nodes, and multiple descriptions of the traffic matrix are computed at local nodes. These local descriptions are communicated to the central fusion center 402 and fused together to create the estimate of the traffic matrix. In this case, producing multiple descriptions is a design problem involving several factors. Among these factors are the communication cost and delay, which impact the design. Communication cost and delay depend on several factors, such as, the bandwidth of links and the distance between nodes. The communication cost and delay represent the cost and delay of transferring all link counts and local descriptions to the local monitors or central fusion center 402. In an example implementation, the nodes with largest degree, i.e., the largest number of edges incident to that node, have less communication cost and delay.
Embodiments may be used to improve the performance of estimation in linear inverse problems in various applications, such as network loss inference, and localization in sensor networks. In these applications, a sensing matrix, H, linearly relates an observation vector, Y, and an unknown vector, X, through an under-determined linear system of equations, e.g., Y=HX (Equation 1, where t is dropped for simplicity).
The system 800 may represent the centralized system 400A, or the distributed system 400B, and includes a node 802, in communication with other nodes 804, over a network 806. The node 802 may include a processor 808, which may be connected through a bus 810 to a display 812, a keyboard 814, an input device 816, and an output device, such as a printer 818. The input devices 816 may include devices such as a mouse or touch screen. The server node 802 may also be connected through the bus 810 to a network interface card 820. The network interface card 820 may connect the server 802 to the network 806. The network 806 may be a local area network, a wide area network, such as the Internet, or another network configuration. The network 806 may include routers, switches, modems, or any other kind of interface device used for interconnection. In one example, the network 806 may be the Internet.
The node 802 may have other units operatively coupled to the processor 812 through the bus 810. These units may include non-transitory, computer-readable storage media, such as storage 822. The storage 822 may include media for the long-term storage of operating software and data, such as hard drives. The storage 822 may also include other types of non-transitory, computer-readable media, such as read-only memory and random access memory.
The storage 822 may include the machine readable instructions used in examples of the present techniques. In an example, the storage 822 may include a global solver 824. The global solver 824 may, in a centralized system, partition an under-determined problem into several sub-problems of a reduced order. The nodes 804 may each include a local solver 826 to solve one or more of the sub-problems, generating local solutions. The global solver 824 may fuse the local solutions to provide a global solution to the under-determined problem.
In a distributed system, the global solver 824 may determine a clustering of the nodes 804 to generate the local solutions. In such a system, the nodes 804 collect measurement statistics. The local solver 826 produces the local estimate, and fuses local estimates from the other nodes 804 to generate the global solution.
When read and executed by a processor 902, the instructions stored on the machine-readable medium 900 are adapted to cause the processor 902 to solve an under-determined linear problem. The medium 900 includes partitioning method 906, a fusing method 908, and a traffic matrix estimation method 910.
The partitioning method 906 partitions an under-determined problem into several sub-problems of a reduced order. The traffic matrix estimation method 910 produces local solutions of the sub-problems, and the fusing method 908 fuses the local solutions to produce a global solution.