Patent Grant
11463352
This disclosure generally relates to networking systems and, in particular, to analysis of perturbation of links and/or flows in a network, and to network manipulation.
The problem of congestion control is a widely studied areas in data networks. After the earliest congestion control algorithms for the Internet were developed and implemented as part of the TCP protocol, a more than three-decade long period of intense research continued. This has resulted in a long list of congestion control algorithms, including the BBR algorithm recently proposed by Google. Many of these algorithms are based on the belief that the performance of a flow is solely characterized by the state of its bottleneck. The problem of traffic engineering (TE) has also been widely studied and continues to be a very active area of research and development. For instance, Google provides Bandwidth Enforcer (BwE), a centralized bandwidth allocation infrastructure for wide area networking that targets high network utilization.
Bottleneck links in congestion-controlled networks do not operate as independent resources, however. For instance, Mathis equation does not take into account the system-wide properties of a network, including its topology, the routing and the interactions between flows. In reality, bottleneck links generally operate according to a bottleneck structure described herein that can reveal the interactions of bottleneck links, and the system-wide ripple effects caused by perturbations in the network. Techniques using the bottleneck structure, such as the GradientGraph method described below, can addresses a gap in the analysis performed by the conventional techniques, and can provide an alternative methodology to estimate network flow throughput.
Specifically, we present a quantitative technique for expressing bottleneck structures, a mathematical and engineering framework based on a family of polynomial-time algorithms that can be used to reason and identify optimized solutions in a wide variety of networking problems, including network design, capacity planning, flow control and routing. For each of these applications, we present examples and experiments to demonstrate how bottleneck structures can be practically used to design and optimize data networks. Various techniques described herein take into account the system-wide properties of a network, including its topology, the routing and the interactions between flows, and can numerically estimate flow throughput.
The bottleneck structure of a network can be represented qualitatively, via a bottleneck precedence graph (BPG), a structure that organizes the relationships among links. Techniques disclosed herein feature an enhanced analysis of a bottleneck structure that takes into account the relationships among flows and links, not just links, providing a more comprehensive view of the network or a system modeled as a network. As such, embodiments of this technique may provide a framework to quantify the interactions among flows and links, resulting in a new class of algorithms to optimize network performance.
Accordingly, in one aspect a method is provided for jointly performing routing and congestion control in a system modeled as a network. The method comprising performing by a processor the steps of: receiving, for a system modeled as a network, a network topology, and receiving a request for routing a flow between a source node and a destination node. The network topology may include several nodes, a number of links between respective pairs of nodes, and a number of flows via the several links. The method also includes the step of selecting a path for the flow based on a maximum achievable flow rate therefor. The maximum achievable flow rate is derived by generating a bottleneck structure of the network that is modified to have the network topology and the flow.
The flow may include a new flow or an existing flow. In the latter case, the method may further include removing the existing flow from the several flows in the network topology. The removing step is performed before the selecting step. Each link in the number of links may represent a respective availability of a system resource. A respective flow rate associated with each flow in the number of flows may represent a respective utilization of one or more system resources. The selected path may be different from a shortest hop path between the source node and the destination node.
In some embodiments, the bottleneck structure includes several link elements and several flow elements. A dependence from a first link element to a first flow element may indicate that a first flow corresponding to the first flow element is bottlenecked at a first link corresponding to the first link element. A dependance from a second flow element to a second link element may indicate that a second flow corresponding to the second flow element traverses a second link corresponding to the second link element.
In some embodiments, the method further includes selecting, from the several flows, a flow to be accelerated, and determining, by traversing the bottleneck structure, a target flow associated with a positive flow gradient. The these embodiments, the method also includes computing a leap and a fold for the target flow. The leap may represent a maximum perturbation of flow rate of the target flow while maintaining the relative order of the several links according to their respective fair shares. The method may also include reducing flow rate of the target flow using a traffic shaper by a factor up to the leap, and increasing flow rate of the flow to be accelerated up to a product of the leap and a gradient of the flow to be accelerated.
The factor may be selected to preserve completion time of slowest of the several flows. The method may further include repeating the determining, computing, reducing, and increasing steps with respect to a different target flow. The system modeled as a network may include a system modeled as: a data network, a transportation network, an energy distribution network, a fluidic network, or a biological network.
In another aspect, an apparatus is provided for jointly performing routing and congestion control in a system modeled as a network. The apparatus includes a first processor and a first memory in electrical communication with the first processor. The first memory includes instructions that, when executed by a processing unit that includes one or more computing units, where one of such computing units may include the first processor or a second processor, and where the processing unit is in electronic communication with a memory module that includes the first memory or a second memory, program the processing unit to: receive, for a system modeled as a network, a network topology, and a request for routing a flow between a source node and a destination node.
The network topology may include several nodes, a number of links between respective pairs of nodes, and a number of flows via the several links. The instructions also program the processing unit to select a path for the flow based on a maximum achievable flow rate therefor. The maximum achievable flow rate is derived by generating a bottleneck structure of the network modified to have the network topology and the flow. In various embodiments, the instructions can program the processing unit to perform one or more of the method steps described above.
The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.
The present disclosure will become more apparent in view of the attached drawings and accompanying detailed description. The embodiments depicted therein are provided by way of example, not by way of limitation, wherein like reference numerals/labels generally refer to the same or similar elements. In different drawings, the same or similar elements may be referenced using different reference numerals/labels, however. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating aspects of the invention. In the drawings:
Research on the problem of congestion control for data networks is generally based on the principle that the performance of a flow is solely determined by the state of its bottleneck link. This view was presented in one of the earliest congestion control algorithm. This helped the Internet recover from congestion collapse in 1988, and this view persisted throughout the more than 30 years of research and development that followed, including Google's new BBR algorithm. While it is generally true that a flow's performance is limited by the state of its bottleneck link, we describe a deeper view of network behavior, describing how bottlenecks interact with each other through a latent structure—called the bottleneck structure—that depends on the topological, routing, and flow control properties of the network. This latent structure explains how the performance of one bottleneck can affect other bottlenecks, and provides a framework to understand how perturbations in the capacity of a link or the rate of a flow propagate through a network, affecting other links and flows.
A related structure is described in co-pending U.S. patent application Ser. No. 16/580,718, (the “'718 application) titled “Systems and Methods for Quality of Service (QoS) Based Management of Bottlenecks and Flows in Networks,” filed on Sep. 24, 2019, which is incorporated herein by reference. The '718 application generally describes qualitative properties of the bottleneck precedence graph (BPG), a structure that analyzes the relationships among links. The '718 application is incorporated herein by reference in its entirety.
In the discussion below we present a quantitative theory of bottleneck structures (QTBS), a mathematical framework that yields a set of polynomial time and/or memory-efficient algorithms for quantifying the ripple effects of perturbations in a network. Perturbations can either be unintentional (such as the effect of a link failure or the sudden arrival of a large flow in a network) or intentional (such as the upgrade of a network link to a higher capacity or the modification of a route with the goal of optimizing performance). With QTBS, a network operator can quantify the effect of such perturbations and use this information to optimize network performance.
The techniques described herein are generally applicable to networks that transport commodity flows and also to systems that can be modeled as networks. In addition to communication networks, examples include (but are not limited to) vehicle networks, energy networks, fluidic networks, and biological networks. For example, the problem of vehicle networks generally involves identifying optimized designs of the road system that allows for a maximal amount of vehicles that can circulate through the network without congesting it or, similarly, minimizing the level of congestion for a given amount of circulating vehicles. In this case, vehicles are analogous to packets in a data network, while flows correspond to the set of vehicles going from location A to location B at a given time that follow the same path.
The capacity planning techniques described below can be used to analyze the need to construct a road to mitigate congestion hotspots, compute the right amount of capacity needed for each road segment, and to infer the projected effect on the overall performance of the road system. Similarly, the routing techniques described below can be used to suggest drivers alternative paths to their destination that would yield higher throughput or, equivalently, lower their destination arrival time.
The problem of energy networks generally includes transporting energy from the locations where energy is generated to the locations where it is consumed. For instance, energy can be in the form of electricity carried via the electrical grid. Other examples include fluidic networks, which can carry crude oil, natural gas, water, etc., or biological networks that may carry water, nutrients, etc.
Biological networks, through evolution, may tend to organize themselves in optimized structures that maximize their performance (in terms of transporting nutrients) and/or minimize the transportation costs. For instance, a tree transports sap from the root to its branches and in both directions. The sap transported from the root to its branches and leaves is called xylem, which carries energy and nutrients found from the soil where the tree is planted. The sap transported from the leaves and branches to the root is called phloem, which carries also important nutrients obtained from the biochemical process of photosynthesis performed in the cells of the leaves. In both networks (upward and downward), it is likely that the network transporting the sap performs optimally in terms of minimizing the amount of energy required to transport a given amount of sap. Such optimized designs can be generated for other types of networks, using the bottleneck structures and perturbation propagation based thereon, as discussed below. Biological networks can themselves be optimized based on such analysis.
Certain contributions of this disclosure are as summarized below:
Applications demonstrating the practical implications of QTBS are provided in the areas of routing, capacity planning, and flow control. In each of these applications, we show how QTBS can potentially alter some of the established conventional best practices. Some of our contributions regarding the application of QTBS are listed below:
2.1 Network Model
In their simplest form, networks are systems that can be modeled using two kinds of elements: links, which offer communication resources with a limited capacity; and flows, which make use of such communication resources. We formalize the definition of network as follows:
Definition 2.1 Network. We say that a tuple =
L, F, {cl, ∀l∈L}
is a network if:
Each flow f traverses a subset of links Lf⊂L and, similarly, each link l is traversed by a subset of flows Fl⊂F. We will also adopt the convenient notation f=Lf and l=Fl. That is, a flow is the list of links that it traverses and a link is the list of flows that traverse it. Finally, each flow f transmits data at a rate rf and the capacity constraint Σ∀f∈lrf≤cl must hold for all l∈L.
A core concept upon which our framework resides is the notion of a bottleneck link. Intuitively, a link in a network is a bottleneck if its capacity is fully utilized. Mathematically and in the context of this work, we will use a more subtle definition:
Definition 2.2 Bottleneck link. Let N=L, F, {cl, ∀l∈L}
be a network where each flow f∈F transmits data at a rate rf determined by a congestion control algorithm (e.g., TCP's algorithm). We say that flow f is bottlenecked at link l—equivalently, that link l is a bottleneck to flow f—if and only if:
to denote the left derivative. This subtlety is necessary because a flow can have multiple bottleneck links. In this case, decreasing the capacity of only one bottleneck would affect the rate of the flow, while increasing its capacity would not; thus, the (two-sided) derivative would not exist.
This definition of bottleneck generalizes some of the classic definitions found in the literature, while differing from them in that it focuses on the notion of perturbation, mathematically expressed as a derivative of a flow rate with respect to the capacity of a link,
(As an example to illustrate our definition of bottleneck is relatively flexible, in Section 7.1 we show that it corresponds to a generalization of the classic max-min definition.) The general character of the bottleneck definition used in various embodiments described herein is relevant in that it makes our framework applicable not just to specific rate allocation assignments (e.g., max-min, proportional fairness, etc.) or to specific congestion control algorithms (e.g., BBR, Cubic, Reno, etc.), but to any class of congestion control solutions, such as those available in today's networks and those may be developed subsequently, provided that the two conditions in Definition 2.2 hold.
We complete the description of the network model introducing the concept of fair share:
Definition 2.3 Fair share of a link. Let N=L, F, {cl, ∀l∈L}
be a network. The fair share sl of a link l∈L is defined as the rate of the flows that are bottlenecked at such link.
The flows bottlenecked at a link may all have the same rate that may be the same as the faire share of the link. As used throughout the discussion below, the concept of link fair share is dual to the concept of flow rate. That is, all the mathematical properties that are applicable to the rate of a flow, are also applicable to the fair share of a link.
2.2 The Gradient Graph
Our objective is to derive a mathematical framework capable of quantifying the effects that perturbations on links and flows exert on each other. Because the bottleneck structure described in U.S. patent application Ser. No. 14/580,718 considers only the effects between bottleneck links, we need a generalization of such structure that can also describe the effects of perturbations on flows. We refer to this data structure as the gradient graph, formally defined as follows (the name of this graph derives from the fact that perturbations can mathematically be expressed as derivatives or, more generically, as gradients):
Definition 2.4A Gradient graph. The gradient graph is a digraph such that:
We may also employ a variation of the Definition 2.4A as:
Definition 2.4B Gradient graph. The gradient graph is a digraph such that:
By way of notation, in the discussion below we will use the terms gradient graph and bottleneck structure indistinguishably. Intuitively, a gradient graph describes how perturbations on links and flows propagate through a network as follows. A directed edge from a link l to a flow f indicates that flow f is bottlenecked at link l (Condition 2(a) in Definitions 2.4A and 2.4B). A directed edge from a flow f to a link l indicates that flow f traverses but is not bottlenecked at link l (Condition 2(b) in Definition 2.4A), and a bidirectional edge from a flow f to a link l indicates that flow f traverses (and is bottlenecked at) link l (Condition 2(b) in Definition 2.4B).
From Definition 2.2, this necessarily implies that a perturbation in the capacity of link l will cause a change on the transmission rate of flow f,
A change in the value of rf, in turn, creates a perturbation that propagates to all the other links traversed by flow f, following the direction of those edges departing from flow f and arriving at such links (Conditions 2(b) in Definitions 2.4A or 2.4B). This basic process of (1) inducing a perturbation in a vertex in a graph (either in a link or a flow vertex) followed by (2) propagations in the departing edges of the vertex, creates a ripple effect in the bottleneck structure, terminating at the leaves of the gradient graph.
The utility of our definition of gradient graph as a data structure for understanding network performance is captured in the following theorem.
Theorem 2.5 Propagation of Network Perturbations.
Let x, y∈L∪F be a pair of links or flows in the network. Then a perturbation in the capacity cx (for x∈L) or transmission rate rx (for x∈F) of x will affect the fair share sy (for y∈L) or transmission rate ry (for y∈F) of y if only if there exists a directed path from x to y in the gradient graph.
Intuitively, the gradient graph of a network describes how perturbations in link capacities and flow transmission rates propagate through the network. Imagine that flow f is bottlenecked at link l. From Definition 2.2, this necessarily implies that a perturbation in the capacity of link l will cause a change on the transmission rate of flow f,
This is reflected in the gradient graph by the presence of a directed edge from a link l to a flow f (Condition 2a in Definitions 2.4A and 2.5B). A change in the value of rf, in turn, affects all the other links traversed by flow f. This is reflected by the directed edges from f to the links it traverses (e.g., Condition 2b in Definition 2.4B). This basic process of (1) inducing a perturbation in a vertex (either in a link or a flow vertex) followed by (2) propagating the effects of the perturbation along the departing edges of the vertex creates a ripple effect in the bottleneck structure as described in Theorem 2.5. Leveraging Theorem 2.5, we are now in a position to formally define the regions of influence of a data network.
Definition 2.6 Regions of influence in a data network. We define the region of influence of a link or flow x, denoted as R(x), as the set of links and flows y that are reachable from x in the gradient graph.
In the case of the region of influence of a link l, the other links and flows are affected by a perturbation in the capacity cl of link l, according to Theorem 2.5. Similarly, in the case of the region of influence of a flow f, the set of links and other flows are affected by a perturbation in the transmission rate rf of flow f, according to Theorem 2.5.
From Theorem 2.5, we know that the region of influence of a link (or a flow) corresponds to its descendants in the gradient graph. The region of influence is an important concept in network performance analysis and optimization because it describes what parts of a network are affected by perturbations in the performance of a link or a flow. In Section 2.3, it is discussed how such influences can be quantified using the concept of link and flow gradient.
We now introduce the GradientGraph (Algorithm 1A,
Lemma 2.7A states the time complexity of the GradientGraph algorithm:
Lemma 2.7A Time complexity of the GradientGraph algorithm. The time complexity of running GradientGraph( ) is O(H·|L|2+|L|·|F|), where H is the maximum number of links traversed by any flow.
Once this link is selected, all unresolved flows remaining in the network that traverse it are resolved. That is, their rates are set to the fair share of the link (line 12) and they are added to the set of vertices of the gradient graph V (line 13). In addition, directed edges are added in the gradient graph between the link and all the flows bottlenecked at it (line 10) and from each of these flows to the other links that they traverse (line 15). Lines 16-17-18 update the available capacity of the link, its fair share, and the position of the link in the min-heap according to the new fair share. Finally, the link itself is also added as a vertex in the gradient graph (line 22). This iterative process may be repeated until all flows have been added as vertices in the gradient graph (line 7). The algorithm returns the gradient graph G, the fair share of each link {sl, ∀l∈L} and the rate of each flow {rf, ∀f∈F}.
Lemma 2.7B provides the run-time complexity of this embodiment of the GradientGraph( ) algorithm:
Lemma 2.7B. Time complexity of GradientGraph( ) The time complexity of running GradientGraph( ) is O(|L| log |L|·H), where H is the maximum number of flows that traverse a single link.
The GradientGraph is memory efficient, as well. In particular, various embodiments of the GradientGraph include a respective vertex for each link and a respective vertex for each flow. As such, the number of vertices in a GradientGraph is O(|L|+|F|). The edges in the graph from a link vertex to one or more flow vertices do not include, however, an edge to each and every flow vertex where that flow vertex represents a flow traversing the link corresponding to the link vertex. Rather, edges exist from a link vertex to a flow vertex only if, as described above, a flow corresponding to that flow vertex is bottlenecked at the link corresponding to the link vertex. This minimizes the total number of edges in various embodiments and implementations of GradientGraph.
Since the memory required to construct a GradientGraph is a function of (e.g., proportional to the total number of vertices and the total number of edges, the identification of the bottleneck structure facilitates efficient memory allocation in various embodiments. Specifically, in some cases, the memory to be allocated can be a function of the total number of link vertices to flow vertices edges, denoted (|Ebl→f|) where |Ebl→f| is a sum of the number of bottlenecked flows at each link. The required memory may be proportional to O(|L|+|F|+|E|), where the set {E} includes the set of edges from flow vertices to link vertices, denoted {Ef→l} and the set of edges from link vertices to flow vertices corresponding to bottlenecked flows, denoted {El→f}. In some cases, the total number of flows bottlenecked at a link l is less than the total number of flows traversing the link l, minimizing the number of edges |El→f|.
Since, for one or more links, all flows traversing such links may not be bottlenecked at those respective links, the total number of link-to-flow edges (or the total number of bidirectional link-to-flow edges) that are required may be minimized compared to a network graph structure having, for each link, and edge from a corresponding link vertex to vertices corresponding to all flows traversing the link. This can facilitate a memory efficient storage of the gradient graph. Thus, the derivation of the bottleneck structure can minimize the memory required to store and manipulate such a structure, in various embodiments.
2.3 Link and Flow Gradients
In this section, we focus on the problem of quantifying the ripple effects created by perturbations in a network. Because networks include links and flows, generally there are two possible causes of perturbations: (1) those originating from changes in the capacity of a link and (2) those originating from changes in the rate of a flow. When such changes occur, the congestion control algorithm typically adjusts its allocation of bandwidth to the flows so as to maintain two objectives: (1) maximizing network utilization while (2) ensuring fairness among competing flows. The congestion control algorithm acts like a function mapping network conditions (including its topology, link capacities, and flow paths) to rate allocations. Large changes in any of these inputs can have complicated ripple effects on the flow rates, but for sufficiently small changes, the bandwidth allocation function is linear. Technically, it is piecewise linear, like the absolute value function, so picking a linear function that locally approximates it requires knowing the direction of the change. This local linearity property is used to form the concept of link and flow gradients:
Definition 2.8 Link and flow gradients. Let N=L, F, {cl, ∀l∈L}
be a network. We define:
Intuitively, the gradient of a link measures the impact that a fluctuation on the capacity of a link has on other links or flows. In real networks, this corresponds to the scenario of physically upgrading a link or, in programmable networks, logically modifying the capacity of a virtual link. Thus, link gradients can generally be used to resolve network design and capacity planning problems. Similarly, the gradient of a flow measures the impact that a fluctuation on its rate has on a link or another flow. For instance, this scenario corresponds to the case of traffic shaping a flow to alter its transmission rate or changing the route of a flow—which can be seen as dropping the rate of that flow down to zero and adding a new flow on a different path. Thus, flow gradients can generally be used to resolve traffic engineering problems. (In Section 3 applications in real networks that illustrate each of these scenarios are provided.)
Before describing how link and flow gradients can be efficiently computed using the gradient graph, we introduce the concept of flow drift:
Definition 2.9 Drift. Let N=L, F, {cl, ∀l∈L}
be a network and assume
g, {sl, ∀l∈L}, {rf, ∀f∈F}
is the output of GradientGraph(N) (Algorithms 1A or 1B). Let δ be an infinitesimally small perturbation performed on the capacity of a link l*∈L (equivalently, on the rate of a flow f*∈F). Let also sl+Δl and rf+Δf be the fair share of any link l∈L and the rate of any flow f∈F, respectively, after the perturbation δ has propagated through the network. We will call Δl and Δf the drift of a link l and a flow f, respectively, associated with perturbation δ.
Intuitively, the drift corresponds to the change of performance experienced by a link or a flow when another link or flow is perturbed. With reference to
The thrust of the algorithm is as follows. For all l∈L, let Δl be the change in the fair share rate of link l. For all f∈F, let Δf be the change in the rate of flow f. We call these variables the “drifts” caused by a perturbation. Before the perturbation, Δl=Δf=0 for all links and flows. To begin the algorithm, we make an infinitesimally small perturbation in the independent variable (the one in the “denominator” of the derivative) that can be positive or negative. If the independent variable x is a flow f, we set Δf=δ (line 2). If it is a link l, and Sl is the set of direct successors of node l in the gradient graph, we set Δl=δ Sl (line 3). This is done since, by definition of the gradient graph, Sl is the number of flows bottlenecked at l and the change in l's capacity will be distributed evenly among these flows. To determine how this perturbation propagates to the rest of the network, we follow all directed paths from that vertex and update the drifts according to the following two invariants:
Gradient graph invariants. Let N=L, F, {cl, ∀l∈L}
be a network and let g be its gradient graph. Let δ be an infinitesimally small perturbation performed on the capacity of a link l*∈L (equivalently, on the rate of a flow f*∈F) and let Δl and Δf be the drifts caused on a link l∈L and a flow f∈F, respectively, by such a perturbation. Assume also that the perturbation propagates according to the gradient graph g by starting on the link vertex l* (equivalently, on the flow vertex f*) and following all possible directed paths that depart from it, while maintaining the following invariants at each traversed vertex:
Invariant 1: Flow Equation. A flow's drift Δf equals the minimum drift of its bottleneck links. That is, Δf=
where Pf is the set of links visited directly before flow vertex f on a path from the starting vertex x (the predecessors in the graph).
Invariant 2: Link Equation. A link's drift Δl is the negative of the flow drifts entering its vertex, divided by the number of flow drifts leaving it. That is, Δl=−Σf∈P
Finally, the derivative of a given variable with respect to the independent variable that we perturbed can be calculated by dividing its drift by δ. In particular, assume the capacity of link l is the independent variable that we perturbed and let the rate of flow f be the dependent variable in which we want to measure the effect of this perturbation. Then, ∂rf/∂cl=Δf/δ.
Since the flow and link equations lie at the heart of the algorithm, we provide some further explanation. Invariant 1 ensures that the capacity limits are respected and the network's resources are not wasted. Each flow must use exactly the amount of bandwidth allocated by its bottleneck link, so if the bottleneck's fair share changes, the flow's rate must change too. It also ensures fairness, since each flow bottlenecked at a certain link will experience the same drift. Invariant 2 ensures that capacity is neither created nor destroyed through the process of propagating a perturbation, except at the link whose capacity was initially perturbed. If a link's predecessors are using less bandwidth than before, then the savings must be redistributed evenly among the other flows that traverse the link.
Let also g′ be the gradient graph of the resulting network after the perturbation has propagated. Then, if g=g′, the link and flow gradients can be computed as follows:
This states that if the gradient graph does not change its structure upon a small perturbation (i.e., g=g′) and the two invariants are preserved, then such a perturbation can be measured directly from the graph. The first invariant is a capacity feasibility constraint, ensuring that a flow's drift is limited by its most constrained bottleneck. The second invariant ensures that (1) the sum of the drifts arriving to and departing from a link vertex are equal to zero and (2) the drifts departing from a link vertex are equally distributed. Intuitively, this is needed to preserve the congestion control algorithm's objective to maximize network utilization while ensuring fairness among all flows.
this generates a drift on this link of
Applying again the flow equation on f2, we obtain the flow drift
Thus, the gradient of link ll with respect to flow f2 is
It should be noted that it is feasible for a link or flow gradient to have a value larger than 1. Such gradients are of interest because they mean that an initial perturbation of one unit at some location of a network, generates a perturbation at another location of more than one unit. For instance, a gradient of the form ∇f·(f)>1 implies that reducing the rate of flow f* by one unit creates a perturbation that results in an increase on the rate of flow f by more than one unit, thus creating a multiplicative effect. Such gradients can be used to identify arbitrage situations—e.g., configurations of the network that increase the total flow of a network. Because of their relevance, we will use the term power gradient to refer to such effect:
Definition 2.10 Power gradient. Let N=L, F, {cl, ∀l∈L}
be a network and let δ be an infinitesimally small perturbation performed on a flow or link x∈L∪F, producing a drift Δy, for all y∈L∪F. If Δy>δ, equivalently ∇x(y)>1, then we will say that ∇x(y) is a power gradient. In Section 3, we provide examples of power gradients. For now, we conclude this section stating a property of boundedness that all gradients in congestion-controlled networks satisfy:
Property 1 Gradient bound. Let N=L, T, {cl, ∀l∈L}
be a network and let g be its gradient graph. Let δ be an infinitesimally small perturbation performed on a flow or link x∈L∪F, producing a drift Δy, for all y∈L∪F. Then,
where D(X) is the diameter function of a graph X and d is the maximum indegree and outdegree of any vertex in the graph.
2.4 Leaps and Folds
The concepts of link and flow gradients introduced in the previous section provide a methodology to measure the effect of perturbations on a network that are small enough (infinitesimally small) to avoid a structural change in the gradient graph. In this section, we introduce the concepts of leap and fold, which allow us to generalize the framework to measure perturbations of arbitrary sizes. Two simple and intuitive examples of such kind of perturbations found in real networks include: a link failure, which corresponds to the case its capacity goes down to zero; or the re-routing of a flow, which corresponds to the case its rate goes down to zero and a new flow is initiated.
We know that if a perturbation in the network is significant enough to modify the structure of the gradient graph (i.e., g≠g′), then the link and flow equations (
Definition 2.11 Gradient leap. Let ∇x(y) be a gradient resulting from an infinitesimally small perturbation δ on a link or flow x, where x, y∈L∪F. Suppose that we intensify such a perturbation by a factor k, resulting in an actual perturbation of λ=k·δ, for some k>0. Further, assume that k is the largest possible value that keeps the structure of the gradient graph invariant upon perturbation λ. Then, we will say that λ is the leap of gradient ∇x(y).
The following lemma shows the existence of folds in the bottleneck structure when its corresponding network is reconfigured according to the direction indicated by a gradient and by an amount equal to its leap:
Lemma 2.12 Folding links. Let N=L, F, {cl, ∀l∈L}
be a network and let g be its gradient graph. Let λ be the leap of a gradient ∇x(y), for some x, y∈L∪F. Then, there exist at least two links l and l′ such that: (1) for some f∈F, there is a directed path in g of the form l→f→l′; and (2) sl=sl′ after the perturbation has propagated through the network.
Intuitively, the above lemma states that when a perturbation is large enough to change the structure of the gradient graph, such structural change involves two links l and l′ directly connected via a flow f (i.e., forming a path l→f→l′) that have their fair shares collapse on each other (s′l=s′l′) after the perturbation has propagated. The faire shares can be substantially or approximately equal (e.g., the difference between the faire shares can be zero or less than a specified threshold, e.g., 10%, 5%, 2%, 1%, or even less of the fair share of one of the links.) Graphically, this corresponds to the folding of two consecutive levels in the bottleneck structure. We can now formalize the definition of fold as follows.
Definition 2.13 Fold of a gradient. Let λ be the leap of a gradient ∇x(y), for some x, y∈L∪F, and let l and l′ be two links that fold once the perturbation λ has propagated through the network (note that from the discussion above, such links must exist). We will refer to the tuple (l, l′) as a fold of gradient ∇x(y).
The concept of leap and fold is relevant in that it enables a methodology to efficiently travel along the solution space defined by the bottleneck structure, towards reaching a certain performance objective is achieved. Specifically, for some x, y∈L∪F, if x is perturbed negatively so as to benefit another flow or link in the network, but only up to the leap of x, i.e., λ, the negative and positive changes may be balanced. On the other hand, if x is perturbed negatively by more than its λ, the positive impact of this perturbation on another flow or link would not exceed λ, potentially resulting in degradation of the overall network performance.
We introduce a method/algorithm MinimizeFCT( ) shown in
From the discussion above, we know that the additional traffic shaper changes the structure of the gradient graph, at which point we need to iterate again the procedure (line 1) to recompute the new values of the gradients based on the new structure. This process is repeated iteratively until either no more positive gradients are found or the performance of fs has increased above a given rate target ρ (lines 3 and 4). In the next section, an example is presented demonstrating how embodiments of MinimizeFCT( ) may be used to optimize the performance of a time-bound constrained flow.
To provide the process illustrated using
The next two theorems show that Algorithm 2 is both correct and efficient.
Theorem 2.9. Correctness of ForwardGrad®. Let =
,
, {cl, ∀l∈L}
be a network and let g be the corresponding gradient graph. Let x∈L∪F. After running Algorithm 2, Δs
Theorem 2.10. Time complexity of ForwardGrad( ). Let x∈L∪F. Then Algorithm 2 finds the gradients of all links and flows in the network with respect to x in time O(|R(x))|·log(|R(x)|)
To conclude and complement this section, we state an upper bound on the value of the gradients:
Property 2.11. Gradient bound. Let =
,
, {cl, ∀l∈L}
be a network and let g be its gradient graph. Let δ be an infinitesimally small perturbation performed on a flow or link x∈L∪F, producing a drift Δy, for all y∈L∪F. Then,
where D(X) is the diameter of a graph X and d is the maximum indegree and outdegree of any vertex in the graph.
Because bottleneck structures are a fundamental property intrinsic to any congestion-controlled data network, its applications span a variety of networking problems. In this section, our goal is to present examples and experiments illustrating how QTBS can be used to resolve some of these problems. We will see that in each of them, the framework is able to provide new insights into one or more operational aspects of a network. The examples presented in this section are not exhaustive, but only illustrative. To help organize the applications, we divide them in two main classes: traffic engineering and capacity planning. For each of these classes, we provide specific examples of problems that relate to applications commonly found in modern production networks.
To experimentally demonstrate that data networks behave qualitatively and quantitatively according to QTBS, we use Mininet-Extensions-Anonymized, a network emulation framework developed by our team that consists of a set of software modules and extensions to Mininet. Leveraging software define networking (SDN), Mininet-Extensions-Anonymized enables the creation and analysis of arbitrary network architectures using real production TCP/IP code, including production-grade implementations of congestion control algorithms such as BBR, Cubic or Reno.
All the experimental results presented in this section are based on Google's BBR congestion control algorithm and Cubic. For each experiment, we used Jain's fairness index as an estimator to measure how closely the predictions of the theory of bottleneck structure model match the experimental results. For all BBR experiments presented in the next sections, this index was above 0.99 accuracy on a scale from 0 to 1, reflecting the strength of QTBS in modeling network behavior.
3.1 Traffic Engineering: Computation of the Highest-Throughput Route
In traditional IP networks, the problems of flow routing and congestion control are separately resolved by following a two-step process: first, a routing protocol (e.g., BGP, OSPF, etc.) is used to determine the path between any two nodes in a network; then, flows are routed according to such paths and their transmission rates are regulated using a congestion control algorithm (e.g., BBR). This layered and disjoint approach is known generally to be scalable but suboptimal because the routing algorithm identifies paths without taking into account the flow transmission rates assigned by the congestion control algorithm. In this section, we use QTBS to resolve the following joint routing and congestion control problem in a scalable manner.
Definition 3.1. Flow-rate maximal routing. Let =
, T, {cl, ∀l∈
}
be a network and suppose that a new flow f arrives. We will say that a routing algorithm is flow-rate maximal if it routes flow f through a path that maximizes its transmission rate rf.
In traditional IP routing, all packets transmitted from a source to a destination node follow the same lowest-cost route. This rigidity leads to the well-known fish problem, whereby certain paths in a network become congested while other paths are underutilized. Various embodiments of the flow-rate maximal algorithm, instead, are able to bypass points of congestion by assigning new flows to the highest-throughput path available given the current usage of the network.
One might mistakenly think that the least congested path can be identified by looking for links with small fair shares (Definition 2.3). However, the placement of a new flow onto a given path will itself alter the state of the network, changing those fair shares and potentially rendering the chosen path sub-optimal. In this section, we show that QTBS can be used to identify the maximal-rate path for a flow while taking into account the perturbations created by the placement of the flow itself, thus solving the flow-rate maximal routing problem.
MaxRatePath( ) (Algorithm 3 shown in L, F, {cl, ∀l∈L}
, the set of routers U, and the source and the destination routers of the flow we intend to route, us and ud. By convention, a link l∈L is identified with the tuple l=(ux, uy) where ux, uy are the two routers connected by link l. The algorithm returns the new flow f, expressed as the set of links it traverses, guaranteeing they form a path from us to ud that yields the maximal rate rf for f.
As the pseudocode shows, MaxRatePath( ) is based on Dijkstra's shortest path algorithm, with routers as vertices and links as edges in the network topology graph. The difference resides in the way the “distance” to a neighboring router u′ is calculated (lines 12-14). In MaxRatePath( ) this value represents not the number of hops on the shortest path from us to u′, but the inverse of the largest possible rate that a flow would experience if it were added on some path from us to u′. That is, the distance to u′ is the smallest possible time needed to send 1 bit of information from us to u′.
Unlike in the standard Dijkstra's algorithm, this value cannot be computed by adding an edge length to du, the distance to a neighbor of u′. Instead, we create a new flow f by extending the optimal path from us to u. So, at each iteration of the algorithm, f takes the path us→ . . . →u→u′ (line 12). We then construct the gradient graph that would correspond to this network if the new flow f were added (line 13). Finally, we use the inverse of the rate assigned to the new flow rf as the distance value (line 14). In the pseudocode, we invoke the GradientGraph( ) algorithm in line 13, reconstructing the gradient graph to include the new flow.
Lemma 3.2. Correctness of the MaxRatePath algorithm. Let N=L, T, {cl, ∀l∈L}
be a network and U the set of its routers. Suppose that f and f′ are two flows not in F that originate at router us and end at router ud. Then f=MaxRatePath(N, U, us, ud) implies rf≥rf′.
To illustrate how we can use QTBS and the MaxRatePath( ) algorithm to compute the highest-throughput path for a given flow, consider the network shown in
Note also that all the top-level flows operate at a lower transmission rate (with all rates at 1.667) than the bottom-level flows (with rates between 2.143 and 3). This in general is a property of all bottleneck structures: flows operating at lower levels of the bottleneck structure have higher transmission rates than those operating at levels above. Under this configuration, suppose that we need to initiate a new flow f25 to transfer a large data set from data center 4 to data center 11. For instance, this flow could correspond to the transmission of a terabyte data set from a data center in the US to another in Europe. Our objective in this exercise is to identify a high-throughput route to minimize the time required to transfer the data.
Because the bottleneck structure reveals the expected transmission rate of a flow based on the path it traverses, we can use QTBS to resolve this problem. In
Note that the presence of this new flow slightly modifies the performance of some of the flows on the first level (flows {f1, f3, f4, f5, f7, f8} experience a rate reduction from 1.667 to 1.429), but it does not modify the performance of the flows operating at the bottom level. This is because, for the given configuration, the new flow only creates a shift in the distribution of bandwidth on the top level, but the total amount of bandwidth used in this level stays constant. (In
While l15→l10 is the shortest path, it is not the path with the highest throughput. To find such a path, we run an embodiment of the MaxRatePath procedure (Algorithm 3) and obtain the solution l16→l8→l19. The resulting bottleneck structure is shown in
In the remainder of this section, we set out to empirically confirm these results. We start by creating the B4 network configuration shown in
This experiment illustrates that using QTBS, it is possible to identify routes that are highly efficient from a congestion control standpoint. Note that this contrasts with traditional approaches that perform traffic engineering by separating the routing and congestion control problems, so that the routing algorithm is unaware of the choices made by the congestion control algorithm and vice versa. We reason that QTBS provides a mathematical framework to connect both problems, identifying routes that are globally efficient from both a topological and a congestion control standpoints.
The above-described technique is not limited to adding new flows to a network. An existing flow may be rerouted using the technique described above. If an existing flow is to be rerouted, the existing flow may be terminated and removed from the network topology. A few flow may then be added between the source and destination of the removed flow, as discussed above.
3.2 Capacity Planning: Design of Optimal Fat-Tree Networks in Data Centers
Fat-trees are generally understood as universally efficient networks in the following sense: for a given network size s, a fat-tree can emulate any other network that can be laid out in that size s with a performance slowdown at most logarithmic in s. This property makes fat-tree topologies highly competitive and is one of the reasons they are so widely used in large-scale data centers and high-performance computing (HPC) networks. In the context of data centers, fat-tree networks are also known as folded-clos or spine-and-leaf networks. In this experiment, we use QTBS to demonstrate that, due to the effects of the congestion control algorithm, there exists an optimal trade-off in the allocation of capacity at the top levels of the fat-tree. Further, we show that the optimal bandwidth allocation on the top level deviates from commonly accepted best practices in the design of full fat-tree networks that tend to equate the amount of bandwidth going up and down the tree at each switch.
Consider the network topology in
f10
f11
f12
We fix the capacity of the leaf links to a value λ (i.e., cl
The focus of our experiment is to use the bottleneck structure analysis to identify optimized choices for the tapering parameter τ. In
The first bottleneck structure (
By looking at the bottleneck structure in
On the other hand, the link gradient of any of the spine links with respect to any of the low-level flows is ∇l(f)=−0.25, for all l∈{l5, l6} and f∈{f1, f4, f9, f12}. That is, an increase by one unit on the capacity of the spine links increases the rate of the top-level flows by 0.125 and decreases the rate of the low-level flows by 0.25. Since the rates of the top and low-level flows are 2.5 and 5, respectively, this means that the two levels will fold at a point where the tapering parameter satisfies the equation 2.5+0.125·τ·λ=5−0.25·τ ·λ, resulting in
and, thus, cl
Note that this value corresponds exactly to the leap of the spine links gradient, and thus can also be programmatically obtained using Algorithm 2 (
What is the effect of increasing the tapering parameter above
This result is shown in
times the capacity of the leaf links.
In summary, for the fat-tree network shown in
should not be used, since the resulting network is just as efficient as a design with
but more costly.
is optimal in that it minimizes the flow completion time of the slowest flow. This should be the preferred design in symmetric workloads that transfer about the same amount of data between any two nodes.
can be used if workloads are asymmetric, identifying the right value of τ that produces the right amount of bandwidth at each level of the bottleneck structure according to the workload.
In the rest of this section, we empirically demonstrate the existence of an optimal fat-tree design at τ=4/3 using Mininet-Extensions-Anonymized configured with the congestion control algorithm BBR.
Each plot shows the transmission rate of all twelve flows as part of the network configuration, with each flow transmitting a total of 64 MB of data. Following the example in Section 3.2.1, the link capacities are set as follows: cl
As predicated by QTBS, the case τ=1 has flows operating at one of two bottleneck levels, close to the rates predicted by the bottleneck structure (2.5 Mbps for the upper-level flows and 5 Mbps for the lower-level flows, see
If we want to maximize the rate of the slowest flow, QTBS tells us that the right tapering parameter value is 4/3. This case is presented in
Note that the existence of an optimal design with a tapering parameter τ=4/3 argues against some of the established conventional best practices in fat-tree networks. For instance, while a full fat-tree (τ=2) is considered generally to be universally efficient, the analysis of its bottleneck structure demonstrates that such design is in general inefficient when flows are regulated by a congestion-control protocol. This is because the fairness and throughput maximization objectives targeted by the congestion control algorithm effectively bends the solution space and, as a result, the optimal fat-tree design deviates from the general full fat-tree configuration. This result has implications in the design of data centers that use fat-tree topologies (also known as folded-Clos). In this section, we have illustrated how QTBS can be used to optimize a simple fat-tree topology for the case of a symmetric workload pattern.
3.3 Traffic Engineering: Accelerating Time-Bound Constrained Flows
Suppose that our goal is to accelerate a flow fs∈F in a network N with the objective that such flow is completed before a certain time-bound requirement or a target time. A common application for the optimization of time-bound constrained flows can be found in research and education networks, where users need to globally share data obtained from their experiments, often involving terabytes or more of information—e.g., when scientists at the European Organization for Nuclear Research (CERN) need to share data with other scientific sites around the world using the LHCONE network. Another common use case can be found in large scale data centers, where massive data backups need to be transferred between sites to ensure redundancy. In this context, suppose the operators are only allowed to sacrifice the performance of a subset of flows F′⊂F{fs}, considered of lower priority than fs. What flows in F′ present an optimal choice to traffic shape so as to accelerate fs? By what amount should the rate of such flows be reduced? And by what amount will flow fs be accelerated?
To illustrate that we can use QTBS to resolve this class of problems, consider the network shown in
We are interested in finding the gradient of a flow in F that has the highest negative value, so that the traffic shaping of such a flow (i.e., the reduction of its rate) creates a maximal positive increase in the rate of f7. We have that flow f4 has the highest negative gradient with a value of −2, yielding an optimal traffic shaping decision. From
We can use the bottleneck structure again to calculate the exact value of the traffic shaper—i.e., the rate reduction applied to flow ff. The core idea is that traffic shaping flow f4 may be an optimal decision as long as the bottleneck structure does not change, since a change in the structure would also imply a change in the gradients. As the rate of flow f4 is reduced, some levels in the bottleneck structure will become further away from each other, while the others will become closer to each other. Thus, the latter set will fold if the rate reduction imposed by the traffic shaper is large enough. The speed at which two links in the bottleneck structure get closer to (or further away from) each other is given by their gradients. In particular, if the traffic shaper reduces the rate of flow f4 by an amount of ρ bps, then two links l and l′ in the bottleneck structure will collide at a value of ρ that satisfies the equation sl−ρ ·∇f
From the bottleneck structure (
Thus, we conclude that to maximally increase the rate of flow f7, an optimal strategy is to decrease the rate of flow f4 by an amount of 0.5 units of bandwidth. The resulting bottleneck structure is presented in
Therefore, we add two new traffic shapers l8 and l9 to throttle the rate of flows f3 and f8, respectively, down from their current rates of 6.875 and 11.25. That is: cl
In summary, a strategy to accelerate the performance of flow f7 includes traffic shaping the rates of flows f3, f4 and f8 down to 1.25, 1.875, and 5.625, respectively. Such a configuration results in a theoretical increase to the rate of flow f7 from 10.25 to 16.875, while ensuring no flow performs at a rate lower than the slowest flow in the initial network configuration. Note that among all the low priority flows in F′, in the above process we opted for not reducing the rate of flow f1. Indeed, the three bottleneck structures (
Thus, the quantitative analysis resulting from the bottleneck structure of the network reveals not only the set of flows that should be traffic shaped, but also the flows that should not be traffic shaped, as doing so would actually hurt the performance of the flow we intend to accelerate. Note that this result challenges some of the established best practices for traffic engineering flows, which include many proposed algorithms that focus on reducing the rate of the heavy-hitter flows to improve high-priority flows. As shown in this example, without taking into account the bottleneck structure of a network, such algorithms may recommend a traffic shaping configuration that actually has the opposite of the intended effect.
To empirically demonstrate the accuracy of QTBS in identifying the set of traffic shapers and their optimal rate, we reproduce the experiments described in this section using Mininet-Extensions-Anonymized.
Table 4 shows the average transmission rate obtained for each of the flows and for each of the three experiments. Next to each experimental rate value, this table also includes the theoretical flow transmission rate according to the bottleneck structure. It can be seen that these values match the transmission rate rf shown next to each flow vertex (gray vertices) from the corresponding bottleneck structures in
The analytical strength of a bottleneck structure stems from its ability to capture the solution-space produced by a congestion-control algorithm taking into account the topological and routing constraints of the network. Based on this concept, we develop a quantitative theory of bottleneck structures (QTBS), a new mathematical framework that allows to optimize congestion-controlled networks by providing very efficient algorithms to compute derivatives on the performance parameters of links and flows. To explore the analytical power of QTBS, we use it to reveal insights in traffic engineering and network design problems that are themselves contributions to the literature. In one experiment, we use QTBS to develop a novel routing algorithm that identifies maximal throughput paths, enabling a scalable methodology to jointly solve the problems of routing and congestion control. In another experiment, we use QTBS to reveal the existence of optimal capacity allocations in the spine links of a fat-tree network that outperform (in cost and/or performance) the traditional full fat-tree network designs found in some large-scale data centers and supercomputers. In a third experiment, we demonstrate how to use bottleneck structures to compute the numerical values of optimal rate settings in traffic shapers to help improve the performance of high-priority flows. We present the concept of bottleneck structures as a promising analytical framework to optimize network performance. In general, this technique can be applied to any system that can be modeled as a network.
The overall network analysis and/or manipulation or control processes described herein begin with the collection of network information including flow information, link information, and topology. The flow information generally includes the identities of flow, the total count of flows, and the rates of the identified flows during a specified observation window, which can be a few minutes, a few hours, a few days, or longer. The link information includes the number of active links, their identities, and their designated and/or maximum capacities during the specified observation window. The network topology includes the network nodes and the links, typically direct links, interconnecting such nodes.
In case of data networks, the nodes may be data centers and/or computing centers, the links include data links, whether cable, wireless, or satellite based, the flow rates may include number of bits, bytes, packets, etc., passing through the links, and link capacities may be expressed in terms of available or allotted bandwidth or bit rate. In case of transportation networks, the nodes can be cities, locations within cities or a metropolitan area, airports, marine ports, etc., the links can be roadways, railways, subway routes, airline routes, marine routes, etc., the flow rates and link capacities can be expressed in terms of the number of passengers or travelers, the number of vehicles, etc.
In case of energy networks, the nodes can be energy generators such as power plants and consumers, such as towns, cities, industrial complexes, shopping centers, etc. The links include energy delivery systems including high-voltage transmission lines, substations, local energy distribution lines, etc. The flow rates and link capacity can be expressed in terms of peak energy demand, average energy demand, etc.
In case of fluidic or biological networks, the nodes can be sources and consumers of material, such as oil, gas, nutrients, blood, etc., and the link capacity can be the sizes of conduits or vessels carrying the fluids or biological materials, the pressure in such conduits or vessels, etc. In some cases, the capacity and/or rate of flow in one or more conduits/vessels can be adjusted by shutting off or pruning other conduits/vessels. The flow rate optimization and/or capacity planning can thus be used to manage or control irrigation systems, fertilizer delivery system, plant/crop disease control systems, etc.
After collecting the required information, the GradientGraph that includes various flow and link gradients is generated using embodiments of Algorithms 1A or 1B (
The effect of this perturbation can be observed on the flow(s) and/or link(s) of interest, and the process may be repeated a specified number of times, until a desired effect (e.g., increase in the rate of a flow of interest) is attained, or a maximum feasible change can be attained. Such iterations may be performed under constraints, such as not permitting the flow rate of any flow below the current minimum or a specified lower-bound rate, maintaining the relative order of the flow rates, allotting at least a specified lower-bound capacity to each link, etc.
It is clear that there are many ways to configure the device and/or system components, interfaces, communication links, and methods described herein. The disclosed methods, devices, and systems can be deployed on convenient processor platforms, including network servers, personal and portable computers, and/or other processing platforms. Other platforms can be contemplated as processing capabilities improve, including personal digital assistants, computerized watches, cellular phones and/or other portable devices. The disclosed methods and systems can be integrated with known network management systems and methods. The disclosed methods and systems can operate as an SNMP agent, and can be configured with the IP address of a remote machine running a conformant management platform. Therefore, the scope of the disclosed methods and systems are not limited by the examples given herein, but can include the full scope of the claims and their legal equivalents.
The methods, devices, and systems described herein are not limited to a particular hardware or software configuration, and may find applicability in many computing or processing environments. The methods, devices, and systems can be implemented in hardware or software, or a combination of hardware and software. The methods, devices, and systems can be implemented in one or more computer programs, where a computer program can be understood to include one or more processor executable instructions. The computer program(s) can execute on one or more programmable processing elements or machines, and can be stored on one or more storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), one or more input devices, and/or one or more output devices. The processing elements/machines thus can access one or more input devices to obtain input data, and can access one or more output devices to communicate output data. The input and/or output devices can include one or more of the following: Random Access Memory (RAM), Redundant Array of Independent Disks (RAID), floppy drive, CD, DVD, magnetic disk, internal hard drive, external hard drive, memory stick, or other storage device capable of being accessed by a processing element as provided herein, where such aforementioned examples are not exhaustive, and are for illustration and not limitation.
The computer program(s) can be implemented using one or more high level procedural or object-oriented programming languages to communicate with a computer system; however, the program(s) can be implemented in assembly or machine language, if desired. The language can be compiled or interpreted. Sets and subsets, in general, include one or more members.
As provided herein, the processor(s) and/or processing elements can thus be embedded in one or more devices that can be operated independently or together in a networked environment, where the network can include, for example, a Local Area Network (LAN), wide area network (WAN), and/or can include an intranet and/or the Internet and/or another network. The network(s) can be wired or wireless or a combination thereof and can use one or more communication protocols to facilitate communication between the different processors/processing elements. The processors can be configured for distributed processing and can utilize, in some embodiments, a client-server model as needed. Accordingly, the methods, devices, and systems can utilize multiple processors and/or processor devices, and the processor/processing element instructions can be divided amongst such single or multiple processor/devices/processing elements.
The device(s) or computer systems that integrate with the processor(s)/processing element(s) can include, for example, a personal computer(s), workstation (e.g., Dell, HP), personal digital assistant (PDA), handheld device such as cellular telephone, laptop, handheld, or another device capable of being integrated with a processor(s) that can operate as provided herein. Accordingly, the devices provided herein are not exhaustive and are provided for illustration and not limitation.
References to “a processor”, or “a processing element,” “the processor,” and “the processing element” can be understood to include one or more microprocessors that can communicate in a stand-alone and/or a distributed environment(s), and can thus can be configured to communicate via wired or wireless communication with other processors, where such one or more processor can be configured to operate on one or more processor/processing elements-controlled devices that can be similar or different devices. Use of such “microprocessor,” “processor,” or “processing element” terminology can thus also be understood to include a central processing unit, an arithmetic logic unit, an application-specific integrated circuit (IC), and/or a task engine, with such examples provided for illustration and not limitation.
Furthermore, references to memory, unless otherwise specified, can include one or more processor-readable and accessible memory elements and/or components that can be internal to the processor-controlled device, external to the processor-controlled device, and/or can be accessed via a wired or wireless network using a variety of communication protocols, and unless otherwise specified, can be arranged to include a combination of external and internal memory devices, where such memory can be contiguous and/or partitioned based on the application. For example, the memory can be a flash drive, a computer disc, CD/DVD, distributed memory, etc. References to structures include links, queues, graphs, trees, and such structures are provided for illustration and not limitation. References herein to instructions or executable instructions, in accordance with the above, can be understood to include programmable hardware.
Although the methods and systems have been described relative to specific embodiments thereof, they are not so limited. As such, many modifications and variations may become apparent in light of the above teachings. Many additional changes in the details, materials, and arrangement of parts, herein described and illustrated, can be made by those skilled in the art. Accordingly, it will be understood that the methods, devices, and systems provided herein are not to be limited to the embodiments disclosed herein, can include practices otherwise than specifically described, and are to be interpreted as broadly as allowed under the law.
This application claims priority to and benefit of U.S. Provisional Patent Application No. 63/106,141, titled “Manipulating Bottlenecks in Data Networks,” filed on Oct. 27, 2020, the entire contents of which are incorporated herein by reference.
This invention was made with government support under Award No. DE-SC0019523 awarded by the U.S. Department of Energy. The government has certain rights in the invention
| Number | Name | Date | Kind |
|---|---|---|---|
| 9497039 | Greenberg | Nov 2016 | B2 |
| 9705798 | Abts | Jul 2017 | B1 |
| 9929960 | Abts | Mar 2018 | B1 |
| 9960878 | Gumaste | May 2018 | B2 |
| 10797943 | Fawcett | Oct 2020 | B2 |
| 10904077 | Fawcett | Jan 2021 | B2 |
| 11075798 | Fawcett | Jul 2021 | B2 |
| 20190304162 | Bakalash | Oct 2019 | A1 |
| 20220078130 | Ros-Giralt | Mar 2022 | A1 |
| Entry |
|---|
| Jordi Ros-Giralt et al. “On the Bottleneck Structure of Congestion-Controlled Networks”, Proc. ACM Meas. Anal. Comput. Syst., vol. 3, No. 3, Article 59, Dec. 2019, 31 pages. (Year: 2019). |
| Mohammad Al-Fares et al. “Hedera: Dynamic Flow Scheduling for Data Center Networks”, 2010, 15 pages. (Year: 2010). |
| Noah Amsel et al. “Computing Bottleneck Structures at Scale for High-Precision Network Performance Analysis”, 7th Annual International Workshop on Innovating the Network for Data-Intensive Science, Nov. 2020, 34 pages. (Year: 2020). |
| Jordi Ros-Giralt et al. “GradientGraph: A Network Optimization Framework for High-Precision Analysis of Bottleneck and Flow Performance”, Supercomputing INDIS Workshop 2019, Nov. 18, 2019, 40 pages. (Year: 2019). |
| Number | Date | Country | |
|---|---|---|---|
| 63106141 | Oct 2020 | US |
