This description relates to systems and techniques for weighted load balancing in a multistage network.
A network may include a multistage network, which may include a wired and/or wireless network. In some example implementations, a multistage network may be used in the context of telecommunication and data centers to realize large-scale networks. A multistage network in a data center may have a large shared infrastructure of shared resources. In the multistage network, there may be multiple paths on which to send traffic between a source device and a destination device using the shared infrastructure. Weighted routing of data traffic over such multiple paths can be used to load balance data traffic in a multistage networks. However, current approaches for weighted may be complicated to implement in large multistage networks, such as, for example, data center networks.
According to one general aspect, a method for routing packets in a network includes receiving, at a data switch, a data packet, the data switch including a plurality of egress ports. The method includes, for each of the egress ports, generating, by the data switch, an independent hash value based on one or more fields of the data packet and generating a weighted hash value by scaling the hash value using a scaling factor, the scaling factor being based on at least two traffic routing weights of a plurality of respective traffic routing weights associated with the plurality of egress ports. The method also includes selecting an egress port of the plurality of egress ports based on the weighted hash value for each of the egress ports and transmitting, by the data switch, the data packet using the selected egress port.
Implementations can include one or more of the following features. For example, the one or more fields of the data packet can include one or more fields of a header of the data packet, where the one or more fields of the header of the data packet can have fixed values for each data packet of a data flow associated with the data packet.
Generating independent hash values for the plurality of egress ports can include generating a first hash value using a first hash function, where the first hash value is associated with a first egress port of the plurality of egress ports, and generating a second hash value using a second hash function, where the second hash value is associated with a second egress port of the plurality of egress ports, The second hash function can be different than, and independent of, the first hash function. Generating independent hash values for the plurality of egress ports further can include generating a third hash value using a third hash function. The third hash value can be associated with a third egress port of the plurality of egress ports. The third hash function can be different than, and independent of, the first hash function and the second hash function.
A scaling factor for a given egress port of the plurality of egress ports can be a ratio of a probability of the given egress port being selected in a joint probability distribution for the plurality of egress ports with a probability of an egress port with a lowest routing weight of the respective traffic routing weights being selected in the joint probability distribution. The probability of the given egress port being selected in the joint probability distribution can be proportional with a routing weight associated with the given egress port.
The plurality of routing weights can be normalized such that a smallest routing weight of the plurality of routing weights has a normalized value of 1. Generating the independent hash value for each of egress ports can include normalizing a plurality of respective independent hash values for the plurality of egress ports to respective values in a range of 0 to 1. Selecting the egress port of the plurality of egress ports can include selecting an egress port of the plurality of egress ports having a highest respective scaled hash value.
In another general aspect, a data switch including a plurality of egress ports includes at least one memory that is configured to store instructions and at least one processor that is operably coupled to the at least one memory and that is configured to process the instructions to cause the data switch to receive a data packet. The instructions further cause the data switch, for each of the egress ports, to generate an independent hash value based on one or more fields of the data packet and generate a weighted hash value by scaling the hash value using a scaling factor. The scaling factor is based on at least two traffic routing weights of a plurality of respective traffic routing weights associated with the plurality of egress ports. The instructions further cause the data switch to select an egress port of the plurality of egress ports based on the weighted hash value for each of the egress ports and transmit the data packet using the selected egress port.
Implementations can include one or more of the following features. For example, generating independent hash values for the plurality of egress ports can include generating a first hash value using a first hash function, where the first hash value is associated with a first egress port of the plurality of egress ports, and generating a second hash value using a second hash function, where the second hash value is associated with a second egress port of the plurality of egress ports, The second hash function can be different than, and independent of, the first hash function. Generating independent hash values for the plurality of egress ports further can include generating a third hash value using a third hash function. The third hash value can be associated with a third egress port of the plurality of egress ports. The third hash function can be different than, and independent of, the first hash function and the second hash function.
A scaling factor for a given egress port of the plurality of egress ports can be a ratio of a probability of the given egress port being selected in a joint probability distribution for the plurality of egress ports with a probability of an egress port with a lowest routing weight of the respective traffic routing weights being selected in the joint probability distribution. The probability of the given egress port being selected in the joint probability distribution can be proportional with a routing weight associated with the given egress port.
Selecting the egress port of the plurality of egress ports can include selecting an egress port of the plurality of egress ports having a highest respective scaled hash value.
The one or more fields of the data packet can include one or more fields of a header of the data packet, the one or more fields of the header of the data packet having fixed values for each data packet of a data flow associated with the data packet.
In another general aspect, a method includes receiving, at a data switch, a set of respective traffic routing weights for a plurality of egress ports of the data switch, each egress port of the plurality of egress ports being associated with a respective independent hash function. The method also includes determining, for the plurality of egress ports, a plurality of respective hash function scaling factors, the plurality of respective hash function scaling factors being based on the set of routing weights and a joint probability distribution for the plurality of egress ports. The method also includes routing a data packet using the respective independent hash functions and the respective hash function scaling factors.
Implementations can include one or more of the following features. For example, a scaling factor for a given egress port of the plurality of egress ports can be a ratio of a probability of the given egress port being selected in a joint probability distribution for the plurality of egress ports with a probability of an egress port with a lowest routing weight of the respective traffic routing weights being selected in the joint probability distribution. The probability of the given egress port being selected can be proportional with a routing weight associated with the given egress port. The plurality of routing weights can be normalized such that a smallest routing weight of the plurality of routing weights has a normalized value of 1. Determining the plurality of respective hash function scaling factors includes iteratively determining the plurality of hash function scaling factors.
In another general aspect, a data switch includes at least one memory that is configured to store instructions and at least one processor that is operably coupled to the at least one memory and that is configured to process the instructions to cause the data switch to receive a set of respective traffic routing weights for a plurality of egress ports of the data switch, each egress port of the plurality of egress ports being associated with a respective independent hash function. The instructions further cause the data switch to determine, for the plurality of egress ports, a plurality of respective hash function scaling factors, the plurality of respective hash function scaling factors being based on the set of routing weights and a joint probability distribution for the plurality of egress ports. The instructions also cause the data switch to route a data packet using the respective independent hash functions and the respective hash function scaling factors.
Implementations can include one or more of the following features. For example, a scaling factor for a given egress port of the plurality of egress ports can be a ratio of a probability of the given egress port being selected in a joint probability distribution for the plurality of egress ports with a probability of an egress port with a lowest routing weight of the respective traffic routing weights being selected in the joint probability distribution. The probability of the given egress port being selected can be proportional with a routing weight associated with the given egress port
The plurality of routing weights can be normalized such that a smallest routing weight of the plurality of routing weights has a normalized value of 1. The plurality of respective hash function scaling factors can include iteratively determining the plurality of hash function scaling factors.
The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.
This document describes systems and techniques for routing packets in a network. For example, a source device may transmit packets to a destination device using a multi-level network, where multiple data paths (links) may be available (used) to transmit data between the source device and the destination device. Data switches (or other data routing devices) in the multi-level network may use a weighted cost multi-path (WCMP) routing module to balance data traffic (between the source device and the destination device) over the multiple paths between the source device and the destination device. Such load balancing (e.g., WCMP data traffic routing) may be implemented using the techniques described herein.
The network 100 illustrated in
In such multistage networks, data traffic may be routed using equal cost multi-path (ECMP) routing for load-balancing data traffic across the different switches at each stage. When the data switches in a multistage network, such as the data switches in the network 100, have more than one neighbor on which they can forward data traffic toward a given destination (e.g., with all resulting routes preferably being of a same length), ECMP randomly distributes (e.g., using a hash function and a corresponding modulus operation) to route different data traffic flows destined for the given destination across a group of ports connected to those neighbors, with substantially the same probability of each path being selected for a given data flow. For instance, if two communication links connect a first data switch and a second data switch, using ECMP would result, over a sufficient sample size, in fifty percent of data traffic that is communicated between the first switch and the second switch being communicated over one data communication link and the other fifty percent of data traffic being communicated over the other data communication link. Accordingly, such ECMP approaches may provide substantially equally balanced traffic distribution in multistage networks that are of uniform topology (e.g., there is a same number of data links from a given data switch to each neighboring switch along parallel data paths).
However, ECMP approaches would not (e.g., presuming the use of statistically sufficient random selection) provide such evenly balanced traffic distribution in multistage networks that have uneven inter-stage connectivity topologies. For example, in a network with such an uneven inter-stage connection topology, the ports (ingress ports and egress ports, which may also be referred to as communication links, or links) that connect a sending switch with its neighboring switches on parallel paths are not evenly distributed.
As illustrated in
Such uneven inter-stage connectivity in such a multistage network may occur, for instance, by design. This design choice may occur because of the number of data switches present at each stage (e.g., due to the configuration of the particular network), or may occur because the multistage network was intentionally configured to leverage certain traffic locality patterns with uneven inter-stage connectivity. In other instances, uneven inter-stage connectivity in a multi-stage network may occur as a result of link failures between data switches in the network.
Weighted cost multi-path (WCMP) data routing may be used to overcome, at least some of, the shortcomings of ECMP to evenly balance data traffic in such multi-stage networks with uneven inter-stage topologies. Because ECMP, which inherently assumes that all paths to a certain destination have a same capacity, balancing data traffic across unevenly distributed links with equal probability results in unequal traffic loading.
For example and purposes of illustration, with reference to
Accordingly, using ECMP in the network 100 (with its uneven inter-stage topology) leads to (for data traffic from the switch 110 to the switch 140) underutilizing the path through data switch 130 by 50% compared to using WCMP, where the links 112, 114 (from the switch 110 to 120) each have a weight of “1” and the link 116 (from the switch 110 to the switch 130) has a weight of “2”, resulting in twice as much data traffic (from the switch 110 to the switch 140) being routed on the link 116 as each of the links 112,114.
Currently data switching devices (e.g., commodity data switches) support ECMP functionality, but have no direct functional support (hardware, software and/or firmware) for WCMP data traffic routing. Instead, WCMP functionality is accomplished by overloading (replicating entries in) an ECMP table to achieve WCMP traffic routing. Referring still to
While in the above example, WCMP may easily implemented by replicating the egress port listing for the link 116 in an EMCP table, such an approach does not work well for large scale multi-level networks, such as on networks on the order a commercial DCN. Scalability of WCMP in such networks has multiple considerations. For example, one consideration is the ability to support an arbitrary mix of weights with sufficient resolution in the switches of such networks. In large scale networks (e.g., with uneven inter-stage topologies), using ECMP tables with replicated entries to implement WCMP would typically require a prohibitive degree of replication (i.e., the required table sizes would likely far exceed the capacity of memory structures currently used to implement ECMP tables).
Another scalability concern for WCMP in large scale multi-level networks is the complexity of updating routing weights in the switches used to implement those networks. For instance, in such large scale multi-level networks (e.g., DCNs), relatively simple changes in routing weights may become large complex operations. For example, consider changing a routing weight of “5” to a routing weight of “4” in for a group with current weights 5, 30, 75, 40. Because current implementation do not take into account previous table configurations when altering routing weights, such a change could require rewriting, potentially, 149 table entries (e.g., 4+30+75+40). Accordingly, the amount of work can be arbitrarily large for even a small delta in weights.
The method 200 includes, at block 210, receiving a set of routing weights for a group of egress ports of a data switch. For instance, the routing weights received at block 210 may correspond with a group of egress ports that can be used to send data traffic to a particular destination. The routing weights of block 210 may be used to implement WCMP data traffic routing for data traffic sent from the data switch to that particular destination. The routing weights may be provided to the data switch using a number of approaches. For example, the routing weights may be provided to the data switch from a network configuration system. The data switch, at block 210, may receive the routing weights via a management port, via a configuration packet, or by a number of other approaches.
At block 220, the method 200 includes processing (which may also be referred to as pre-processing) the routing weights received at block 210, such as using hardware or software (or firmware) for use in a WCMP table (or WCMP tables). At block 230, the processed (pre-processed) weights of block 220 may be used (further processed) to create one or more WCMP data routing tables, where the WCMP table(s) created at block 230 may be used to route data traffic from the data switch of block 210 to the particular destination with which the received routing weights are associated. The one more WCMP tables of block 230 may be stored in the data switch using hardware, software, or a combination thereof.
At block 240, a data packet may be received at the data switch of block 210. When the data packet is received, the data switch may determine a destination address of the data packet. The destination address may then be looked up in a forwarding table to determine a corresponding routing table (or tables) to use to determine an egress port to use to send the received data packet along to its destination. If the destination address of the data packet received at block 240 corresponds with the destination address associated with the routing weights received at block 210, the forwarding table lookup may return a pointer to the WCMP table (or tables) generated at block 230.
After a determination is made, at block 240, to use the WCMP table (or tables) of block 230 to determine an egress port (hardware port or data transmission link) to use for sending the data packet along to its destination, the method 200 includes, at block 250, generating at least one hash value from a header of the received data packet. In order to ensure that all data packets in a given data flow are sent on a same path, the hash value (or values) may be generated using one or more fields of the data packet header that, for a given data flow, have fixed values. For instance a flow identification field, a source address field, a destination address field, a protocol identification field and/or a number of other possible fields with fixed values for a given data flow. The specific header field (or fields) that are used for generating the one or more hash values may depend on the particular network implementation.
At block 260, the method 200 includes determining, by using the one or more hash values as lookup value(s) for the WCMP tables of block 230. The lookup, at block 260 returns (determines) an egress port of the data switch to use to forward the data packet onto its destination. At block 270, the method 200 includes transmitting the data packet using the determined egress port. In the method 200 (as well as other methods for performing WCMP data traffic routing), a number of different hash functions may be used. For instance, a CRC16 hash function, a CRC32 hash function, an AES hash function, an SHA hash function and/or a number of other hash functions may be used. The hash function(s) used should be selected such that the hash values generated are sufficiently random for data routing purposes in a multi-level network, such as those described herein.
Further, in example implementation, a modulus (remainder) of the hash value generated by the selected hash functions may be taken before performing a lookup in the WCMP table(s). The result of the modulus function may be used directly, or indirectly, depending on the implementation, as an index (or lookup value) for the WCMP table(s) of block 230. In such an approach, a modulus function that is applied may be based on the number of entries present in a WCMP table(s) on which the lookup will be performed. For instance, if a 16 bit hash value is generated from a data packet header and an associated WCMP table that will be used to determine an egress port on which to send the data packet has nine entries, the lookup value for the WCMP table may be determined by Equation 1 below:
Lookup value=Hash_value mod 9 Equation 1
Which results in a lookup value between 0 and 8, which may be used as a lookup (or to generate a lookup value or search key) for a WCMP table with indices 0 through 8 (with 9 total entries). In certain implementations, the lookup value may be a function of the result of the modulus operation (e.g., an encoded version of the result of the modulus function). The specific structure and arrangement of the WCMP table(s) (and the associated lookup values) used in the method 200 will depend on the specific implementation.
For instance, the data switch 300 may receive a set of routing weights for a particular destination, e.g., via the ingress port 310 or a management port (not shown) and use those weights to implement one or more WCMP routing tables that may be included in the WCMP routing module 340. The data switch 300 may also use the ECMP routing module 330 to implement ECMP data traffic routing for one or more destinations in a data network in which the data switch 300 is included, where each parallel path (the multi-paths) between the data switch 300 and a given destination have uniform inter-level topologies.
In the data switch 300, when a data packet is received at the ingress port 310, the data switch 300 may determine a destination address for that data packet from its header. If the destination address of the received data packet corresponds with a destination address that is associated with a WCMP table (or tables) in the WCMP module 340 (e.g., corresponds with the received routing weights), the forwarding table 320 (when the destination address is used to perform a lookup in the forwarding table 320) may return a pointer to the corresponding WCMP routing tables in the WCMP routing module 340. An egress port of the egress ports 350 to use to communicate the data packet to its destination may then be determined from the corresponding WCMP table(s) or scaled hash values, such as by using the approaches described herein. After the particular egress port to use is determined, the data packet may be sent on to its destination using the determined egress port.
For purposes of illustration and clarity, the implementations described below with respect to
Likewise, hash values produced by the independent hash functions associated with the egress ports of a WCMP group may be normalized to be in a fixed range of values, e.g., in a range from 0 to 1. The initial (raw) hash values (before being normalized) may be produced using a modulus operation, such as described herein. Alternatively, in the example implementations discussed below, the hash values that are scaled using respective hash function scaling factors can be the “raw” hash values, such as 16 bit hash values, for example. The specific approach will depend on the particular hash functions used for the egress ports of a given WCMP group. For example, in certain implementations, the independent hash functions may produce raw hash values with different numbers of bits. In such an approach, a modulus operation may be used to produce hash values within a fixed integer range (e.g., based on a number of egress ports in an associated WCMP group) before normalizing the hash values.
Also in the examples below, groups of egress ports (e.g., WCMP groups) are described as having respective independent hash functions associated with each egress port of the given group. For purposes of this disclosure, independent hash functions may be defined as hash functions that are not dependent on one another and have equal probability of producing any particular value in a given, fixed range of values for a statistically random population of data packets. The specific hash functions used in a particular implementation will depend on a number of factors, such as the configuration of a particular network in which they are used and/or the configuration of a data switch that is used to implement weighted (WCMP) data traffic routing using the approaches described herein.
As shown in
Each routing weight of the set of routing weights of block 410 may be associated, respectively, with a specific egress port of the data switch. Each of the egress ports of the plurality of egress ports may be associated with an independent hash function, which will be used to generate, in parallel, independent hash values for each data packet (data frame, or other data unit used to transmit data from a source to a destination) received at the data switch. The independent hash functions may each be of a same class of hash functions, or can be of different classes. For example, the independent hash functions may all be independent CRC16 hash function variants. Alternatively, the independent hash functions may be a mix of CRC16, CRC32, AES, SHA and/or other types of hash functions.
At block 420, the method 400 includes determining, for each egress port of the plurality of egress ports, a respective hash function scaling factor, where the hash function scaling factor is based on the set of routing weights (e.g., a ratio of routing weights) and a joint probability distribution for routing data traffic with the plurality of egress ports according to their assigned routing weights. In an example implementation, the scaling factor for a given egress port of the plurality of egress ports may be a ratio of a probability of the given egress port being selected in a joint probability distribution for the plurality of egress ports with a probability of an egress port with a lowest routing weight of the respective traffic routing weights being selected in the joint probability distribution, where the probability of the given egress port being selected is proportional with a routing weight associated with the given egress port. At block 430, the scaling factors may be stored in a WCMP traffic routing module (e.g., the WCMP routing module 340) of the data switch. Example approaches for determining such hash function scaling factors are described below with respect to
At block 440, the method 400 includes routing a data packet using the respective independent hash functions and the respective hash function scaling factors. For instance, respective independent hash values based on one or more fields of the packet (e.g., one or more packet header fields) may be produced by each independent hash function (each being associated with an egress port a WCMP group). Depending on the specific implementation, the independent (parallel) hash values for the data packet may be normalized to a fixed range (e.g., 0 to 1). The independent hash values may then be scaled using their corresponding scaling factors determined at block 420. The scaled hash values may then be examined to select an egress port on which to transmit the data packet towards its destination. For example, an egress port may be selected by determining the highest scaled hash value and selecting the corresponding egress port for transmitting the data packet towards its destination. In situations where two or more scaled hash values have the same value (e.g., a “tie”), an egress port with a lowest port index may be selected from the egress port with the same highest scaled hash function value. Alternatively, in the case of a “tie”, an egress port for that packet (and other packets in an associated data flow) can be randomly selected. For instance, for the first data packet in such a flow, a random selection can be made, with all subsequent packets in the data flow being forwarded on that randomly selected egress port.
Briefly, a given scaling factor for a given egress port of a WCMP group that has a normalized routing weight of greater than 1 (e.g., does not have the smallest routing weight of an associated set of routing weights) may be determined by equating a ratio of 1 (e.g., the normalized routing weight of an egress port with the lowest routing weight) and the normalized routing weight of a given egress port with a ratio of the probability of the egress port with the lowest routing weight being selected (in the weighted joint probability distribution) and a probability of the given egress port being selected (in the weighted joint probability distribution). Again, example approaches for determining such scaling factors are described below.
As shown in
As discussed herein, the respective independent hash values for each of the egress ports may be generated using respective independent hash functions associated with each of the egress ports. The respective independent hash functions can be predetermined (coded in software, hardware and/or firmware), assigned by the data switch, assigned by a network administration system or may be determined using a number of other approaches, such as random selection from a pool of available hash function options.
At block 530, the method 500 includes, for each egress port of the plurality of egress ports, generating a weighted hash value by scaling the independent hash value produced at block 520 using a scaling factor, such as a scaling factor that is determined using the approaches described herein. At block 540, the method 500 includes selecting an egress port of the plurality of egress ports based on the respective weighted hash values for the plurality of egress ports. For instance, an egress port associated with the largest scaled hash value for the data packet can be selected. In a case where two or more egress ports “tie” for the largest scaled hash value, the egress port with the smallest index can be selected. In other implementations, different criteria may be used to break “ties” for the largest scaled hash value. At block 550, the method 500 includes transmitting the data packet using the egress port selected at block 540.
By way of illustration, the data switch 600 can generate the respective independent hash values for the plurality of egress ports by generating a first hash value using the hash function_1620, where the first hash value is associated with an egress port 1 (index of 1) of the plurality of egress ports (the WCMP group). Generating the respective independent hash values for the plurality of egress ports in the data switch 600 can further include generating a second hash value using the hash function_2630, where the second hash value is associated with an egress port 2 (index of 2) of the plurality of egress ports (the WCMP group). Still further, generating the respective independent hash values for the plurality of egress ports in the data switch 600 can further include generating a third hash value using the hash function_N 640, where the third hash value is associated with an egress port N (index of N) of the plurality of egress ports. In the example approach, the third hash function is different than, and independent of, the first hash function and the second hash function, and the second hash function is different than, and independent of, the first hash function.
In the following discussion, for illustrative purposes, normalized routing weights of [1, 2] for the ports P1 and P2 will be used, respectively. In other implementations, other routing weights may be used. Also in the following discussion, it is presumed that hash values produced by respective independent hashing functions for each of the egress ports P1, P2 are normalized to have values in a range of 0 to 1, such as has been described herein. In
In this example, the scaling factor for the egress port with the lowest routing weight (egress port P1), which has a normalized routing weight of 1, can be set to a value of a[1]=1. As described herein, the scaling factor for egress port P1 can be used as a multiplier for independent hash values generated by an associated hash function. In this example, the scaling factor for the egress port P1 will be referenced as m[1]. The scaling factor m[1]=1 is shown on the axis designated P1 in
After setting the scaling factor m[1] for P1 at 1, the scaling factor for P2 (m[2]) can be assigned a value of m[2]=a[1]+a[2]=1+a[2]. The scaling factor m[2] is shown on the axis designated P[2] in
Area=a[1]·(1+a[2])=m[1]m[2]=1·2=2
Accordingly, in this situation, the area of the first region 710 in the joint probability distribution 700 would be ½, the area of the right triangle of the first region 710. Therefore, the area of the second region 720 and the joint probability distribution 700 would be 1½ (e.g., the total area of 2 minus the area of the first region 710). Therefore, in this situation, the probability of forwarding on the egress port P1 is the probability that the scaled hash value for the egress port P1 is greater than the scaled hash value for the egress port P2. That probability is equal to the ratio of the area of ½ of the first region 710 to the total area of the joint probability distribution 700 of 2, thus that probability equals ¼ (½ divided by 2). Likewise, the probability of forwarding on the egress port P2 is the probability that the scaled hash value for the egress port P2 is greater than the scaled hash value for the egress port P1. That probability is equal to the ratio of the area of 1½ of the second region 720 to the total area of the joint probability distribution 700 of 2, thus that probability equals ¾ (1½ divided by 2). Accordingly, the resulting forwarding weights ratio would be 1:3 instead of the desired 1:2 ratio for the normalized desired WCMP routing weights for the ports P1, P2 and result in inaccurate traffic weighting.
Referring still to
If we again fix the multiplier for egress port P1 as m[1]=1, we can then find the scaling factor m[2]=1+a[2], where a>0, such that the ratio R=Prob(H[1]≥H[2])/Prob(H[2]>H[1]) is equal to ½ (i.e., the desired ratio of the normalized routing weights). From the joint probability distribution 700 of
In this example, it will be assumed that the desired respective WCMP routing weights (w[1], w[2] and w[3]) for the egress ports P1, P2, P3 have values such that w[1]≤w[2]≤w[3]. As with the example discussed above with respect to
As in the example described above with respect to
As shown in
For purposes of this discussion, the following events (Boolean events) can be defined:
MAX1: max(H[1],H[2],H[3])=H(1)
MAX2: max(H[1],H[2],H[3])=H(2)
MAX3: max(H[1],H[2],H[3])=H(3).
In
In this example, given that the joint probability distribution 800 is a uniform probability distribution, in the region 810, the probability of each of the scaled hash values H[1], H[2], H[3] being the maximum of the three values is equal, or ⅓. In the region 820, the probability of H[1] being the maximum is 0, while the probability of H[2] or H[3] being the maximum is equal or ½. Further, in the region 830, the probability of H[1] or H[2] being the maximum is 0, while the probability of H[3] being the maximum is 1, as H[3] has the only scaled hash values in the region 830.
In terms of probability the union of the regions 810, 820, 830 covers the probability space of volume V. Accordingly the volumes of each region may be used to iteratively determine the scaling factors for each of the associated egress ports. Again, in this example, the scaling factor m[1] for the egress port P1 can be fixed at a value of 1. Accordingly the volume for each of the regions may be expresses as follows:
Volume(region 810)=V[1]=1,
Volume(region 820)=V[2]=(1)(1+a[2])(1+a[2])−V[1]=(1+a[2])2−1,
Volume(region 830)=V[3]=(1)(1+a[2])(1+a[1]+a[2])−V[1]−V[2]=(1+a[2])(a[3]).
By the law of total probability, using the events and volumes discussed above, it can be shown that Prob(M1):Prob(M2):Prob(M3)=V[1]/3:(V[1]/3+V[2]/2):(V[1]/3+V[2]/2+V[3]), which indicates that a[2] can be directly computed from the equation w[1]/w[2]=Prob(M1)/Prob(M2), as there is 1 equation and 1 unknown. Further, after a[2] is determined, a[3] could be then directly computed from the equation w[1]/w[3]=Prob(M1)/Prob(M3), as that would also be a calculation for 1 equation and 1 unknown. Once the values of a[2] and a[3] are determined for a given set of normalized WCMP, such as in this example, the scaling factors (such as for used in the data switch 600) can easily be computed (e.g., m[1]=a[1]=1, m[2]=a[1]+a[2], m[3]=a[1]+a[2]+a[3]).
The concepts discussed above may be extended to implementations where a WCMP group includes more than three egress ports. Again, in such implementations, the WCMP routing weights may be normalized to a have the lowest routing weight of 1 and to also have a routing weight of 1 be the lowest routing weight for an associated WCMP group. For instance, for a data switch that has N egress ports in a given ECMP groups, the joint probability distribution would be N-dimensional. The volumes for each region of an associated joint (uniform) probability distribution may be determined by the following equations, where i is the port index for an egress port that is associated with the volume being calculated and the values of i range from 1 to N (the total number of egress ports in the ECMP group) and Ri is the region of the joint probability distribution associated with the ith egress port.
V(R1)=a[1]N,
V(R2)=a[1]·(a[1]+a[2])N−1−V(R1),
V(R3)=a[1]·(a[1]+a[2])·(a[1]+a[2]+a[3])N−2−V(R2)−V(R1), etc.
The variables a[i] can be iteratively determined in succession, with each successive calculation solving one equation for one unknown by equating the ratio of desired routing weights for two given egress ports with a corresponding ratio of the volumes in the joint probability distribution associated with the given egress ports, such as discussed above with respect to
Expressed generally, probability space volumes for each region of a joint probability distribution for a WCMP group with N members may be expressed as, for the ith egress port: For egress ports where i<N:
V(Ri)=(Πj=1i(Σk=1ja[k]))×(Σk=1ia[k])N−i−(Σk=1i−1V(RK))
And for the Nth egress port:
V(RN)=(Πj=1N(Σk=1ja[k]))−(Σk=1N−1V(RK))
As in the previous examples, the variables a[k] can be determined by iteratively solving the foregoing equations for each egress port in order of their indices, e.g., P1, then P2, then P3, and so forth by equating them with corresponding ratios of the desired WCMP routing weights for the corresponding egress ports. Once the variables a[k] are determined, the scaling factors for scaling the independent hash values for each egress port may be readily determined by the general equation:
m[i]=Σj=1ia[j]
Further, egress ports that have equal routing weights for a given set of routing weights will have equal scaling factors.
Implementations of the various techniques described herein may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Implementations may be implemented as a computer program product, i.e., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable storage device, for execution by, or to control the operation of, data processing apparatus, e.g., a programmable processor, a computer, or multiple computers. A computer program, such as the computer program(s) described above, can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed on one computer (or device) or on multiple computers (or devices) at one site or distributed across multiple sites and interconnected by a communication network.
Method steps may be performed by one or more programmable processors executing a computer program to perform functions by operating on input data and generating output. Method steps also may be performed by, and an apparatus may be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).
Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer or device. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. Elements of a computer or device may include at least one processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer or device also may include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in special purpose logic circuitry.
To provide for interaction with a user, implementations may be implemented on a computer having a display device, e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
Implementations may be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation, or any combination of such back-end, middleware, or front-end components. Components may be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.
While certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. It is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes as fall within the scope of the implementations.
The present application is a continuation of co-pending U.S. patent application Ser. No. 14/217,921, filed Mar. 18, 2014, which claims priority to U.S. Provisional Patent Application No. 61/950,024, filed on Mar. 8, 2014, entitled “WEIGHTED LOAD BALANCING USING PARALLEL HASHING”, both of which are hereby incorporated by reference in their entirety.
Number | Name | Date | Kind |
---|---|---|---|
6356546 | Beshai | Mar 2002 | B1 |
6580721 | Beshai | Jun 2003 | B1 |
6721800 | Basso et al. | Apr 2004 | B1 |
6757742 | Viswanath | Jun 2004 | B1 |
6765866 | Wyatt | Jul 2004 | B1 |
6888797 | Cao | May 2005 | B1 |
7246148 | Phillips | Jul 2007 | B1 |
7523218 | Sahni et al. | Apr 2009 | B1 |
7697432 | Yu et al. | Apr 2010 | B2 |
7818303 | Buehrer | Oct 2010 | B2 |
7843926 | Muller et al. | Nov 2010 | B1 |
7898959 | Arad | Mar 2011 | B1 |
8111649 | Agarwall | Feb 2012 | B1 |
8259585 | S P | Sep 2012 | B1 |
8344853 | Warner | Jan 2013 | B1 |
8589688 | Minematsu | Nov 2013 | B2 |
8982700 | Zhou et al. | Mar 2015 | B1 |
9143449 | Valenza | Sep 2015 | B2 |
9565114 | Kabbani | Feb 2017 | B1 |
9717011 | Huang | Jul 2017 | B2 |
20030016666 | Okamoto | Jan 2003 | A1 |
20030105976 | Copeland, III | Jun 2003 | A1 |
20050013300 | Akahane | Jan 2005 | A1 |
20060018262 | Boulanger | Jan 2006 | A1 |
20060178153 | Tenny et al. | Aug 2006 | A1 |
20060198393 | Lamy | Sep 2006 | A1 |
20070288760 | Euchner et al. | Dec 2007 | A1 |
20090022321 | Saito | Jan 2009 | A1 |
20090116488 | Schollmeier | May 2009 | A1 |
20090268674 | Liu | Oct 2009 | A1 |
20100275028 | Takashima | Oct 2010 | A1 |
20100302935 | Zhang et al. | Dec 2010 | A1 |
20110013638 | Matthews et al. | Jan 2011 | A1 |
20110013639 | Matthews et al. | Jan 2011 | A1 |
20110075559 | Katsura | Mar 2011 | A1 |
20110145260 | Ichino | Jun 2011 | A1 |
20110161657 | So | Jun 2011 | A1 |
20110188503 | Hewson | Aug 2011 | A1 |
20110202612 | Craig et al. | Aug 2011 | A1 |
20110310739 | Aybay | Dec 2011 | A1 |
20120093158 | Chiba | Apr 2012 | A1 |
20120170575 | Mehra | Jul 2012 | A1 |
20120263048 | Chen | Oct 2012 | A1 |
20130155985 | Skoric | Jun 2013 | A1 |
20130201989 | Hu et al. | Aug 2013 | A1 |
20130232104 | Goyal et al. | Sep 2013 | A1 |
20130308455 | Kapadia | Nov 2013 | A1 |
20130329584 | Ghose et al. | Dec 2013 | A1 |
20140032156 | Sanyal et al. | Jan 2014 | A1 |
20140040433 | Russell, Jr. et al. | Feb 2014 | A1 |
20140122743 | Di Benedetto | May 2014 | A1 |
20140181298 | Wang et al. | Jun 2014 | A1 |
20140198656 | Venkatesh et al. | Jul 2014 | A1 |
20140293978 | Yang | Oct 2014 | A1 |
20140301394 | Arad et al. | Oct 2014 | A1 |
20150078375 | Hendel | Mar 2015 | A1 |
20150163140 | Rovner et al. | Jun 2015 | A1 |
20150180782 | Rimmer et al. | Jun 2015 | A1 |
20150180790 | Rimmer et al. | Jun 2015 | A1 |
20150222533 | Birrittella et al. | Aug 2015 | A1 |
20150244617 | Nakil et al. | Aug 2015 | A1 |
20150248458 | Sakamoto | Sep 2015 | A1 |
20150296399 | Huang | Oct 2015 | A1 |
20150326476 | Ye et al. | Nov 2015 | A1 |
20150358284 | Arkko | Dec 2015 | A1 |
20160050272 | Raduchel | Feb 2016 | A1 |
20160191421 | Tsuji | Jun 2016 | A1 |
Entry |
---|
Al-Fares, Mohammad, et al. A Scalable, Commodity Data Center Network Architecture, ACM SIGCOMM Computer Communication Review, vol. 38, No. 4, pp. 63-74, ACM, 2008. |
Liu, et al, “zUpdate: Updating Data Center Networks with Zero Loss”, SIGCOMM'13, Aug. 12-16, 2013, 12 pages. |
Zhou, Junlan, et al. WCMP: Weighted Cost Multipathing for Improved Fairness in Data Centers, Proceedings of the Ninth European Conference on Computer Systems, ACM, Apr. 13-16, 2014. |
Office Action dated Dec. 21, 2015 in U.S. Appl. No. 14/217,921. |
Office Action dated Jun. 8, 2016 in U.S. Appl. No. 14/217,921. |
Notice of Allowance dated Oct. 31, 2016 in U.S. Appl. No. 14/217,921. |
Number | Date | Country | |
---|---|---|---|
20170149877 A1 | May 2017 | US |
Number | Date | Country | |
---|---|---|---|
61950024 | Mar 2014 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 14217921 | Mar 2014 | US |
Child | 15396512 | US |