Not applicable.
Not applicable.
Not applicable.
Network congestion occurs when demand for a resource exceeds the capacity of the resource. In an Ethernet network, when congestion occurs, traffic passing through a congestion point slows down significantly, either through packet drop, congestion notification, or back pressure mechanisms. Some examples of packet drop mechanisms may include tail drop (ID), random early detection (RED), and weighted RED (WIRED). A TD scheme drops packets at the tail end of a queue when the queue is full. A RED scheme monitors an average packet queue size and drops packets based on statistical probabilities. A WRED scheme drops lower priority packets before dropping higher priority packets. Some examples of congestion notification algorithms may include explicit congestion notification (ECN) and quantized congestion control (QCN), where notification messages are sent to cause traffic sources to respond to congestion by adjusting transmission rate. Back pressure employs flow control signaling mechanisms, where congestion states are signaled to upstream hops to delay and/or suspend transmissions of additional packets, where upstream hops refer to network nodes in a direction towards a packet source.
In one embodiment, the disclosure includes a network element (NE) comprising an ingress port configured to receive a first packet via a multipath network, a plurality of egress ports configured to couple to a plurality of links in the multipath network, and a processor coupled to the ingress port and the plurality of egress ports, wherein the processor is configured to determine that the plurality of egress ports are candidate egress ports for forwarding the first packet, obtain dynamic traffic load information associated with the candidate egress ports, and select a first target egress port from the candidate egress ports for forwarding the first packet according to the dynamic traffic load information.
In another embodiment, the disclosure includes an NE, comprising an ingress port configured to receive a plurality of packets via a multipath network, a plurality of egress ports configured to forward the plurality of packets over a plurality of links in the multipath network, a memory coupled to the ingress port and the plurality of egress ports, wherein the memory is configured to store a plurality of egress queues, and wherein a first of the plurality of egress queues stores packets awaiting transmissions over a first of the plurality of links coupled to a first of the plurality of egress ports, and a processor coupled to the memory and configured to send a congestion-on notification to a path selection element when determining that a utilization level of the first egress queue is greater than a congestion-on threshold, wherein the congestion-on notification instructs the path selection element to stop selecting the first egress port for forwarding first subsequent packets.
In yet another embodiment, the disclosure includes a method implemented in an NE, the method comprising receiving a packet via a datacenter network, identifying a plurality of NE egress ports for forwarding the received packet over a plurality of redundant links in the datacenter network, obtaining transient congestion information associated with the plurality of NE egress ports, and selecting a target NE egress port from the plurality of NE egress ports for forwarding the received packet according to the transient congestion.
These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims.
For a more complete understanding of this disclosure, reference is now made to the following brief description, taken in connection with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.
It should be understood at the outset that, although an illustrative implementation of one or more embodiments are provided below, the disclosed systems and/or methods may be implemented using any number of techniques, whether currently known or in existence. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalent.
Multipath routing allows the establishment of multiple paths between a source-destination pair. Multipath routing provides a variety of benefits, such as fault tolerance and increased bandwidth. For example, when an active or default path for a traffic flow fails, the traffic flow may be routed to an alternate path. Load balancing may also be performed to distribute traffic load among the multiple paths. Packet-based load balancing may not be practical due to packet reordering, and thus may be rarely deployed. However, flow-based load balancing may be more beneficial. For example, datacenters may be designed with redundant links (e.g., multiple paths) and may employ flow-based load balancing algorithms to distribute load over the redundant links. An example flow-based load balancing algorithm is the ECMP-based load balancing algorithm. The ECMP-based load balancing algorithm balances multiple flows over multiple paths by hashing traffic flows (e.g., flow-related packet header fields) onto multiple best paths. However, datacenter traffic may be random and traffic bursts may occur sporadically. Traffic burst refers to a high volume of traffic that occurs over a short duration of time. Thus, traffic bursts may lead to congestion points in datacenters. The employment of an ECMP-based load balancing algorithm may not necessarily steer traffic away from the congested link since the ECMP-based load balancing algorithm does not consider traffic load and/or congestion during path selection. Some studies on datacenter traffic indicate that at any given short time interval, about 40 percent (%) of datacenter links does not carry any traffic. As such, utilization of the redundant links provisioned by datacenters may not be efficient.
Disclosed herein are various embodiments for performing congestion avoidance traffic steering (CATS) in a network, such as a datacenter network, configured with redundant links. The disclosed embodiments enable network switches, such as Ethernet switches, to detect traffic bursts and/or potential congestion and to redirect subsequent traffic in real time to avoid congested links. In an embodiment, a network switch comprises a plurality of ingress ports, a packet processor, a traffic manager, and a plurality of egress ports. The ingress ports and the egress ports are coupled to physical links in the network, where at least some of the physical links are redundant links suitable for multipath routing. The network switch receives packets via the ingress ports. The packet processor classifies the received packets into traffic flows and traffic classes. Traffic class refers to the differentiation of different network traffic types (e.g., data, audio, and video), where transmission priorities may be configured based on the network traffic types. For example, each packet may be sent via a subset of the egress ports (e.g., candidates) corresponding to the redundant links available for each traffic flow. The packet processor selects an egress port and a corresponding path for each packet by applying a hash function to a set of the packet header fields associated with the classified traffic flow. After selecting an egress port for the packet, the packet may be enqueued into a transmission queue for transmission to the selected egress port. For example, each transmission queue may correspond to an egress port. The traffic manager monitors utilization levels of the transmission queues associated with the egress ports and notifies the packet processor of egress port congestion states, for example, based on transmission queue thresholds. In an embodiment, the traffic manager may employ different transmission queue thresholds for different traffic classes to provide different quality of service (QoS) to different traffic classes. As such, a particular egress ports may comprise different congestions states for different traffic classes. To avoid congestion, the packet processor excludes the congested candidate egress ports indicated by the traffic manager from path selection, and thus traffic is steered to alternate paths and congestion is avoided. When a congested egress port transitions to a congestion-off state, the packet processor may include the egress port during a next path selection, and thus traffic may be resumed on a congested path that is subsequently free of congestion. In some embodiments, the packet processor and the traffic manager are implemented as application specific integrated circuits (ASICs), which may be fabricated on a same semiconductor die or on different semiconductor dies. The disclosed embodiments may operate with any network software stacks, such as existing transmission control protocol (TCP) and/or Internet protocol (IP) software stacks. The disclosed embodiments may be suitable for use with other congestion control mechanisms, such as ECN, pricing for congestion control (PFCC), RED, and TD. It should be noted that in the present disclosure, path selection and port selection are equivalent and may be employed interchangeably.
In contrast to the ECMP algorithm, the disclosed embodiments are aware of traffic load and/or congestion state of each transmission queue on each egress port, whereas the ECMP algorithm is load agnostic. Thus, the disclosed embodiments may direct traffic to uncongested redundant links that are otherwise under-utilized. In contrast to the packet-drop congestion control method, the disclosed embodiments steer traffic away from potentially congested links to redundant links instead of dropping packets as in the packet-drop congestion control method. The packet-drop congestion control method may relieve congestion, but may not utilize redundant links during congestion. In contrast to the backpressure congestion control method, the disclosed embodiments steer traffic away from potentially congested links instead of requesting packet sources to reduce transmission rates as in the backpressure congestion control method. The backpressure congestion method may relieve congestion, but may not utilize redundant links during congestion. In contrast to distribute congestion-aware load balancing (CONGA), the disclosed embodiment respond to congestion in an order of a few microseconds (μs), where traffic is steered away from potentially congested links to redundant links to avoid traffic discards that are caused by traffic bursts. The CONGA method monitors link utilization and may achieve good load balance. However, the CONGA method is not burst aware, and thus may not avoid traffic discards resulted from traffic bursts. In addition, the CONGA method responds to a link utilization change in an order of a few hundred μs. In addition, the disclosed embodiments may be applied to any datacenters, whereas CONGA is limited to small datacenters with tunnel fabrics.
The packet classifier 210 is configured to classify incoming data packets into traffic flows. For example, packet classification may be performed based on packet headers, which may include Open System Interconnection (OSI) Layer 2 (L2), Layer 3 (L3), and/or Layer 4 (L4) headers. The flow hash generator 220 is configured to compute hash values based on traffic flows. For example, for each packet, the flow hash generator 220 may apply a hash function to a set of packet header fields that defines the traffic flow to produce a hash value. The path selector 230 is configured to select a subset of the egress ports 260 (e.g., candidate ports) for each packet based on the classified traffic flow and select an egress port 260 from the subset of the egress ports 260 based on the computed hash value. For example, the hash function produces a range of hash values and each egress port 260 is mapped to a portion of the hash value range. Thus, egress port 260 that is mapped to a portion corresponding to the computed hash value is selected. After selecting an egress port 260, the path selector 230 enqueues the data packet into an egress queue corresponding to the packet traffic class and associated with the selected egress port 260 for transmission over the link coupled to the selected egress port 260. The traffic manager 240 is configured to manage the egress queues and the transmissions of the packets. The hashing mechanisms may potentially spread traffic load of multiple flows over multiple paths. However, the path selector 230 is unaware of traffic load. Thus, when a traffic burst occurs, the hashing mechanisms may not distribute subsequent traffic to alternate paths.
The packet classifier 311 is configured to classify the incoming packets into traffic flows and/or traffic classes, for example, based on packet header fields, such as media access control (MAC) source address, MAC destination address, IP source address, IP destination address, Ethernet packet type, transport port, transport protocol, transport source address, and/or transport destination address. In some embodiments, packet classification may additionally be determined based on other rules, such as pre-established policies. Packet traffic class may be determined by employing various mechanisms, for example, through packet header fields, pre-established policies, and/or derived from ingress port 350 attributes. After a packet is successfully classified, a list of candidate egress ports 360 is generated for egress transmission. The flow hash generator 312 is configured to compute a hash value for each incoming packet by applying a hash function to a set of the flow-related packet header fields. The list of candidate egress ports 360, the flow hash value, the packet headers, and other packet attributes are passed along to subsequent processing stages including the path selector 313.
The flowlet table 315 stores flowlet entries. In some embodiment, traffic flows determined from the packet classifier 311 may be aggregated flows comprising a plurality of micro-flows, which may comprise more specific matching keys compared with the associated aggregated traffic flows. A flowlet is a portion of a traffic, where the portion spans a short time duration. Thus, flowlets may comprise short aging periods and may be periodically refreshed and/or aged. An entry in the flowlet table 315 may comprise an n-tuple match key, an outgoing interface, and/or maintenance information. The n-tuple match key may comprise match rules for a set of packet header fields that defines a traffic flow. The outgoing interface may comprise an egress ports 360 (e.g., one of the candidate ports) that may be employed to forward packets associated with the traffic flow identified by the n-tuple match key. The maintenance information may comprise aging and/or timing information associated with the flowlet identified by the n-tuple match key. The flowlet table 315 may be pre-configured and updated as new traffic flowlets are identified and/or existing traffic flows are aged.
The port queue congestion table 316 stores congestion statuses or states of transmission queues of the egress ports 360. For example, the network switch 300 may enqueue packets by egress port 360 and traffic class, where each egress port 360 is associated with a plurality of transmission queues of different traffic classes. The congestion states are determined by the traffic manager 320 based on egress queue thresholds, as discussed more fully below. In an embodiment, a link may be employed for transporting multiple traffic flows of different traffic classes, which may guarantee different QoS. Thus, an entry in a port queue congestion table 316 may comprise a plurality of bits (e.g., about 8 bits), each indicating a congestion state for a particular traffic class at an egress port 360.
The path selector 313 is configured to select an egress port 360 for each incoming data packet. The path selector 313 searches the flowlet table 315 for an entry that matches key fields including packet header fields and traffic class of the incoming data packet. When a match is found in the flowlet table 315, the path selector 313 obtains the egress port 360 from the matched flowlet entry and looks up the port queue congestion table 316 to determine whether the transmission queue for the packet traffic class on the egress ports 360 is congested. If the packet traffic class queue on the egress port 360 is not congested, the port from the matching flowlet entry is used for packet transmission. If the packet traffic class queue on the egress port 360 is congested, the path selector 313 chooses a different egress port 360 for transmission. The path selector 313 excludes any congested egress ports 360 during path selection. To choose a different egress port 360, the path selector 313 goes through the list of candidate egress ports 360 determined from the packet classifier 311. For example, for each candidate egress port 360, if the queue for the packet traffic class on the egress port 360 is congested, the egress port 360 is excluded from path selection. The remaining egress ports 360 may be used for port selection based on the flow hash. In an embodiment, the key space of the hash value is divided among the candidate egress ports 360 and each candidate egress port 360 may be mapped to a region of the key space. As an example, the hash value may be 4-bit value between 0 to 15 and the number of candidate egress ports 360 may be four. When splitting the key space equally, each egress port 360 may be mapped to four hash values. However, when one of the candidate egress ports 360 is congested, the path selector 313 excludes the congested candidate egress port 360 and divides the key space among the remaining three candidate egress ports 360. When a match for an incoming packet is not found in the flowlet table 315, the path selector 313 selects an egress port 360 by hashing among the non-congested egress ports 360 and adds an entry to the flowlet table 315. For example, the entry may comprise an n-tuple match key that identifies a traffic flow and/or a traffic class of the incoming packet and the selected egress port 360.
The traffic manager 320 is configured to manage transmissions of packets over the egress ports 360. The traffic manager 320 monitors for congestion states of the egress ports 360 and notifies the packet processor 310 of the egress ports 360's congestion states to enable the packet selector 313 to perform congestion-aware path selection as described above. For example, the packet processor 310 may employ a separate egress queue (e.g., stored in memory) to queue packets for each egress port 360. Thus, the traffic manager 320 may determine congestion states based on the number of packets in the egress queues pending for transmission (e.g., queue utilization levels). In an embodiment, the traffic manager 320 may employ two thresholds, a congestion-on threshold and a congestion-off threshold. The congestion-on threshold and the congestion-off threshold are measured in terms of number of packets in an egress queue. When an egress queue for a particular egress port 360 reaches the congestion-on threshold, the traffic manager 320 may set the congestion state for the particular egress port 360 to congestion-on. When an egress queue for a particular egress port 360 falls below the congestion-off threshold, the traffic manager 320 may set the congestion state for the particular egress port 360 to congestion-off. In some embodiments, the traffic manager 320 may employ different congestion-on and congestion-off thresholds for traffic flows with different traffic classes so that a particular QoS may be guaranteed for a particular traffic class. Thus, for a particular egress port 360, the traffic manager 320 may set different congestion states for different traffic classes. For example, when the network switch 300 supports eight different traffic classes, the traffic manager 320 may indicate eight congestion states for each egress port 360, where each congestion state correspond to one of the traffic classes. It should be noted that the network switch 300 may be configured as shown or alternatively configured as determined by a person of ordinary skill in the art to achieve similar functionalities.
It is understood that by programming and/or loading executable instructions onto the NE 400, at least one of the processor 430 and/or memory device 432 are changed, transforming the NE 400 in part into a particular machine or apparatus, e.g., a multi-core forwarding architecture, having the novel functionality taught by the present disclosure. It is fundamental to the electrical engineering and software engineering arts that functionality that can be implemented by loading executable software into a computer can be converted to a hardware implementation by well-known design rules. Decisions between implementing a concept in software versus hardware typically hinge on considerations of stability of the design and numbers of units to be produced rather than any issues involved in translating from the software domain to the hardware domain. Generally, a design that is still subject to frequent change may be preferred to be implemented in software, because re-spinning a hardware implementation is more expensive than re-spinning a software design. Generally, a design that is stable that will be produced in large volume may be preferred to be implemented in hardware, for example in an ASIC, because for large production runs the hardware implementation may be less expensive than the software implementation. Often a design may be developed and tested in a software form and later transformed, by well-known design rules, to an equivalent hardware implementation in an ASIC that hardwires the instructions of the software. In the same manner as a machine controlled by a new ASIC is a particular machine or apparatus, likewise a computer that has been programmed and/or loaded with executable instructions may be viewed as a particular machine or apparatus.
When the state machine 900 is operating in the CATS congestion-on state 920, the traffic manager continues to monitor the egress queue usage. When the egress queue usage falls below a CATS congestion-off threshold, the state machine 900 returns to the CATS congestion-off state 910 (shown by a solid arrow 942), where the CATS congestion-on threshold is greater than the CATS-congestion-off threshold. Upon detection of the state transition to the CATS congestion-on state 920, the traffic manager notifies the packet processor so that the packet processor may resume assignment of the traffic to the particular egress port.
The network switch 900 may optionally employ the disclosed CATS mechanisms in conjunction with other congestion control algorithms, such as the ECN and PFCC. For example, the traffic manager may configure an additional threshold for entering the congestion-X state 930 for performing other congestion controls, where the additional threshold is greater than the CATS congestion-on threshold. When operating in the CATS congestion-on state 920, the traffic manager may continue to monitor the egress queue usage. When the egress queue usage reaches the additional threshold, the state machine 900 transitions to the congestion-X state 930 (shown by a dashed arrow 943). Similarly, upon detection of the state transition to the congestion-X state 930, the traffic manager notifies the packet processor and the packet processor may perform additional congestion controls, such as ECN, PFCC, TD, RED, and/or WRED. The state machine 900 may return to the CATS congestion-on state 920 (shown by a dashed arrow 944) when the egress queue usage falls below the additional threshold. It should be noted that the state machine 900 may be applied to track congestion state transition for a particular traffic class for a particular egress port.
At step 1040, a determination is made whether a flowlet table entry, such as the flowlet table entries 710, matches the received packet. For example, a match may be determined by comparing a flowlet-related portion (e.g., packet header fields) of the received packet to a match keys, such as the match key 720, in the entries of the flowlet table. If a match is found, next at step 1050, an egress port is selected from the matched flowlet table entry, where the matched flowlet table entry comprises an outgoing interface, such as the outgoing interface 730, indicating a list of one or more egress ports. For example, the egress port may be selected by hashing the flowlet-related portion of the receive packet among the list of egress ports indicated in the matched flowlet table entry. At step 1060, a determination is made whether the selected egress port is congested for carrying traffic of the determined traffic class, for example, by looking up the port queue congestion table. If the selected egress port is congested for carrying traffic of the determined traffic class, next at step 1070, an egress port is selected by hashing the flow-related portion of the received packet among the uncongested egress ports indicated in the matched flowlet table entry. At step 1080, the received packet is forwarded to the selected egress port. At step 1090, the flowlet table is updated, for example, by refreshing the flowlet entry corresponding to the forwarded packet.
If the selected egress port is determined to be not congested for carrying traffic of the determined traffic class at step 1060, the method 1000 proceeds to step 1080, where the received packet is forwarded to the egress port selected from the matched flowlet table entry at step 1050.
If a match is not found at step 1040, the method 1000 proceeds to step 1041. At step 1041, an egress port is selected by hashing the flow-related portion of the received packet among the candidate egress ports that are uncongested for carrying traffic of the determined traffic class, where the congestion states of the egress ports may be obtained from the port queue congestion table. At step 1042, a flowlet table entry is created. For example, the match key of the created flowlet table entry may comprise rules for matching the flowlet-related portion of the received packet. The outgoing interface of the created flowlet table entry may indicate the egress port selected at step 1041.
It should be noted that although the congested egress ports are excluded from selection (e.g., at steps 1041 and 1070), there may be inflight packets that are previously assigned to the congested egress ports, where the inflight packets may be drained (e.g., transmitted out of the congested egress ports) after some duration. After the inflight packets are drained, the congested egress ports may be free of congestion, where the congestion response and congestion resolve time are discussed more fully below. It should be noted that the method 1000 may be performed in the order as shown or alternatively configured as determined by a person of ordinary skill in the art to achieve similar functionalities.
As shown in graph 1300, the network switch may employ an additional threshold 1 and an additional threshold 2 to perform further congestion controls. For example, when the egress queue usage reaches the additional threshold 1, the network switch may start to execute ECN or PFCC congestion controls to notify upstream hops. When the egress queue usage continues to increase to the additional threshold 2, the network switch may start to drop packets, for example, by employing a TD or a RED control method. It should be noted that the egress queue usage may fluctuate depending on the ingress traffic, for example, as shown in the duration 1310 between time T2 and time T3.
As shown in the activity graph 1420, the port resolver assigns and/or enqueues packets into three egress queues, each corresponding to one of the egress ports at the network switch. For example, the port resolver may have similar mechanisms as the path selector 313 and the methods 1000, 1100, and 1200. For example, the solid arrows represent packets assigned to egress port X and/or enqueue into the egress queue X. The dotted arrows represent packets assigned to an egress port Y and/or enqueue into an egress queue Y. The dashed arrows represent packets assigned to an egress port Z and/or enqueue into an egress queue Z.
In the scenario 1400, the CATS state for the egress queue X begins with a CATS congestion-off state. At time T1, the activity graph 1430 shows a burst of packets 1461 are enqueued for transmission via the particular egress port X. At time T2, the activity graph 1440 shows that the network switch detects the burst of packets 1461 at the egress queue X, for example, via a traffic manager, such as the traffic manager 320, based on a CATS congestion-on threshold. When the usage of the egress queue X reaches the CATS congestion-on threshold, the traffic manager transitions the CATS state to a CATS congestion-on state and notifies the port resolver. However, the packets (e.g., in-flight packets 1462) that are already in the pipeline for transmission over the egress port X may continue for a duration, for example, until time T4. At time T3, the activity graph 1420 shows that the port resolver stopped assigning packets to the egress queue X (e.g., no solid arrows over the duration 1463). At time T4, the in-flight packets 1462 in the egress queue X are drained and no new packets are enqueued into the egress queue X. The time duration between the time (e.g., time T2) when a traffic burst is detected to the time when packets are drained at the egress queue X (e.g., time T4) is referred to as the congestion response time 1471.
At time T5, the activity graph 1440 shows that the traffic manager detects that the egress port X is free of congestion, and thus switches the CATS state to a CATS congestion-off state and notifies the port resolver. Subsequently, the activity graph 1420 shows that the port resolver resume packet queuing at the egress queue X, in which packets are enqueued into the egress queue X at time T6 after congestion is resolved. The time duration between the time (e.g., time T2) when a traffic burst is detected to the time when packet enqueuing to the egress queue X is resumed (e.g., time T6) is referred to as the congestion resolve time 1472. It should be noted that the congestion response time 1471 and the congestion resolve time 1472 shown in the scenario 1400 is for illustrative purpose. The number of clocks or the duration of the congestion response time and the congestion resolve time may vary depending on various design and operational factors, such as transmission schedules, queue lengths, and the pipelining architecture of the network switch. However, for bursty traffic, the congestion response time 1471 may be about a few dozen of nanoseconds (ns) and the congestion resolve time 1472 may be within about one scheduling cycle (e.g., about 0.5 μs to about 1 μs).
While several embodiments have been provided in the present disclosure, it should be understood that the disclosed systems and methods might be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are to be considered as illustrative and not restrictive, and the intention is not to be limited to the details given herein. For example, the various elements or components may be combined or integrated in another system or certain features may be omitted, or not implemented.
In addition, techniques, systems, subsystems, and methods described and illustrated in the various embodiments as discrete or separate may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the present disclosure. Other items shown or discussed as coupled or directly coupled or communicating with each other may be indirectly coupled or communicating through some interface, device, or intermediate component whether electrically, mechanically, or otherwise. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and could be made without departing from the spirit and scope disclosed herein.