System and method for facilitating global fairness in a network

Description

BACKGROUND
Field

This is generally related to the technical field of networking. More specifically, this disclosure is related to systems and methods for facilitating global fairness in a network.

Related Art

As network-enabled devices and applications become progressively more ubiquitous, various types of traffic as well as the ever-increasing network load continue to demand more performance from the underlying network architecture. For example, applications such as high-performance computing (HPC), media streaming, and Internet of Things (IOT) can generate different types of traffic with distinctive characteristics. As a result, in addition to conventional network performance metrics such as bandwidth and delay, network architects continue to face challenges such as scalability, versatility, and efficiency.

SUMMARY

A data-driven intelligent networking system that can facilitate global fairness is provided. The system can maintain state information of individual packet flows, which can be set up or released dynamically based on injected data. Each flow can be provided with a flow-specific input queue upon arriving at a switch. Packets of a respective flow are acknowledged after reaching the egress point of the network, and the acknowledgement packets are sent back to the ingress point of the flow along the same data path. As a result, each switch can obtain state information of each flow and enforce global fairness on a per-flow basis.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 shows an exemplary network that facilitates flow channels.

FIG. 2A shows an exemplary switch that facilitates flow channels.

FIG. 2B shows an example of how switches along a data path can maintain flow state information.

FIG. 3A shows an exemplary fabric header for a data packet.

FIG. 3B shows an exemplary acknowledgement (ACK) packet format.

FIG. 3C shows the relationship between different variables used to derive and maintain state information of a flow.

FIG. 4A shows an example of how flow channel tables can be used to deliver a flow.

FIG. 4B shows an example of an edge flow channel table (EFCT).

FIG. 4C shows an example of an input flow channel table (IFCT).

FIG. 4D shows an example of an output flow channel table (OFCT).

FIG. 5A shows an example where an unfair share of link bandwidth can occur in a network.

FIG. 5B shows an exemplary switch that facilitates packet forwarding with flow-based global fairness.

FIG. 5C shows a flow chart of an exemplary packet forwarding process with flow-based global fairness.

FIG. 6 shows an example of endpoint congestion.

FIG. 7A shows a flow chart of an exemplary process of generating an explicit endpoint-congestion-notification ACK.

FIG. 7B shows an exemplary endpoint congestion management logic block.

FIG. 8 shows a flow chart showing of exemplary process of generating an ACK in response to a packet being dequeued from an output buffer.

FIG. 9A shows a flow chart of an exemplary fine grain flow control (FGFC) process.

FIG. 9B shows an example of a FGFC-enabled network interface controller.

FIG. 10 shows an example of fabric link congestion.

FIG. 11 shows a flow chart of an example process of applying credit-based flow control on a congested fabric link.

FIG. 12 shows an exemplary edge switching system that facilitates flow channels.

FIG. 13 shows an exemplary intermediary switching system that facilitates flow channels.

In the figures, like reference numerals refer to the same figure elements.

DETAILED DESCRIPTION

Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present disclosure. Thus, the present invention is not limited to the embodiments shown.

Overview

The present disclosure describes systems and methods that accommodate dynamic data traffic with fast, effective congestion control by maintaining state information of individual packet streams. More specifically, packets injected into a network of switches can be categorized into streams, which can be mapped to their layer-2, layer-3, or other protocol-specific header information. Each stream can be marked by a distinctive identifier that is local to an input port of a switch, and provided with a stream-specific input buffer, so that each stream can be individually flow-controlled. In addition, packets of a respective stream can be acknowledged upon reaching the egress point of the network, and the acknowledgement packets can be sent back to the ingress point of the stream along the same data path in the reverse direction. As a result, each switch can obtain state information of active packet streams it is forwarding, and can perform highly responsive, stream-specific flow control. Such flow control can allow the network to operate at higher capacity while providing versatile traffic-engineering capabilities.

In this disclosure, packet streams can also be referred to as “packet flows,” or simply “flows.” The data path traversed by a flow, together with its configuration information maintained by switches, can be referred to as a “flow channel.” Furthermore, the terms “buffer” and “queue” are used interchangeably in this disclosure.

FIG. 1 shows an exemplary network that facilitates flow channels. In this example, a network 100 of switches, which can also be referred to as a “switch fabric,” can include switches 102, 104, 106, 108, and 110. Each switch can have a unique address or ID within switch fabric 100. Various types of devices and networks can be coupled to a switch fabric. For example, a storage array 112 can be coupled to switch fabric 100 via switch 110; an InfiniBand (IB) based HPC network 114 can be coupled to switch fabric 100 via switch 108; a number of end hosts, such as host 116, can be coupled to switch fabric 100 via switch 104; and an IP/Ethernet network 118 can be coupled to switch fabric 100 via switch 102. In general, a switch can have edge ports and fabric ports. An edge port can couple to a device that is external to the fabric. A fabric port can couple to another switch within the fabric via a fabric link.

Typically, traffic can be injected into switch fabric 100 via an ingress port of an edge switch, and leave switch fabric 100 via an egress port of another (or the same) edge switch. An ingress edge switch can group injected data packets into flows, which can be identified by flow ID's. The concept of a flow is not limited to a particular protocol or layer (such as layer-2 or layer-3 in the Open System Interface (OSI) reference model). For example, a flow can be mapped to traffic with a particular source Ethernet address, traffic between a source IP address and destination IP address, traffic corresponding to a TCP or UDP port/IP 5-tuple (source and destination IP addresses, source and destination TCP or UDP port numbers, and IP protocol number), or traffic produced by a process or thread running on an end host. In other words, a flow can be configured to map to data between any physical or logic entities. The configuration of this mapping can be done remotely or locally at the ingress edge switch.

Upon receiving injected data packets, the ingress edge switch can assign a flow ID to the flow. This flow ID can be included in a special header, which the ingress edge switch can use to encapsulate the injected packets. Furthermore, the ingress edge switch can also inspect the original header fields of an injected packet to determine the appropriate egress edge switch's address, and include this address as a destination address in the encapsulation header. Note that the flow ID can be a locally significant value specific to a link, and this value can be unique only to a particular input port on a switch. When the packet is forwarded to the next-hop switch, the packet enters another link, and the flow-ID can be updated accordingly. As the packets of a flow traverses multiple links and switches, the flow IDs corresponding to this flow can form a unique chain. That is, at every switch, before a packet leaves the switch, the packet's flow ID can be updated to a flow ID used by the outgoing link. This up-stream-to-down-stream one-to-one mapping between flow ID's can begin at the ingress edge switch and end at the egress edge switch. Because the flow ID's only need to be unique within an incoming link, a switch can accommodate a large number of flows. For example, if a flow ID is 11 bits long, an input port can support up to 2048 flows. Furthermore, the match pattern (one or more header fields of a packet) used to map to a flow can include a greater number of bits. For instance, a 32-bit long match pattern, which can include multiple fields in a packet header, can map up 2{circumflex over ( )}32 different header field patterns. If a fabric has N ingress edge ports, a total number of N*2{circumflex over ( )}32 identifiable flows can be supported.

A switch can assign every flow a separate, dedicated input queue. This configuration allows the switch to monitor and manage the level of congestion of individual flows, and prevent head-of-queue blocking which could occur if shared buffer were used for multiple flows. When a packet is delivered to the destination egress switch, the egress switch can generate and send back an acknowledgement (ACK) in the upstream direction along the same data path to the ingress edge switch. As this ACK packet traverses the same data path, the switches along the path can obtain the state information associated with the delivery of the corresponding flow by monitoring the amount of outstanding, unacknowledged data. This state information can then be used to perform flow-specific traffic management to ensure the health of the entire network and fair treatment of the flows. As explained in more detail below, this per-flow queuing, combined with flow-specific delivery acknowledgements, can allow the switch fabric to implement effective, fast, and accurate congestion control. In turn, the switch fabric can deliver traffic with significantly improved network utilization without suffering from congestion.

Flows can be set up and released dynamically, or “on the fly,” based on demand. Specifically, a flow can be set up (e.g., the flow-ID to packet header mapping is established) by an ingress edge switch when a data packet arrives at the switch and no flow ID has been previously assigned to this packet. As this packet travels through the network, flow IDs can be assigned along every switch the packet traverses, and a chain of flow IDs can be established from ingress to egress. Subsequent packets belonging to the same flow can use the same flow IDs along the data path. When packets are delivered to the destination egress switch and ACK packets are received by the switches along the data path, each switch can update its state information with respect to the amount of outstanding, unacknowledged data for this flow. When a switch's input queue for this flow is empty and there is no more unacknowledged data, the switch can release the flow ID (i.e., release this flow channel) and re-use the flow-ID for other flows. This data-driven dynamic flow setup and teardown mechanism can obviate the need for centralized flow management, and allows the network to respond quickly to traffic pattern changes.

Note that the network architecture described herein is different from software-defined networks (SDN's), which typically uses the OpenFlow protocol. In SDN, switches are configured by a central network controller, and packets are forwarded based one or more fields in the layer-2 (data link layer, such as Ethernet), layer-3 (network layer, such as IP), or layer-4 (transport layer, such as TCP or UDP) headers. In SDN such header-field lookup is performed at every switch in the network, and there is no fast flow ID-based forwarding as is done in the networks described herein. Furthermore, because the OpenFlow header-field lookup is done using ternary content-addressable memory (TCAM), the cost of such lookups can be high. Also, because the header-field mapping configuration is done by the central controller, the setup and tear-down of each mapping relationship is relatively slow and could require a fair amount of control traffic. As a result, an SDN network's response to various network situations, such as congestion, can be slow. In contrast, in the network described herein, the flows can be set up and torn down dynamically based on traffic demand; and packets can be forwarded by a fixed-length flow ID. In other words, flow channels can be data driven and managed (i.e., set up, monitored, and torn down) in a distributed manner, without the intervention of a central controller. Furthermore, the flow ID-based forwarding can reduce the amount of TCAM space used and as a result a much greater number of flows can be accommodated.

Referring to the example shown in FIG. 1, suppose that storage array 112 is to send data using TCP/IP to host 116. During operation, storage array 112 can send the first packet with host 116's IP address as the destination address and a predetermined TCP port specified in the TCP header. When this packet reaches switch 110, the packet processor at the input port of switch 110 can identify a TCP/IP 5-tuple of this packet. The packet processor of switch 110 can also determine that this 5-tuple currently is not mapped to any flow ID, and can allocate a new flow ID to this 5-tuple. Furthermore, switch 110 can determine the egress switch, which is switch 104, for this packet based on the destination (i.e., host 116's) IP address (assuming switch 110 has knowledge that host 116 is coupled to switch 104). Subsequently, switch 110 can encapsulate the received packet with a fabric header that indicates the newly assigned flow ID and switch 104's fabric address. Switch 110 can then schedule the encapsulated packet to be forwarded toward switch 104 based on a fabric forwarding table, which can be computed by all the switches in fabric 100 using a routing algorithm such as link state or distance vector.

Note that the operations described above can be performed substantially at line speed with little buffering and delay when the first packet is received. After the first packet is processed and scheduled for transmission, subsequent packets from the same flow can be processed by switch 110 even faster because the same flow ID is used. In addition, the design of the flow channels can be such that the allocation, matching, and deallocation of flow channels can have substantially the same cost. For example, a conditional allocation of a flow channel based on a lookup match and a separate, independent deallocation of another flow channel can be performed concurrently in nearly every clock cycle. This means that generating and controlling the flow channels can add nearly no additional overhead to the regular forwarding of packets. The congestion control mechanism, on the other hand, can improve the performance of some applications by more than three orders of magnitude.

At each switch along the data path (which includes switches 110, 106, and 104), a dedicated input buffer can be provided for this flow, and the amount of transmitted but unacknowledged data can be tracked. When the first packet reaches switch 104, switch 104 can determine that the destination fabric address in the packet's fabric header matches its own address. In response, switch 104 can decapsulate the packet from the fabric header, and forward the decapsulated packet to host 116. Furthermore, switch 104 can generate an ACK packet and send this ACK packet back to switch 110. As this ACK packet traverses the same data path, switches 106 and 110 can each update their own state information for the unacknowledged data for this flow.

In general, congestion within a network can cause the network buffers to fill. When a network buffer is full, the traffic trying to pass through the buffer ideally should be slowed down or stopped. Otherwise, the buffer could overflow and packets could be dropped. In conventional networks, congestion control is typically done end-to-end at the edge. The core of the network is assumed to function only as “dumb pipes,” the main purpose of which is to forward traffic. Such network design often suffers from slow responses to congestions, because congestion information often cannot be sent to the edge devices quickly, and the resulting action taken by the edge devices cannot always be effective in removing the congestion. This slow response in turn limits the utilization of the network, because to keep the network free of congestion the network operator often needs to limit the total amount of traffic injected into the network. Furthermore, end-to-end congestion control usually is only effective provided that the network is not already congested. Once the network is heavily congested, end-to-end congestion control would not work, because the congestion notification messages can be congested themselves (unless a separate control-plane network that is different from the data-plane network is used for sending congestion control messages).

In contrast, the flow channels can prevent such congestion from growing within the switch fabric. The flow channel mechanism can recognize when a flow is experiencing some degree of congestion, and in response can slow down or stop new packets of the same flow from entering the fabric. In turn, these new packets can be buffered in a flow channel queue on the edge port and are only allowed into the fabric when packets for the same flow leave the fabric at the destination edge port. This process can limit the total buffering requirements of this flow within the fabric to an amount that would not cause the fabric buffers to become too full.

With flow channels, the switches have a reasonably accurate state information on the amount of outstanding in-transit data within the fabric. This state information can be aggregated for all the flows on an ingress edge port. This means that the total amount of data injected by an ingress edge port can be known. Consequently, the flow channel mechanism can set a limit on the total amount of data in the fabric. When all edge ports apply this limit action, the total amount of packet data in the entire fabric can be well controlled, which in turn can prevent the entire fabric from being saturated. The flow channels can also slow the progress of an individual congested flow within the fabric without slowing down other flows. This feature can keep packets away from a congestion hot spot while preventing buffers from becoming full and ensuring free buffer space for unrelated traffic.

Operation of Flow Channel

In general, flow channels can define a path for each communication session across the switch fabric. The path and amount of data belonging to each flow can be described in a set of dynamically connecting flow tables associated with each link of the switch fabric. On every ingress port, edge and fabric, a set of flow channel queues can be defined. There can be one queue for each flow channel. As packets arrive, they either can be assigned to a flow channel on an edge port, or have been assigned to a flow channel by the link partner's egress fabric port on a fabric ingress port. The flow channel information can be used to direct the packets into the appropriate flow channel queue.

FIG. 2A shows an exemplary switch that facilitates flow channels. In this example, the switch can include a crossbar switch 202. Crossbar switch 202 can have a number of input ports, such as input port 204, and a number of output ports, such as output 208. Crossbar switch 202 can forward packets from an input port to an output port. Each input port can be associated with a number of input queues, each assigned to a different incoming flow arriving on that input port. For example, data arriving on a given port of the switch can first be separated, based on their individual flows, and stored in flow-specific input queues, such as input queue 206. The packets stored in the input queues can be dequeued and sent to crossbar switch 202 based on scheduling algorithms designed to control congestions (described in more detail in later sections). On the output side, once a packet passes crossbar switch 202, it can be temporarily stored in an output transmission queue, such as output transmission queue 210, which can be shared by all the flows leaving on the same output port. Meanwhile, before a packet is dequeued from the output transmission queue and transmitted on the outgoing link, the packet's header can be updated with the flow ID for the outgoing link. Note that this hop-by-hop flow ID mapping can be done when the first packet in the flow travels across the network. When the packet reaches the next-hop switch, the packet can be stored again in a flow-specific input queue and the same process can be repeated. Note that a flow ID is used to distinguish between flows traveling on the same fabric link, and can be typically assigned by the transmitter end of this link, which is the output port of the switch that is transmitting onto this link.

By providing flow-specific input queues, the switch can allow each flow to move independently of all other flows. The switch can avoid the head-of-queue blocking problem, which is common with shared input buffers. The flow-specific input queue also allows the packets within a single flow to be kept in order. When a flow passes through the switches, a flow-specific input queue on each input port can be allocated for this flow and these input queues become linked, effectively forming one long queue that reaches across the entire fabric for this flow, and the packets of this flow can be kept in order.

The progress of successful delivery of packets belonging to a flow can be reported by a sequence of ACKs generated by the edge port of an egress switch. The ACK packets can travel in the reverse direction along the data path traversed by the data packets and can be forwarded by the switches according to the forwarding information maintained in flow tables. As ACK packets travel upstream, they can be processed by each switch's input queue manager, which can update the corresponding flow's state information based on information carried by the ACK packets. The ACK packets can have a type field to provide advanced information about the downstream data path, such as congestions. A switch's input queue manager can use this information to make decisions, such as throttling the transmission rate or changing the forwarding path, about the pending data packets currently buffered in its input queues. In addition, the input queue manager can update the information carried in an ACK packet based on state information of a buffered flow, so that the upstream switches can make proper decisions. For example, if an input queue for a given flow is experiencing congestion (e.g., the amount of data in the queue is above a predetermined threshold), the input queue manager can update an ACK packet that is being forwarded to the next upstream switch to include this congestion information.

If an ACK corresponds to the last packet of a flow, a switch can determine that there is no more unacknowledged data for that flow. Correspondingly, the switch can free the flow channel by removing the corresponding entry in the flow table.

As mentioned above, the input queue manager at each switch can maintain information about transmitted but unacknowledged data of a given flow. FIG. 2B shows an example of how switches along a data path can maintain flow state information. In this example, the data path taken by a flow can include switches 222, 224, and 226. The amount of transmitted but unacknowledged flow data can be indicated by a variable “flow_extent,” which can be measured in number of fixed-length data units, such as 256 bytes. Furthermore, flow_extent and other flow state information can be maintained by a switch's input queue manager, which can continuously monitor all the flow-specific queues.

In the example in FIG. 2B, the value of flow_extent at the input queue manager of switch is 1, because there is one unit of data that has been sent out of the input queue and forwarded through the crossbar switch. Note that a data packet sent by an input queue might be temporarily buffered in the output transmission buffer due to the scheduling of all the data packets to be transmitted via an output link. When such a packet is buffered in the output port's transmission buffer, the packet can still be considered by the input queue as transmitted for the purpose of updating the flow_extent value.

Correspondingly, because the input queue for the given flow at switch 226 has six queued data units, and two additional data units are in transit between switches 224 and 226, the flow_extent value at switch 224 is 9. Similarly, the flow_extent value at switch 222 is 13, because there are three data units stored in the input queue at switch 224 and one data unit in transit between switches 222 and 224.

In general, a flow channel can remain allocated to a single flow until all the ACKs for all the packets sent on the flow channel have been returned. This means that flow channel table entries can remain active for longer near the fabric ingress edge port than near the egress edge port. If a single packet is injected into the network, a flow channel can be allocated for the ingress edge port and then another flow channel can be allocated for the next fabric link the packet traverses and so on, until the last flow channel is allocated when the packet reaches the last fabric link. Each allocation can generate a flow ID, denoted as variable “flow_id,” to identify the entries of the flow tables of the fabric link. (More details on flow channel tables are provided in the description below in conjunction with FIG. 4A.) This first packet may cause the allocation of a different flow_id, on each of the fabric links the packet traverses across the switch fabric.

At the input queue of each switch, the flow channel table entries can indicate each flow's state information, including the flow_extent value, from this point downstream to the flow's egress destination edge port. Packets received on the local input port can increase this flow_extent value by the amount of incoming data, and ACKs can reduce the flow_extent by the amount of acknowledged, delivered data.

When a packet reaches the final destination egress port, an ACK packet can be generated and returned for that packet. This ACK can be routed using the data path information stored in the corresponding entry of the flow channel tables at every switch along the data path. Optionally, the ACK packet itself does not need to carry path information and therefore can be small and light weight. If no other data packet is sent on the flow, the ACK can release each flow channel in the reverse order. Once released, the flow channel at each switch can be allocated to a different flow.

If another packet follows the first packet on the same flow, the ACK corresponding to the second packet would need to be received before the flow channel can be released at a given switch. In one embodiment, the flow channel can only be released when ACKs for all the transmitted packets of the same flow have been returned.

Typically, various protocols may require in-order packet delivery. The flow channels can be used to guarantee this delivery order, even when the fabric uses adaptive routing for load balancing across multiple data paths. If packets between an ingress edge port and an egress edge port, perhaps in a different switch on the far side of the fabric, are injected at a very low rate, then each packet injected could reach its destination and return an ACK back to the source before the next packet is injected. In this case, each packet can be a lead packet and free to take any path across the fabric, using the best available dynamic adaptive routing choice. This is possible because the first packet can define the flow's path through the fabric.

Now assume that the packet injection rate is increased slightly to the point where the next packet of the same flow is injected before the current packet's ACK has returned to the source. The second packet can pass the ACK of the first packet somewhere along the flow's data path. Beyond this passing point, the ACK will have released the flow channels allocated to the first packet, because the flow_extent value associated with the first packet is returned to zero when the ACK is processed by the flow channel's logic. Meanwhile, the second packet can now define a new flow, because it is again causing flow channels to be allocated on each of the subsequent fabric links. This second packet, while it is causing flow channels to be allocated beyond the passing point, can be forwarded to a different path based on dynamic adaptive routing. On the other hand, before the passing point, the second packet can extend the outstanding flow created by the first packet to include the second packet. This means the first packet's ACK may not reduce the flow_extent value to zero and the flow channels may remain active before the passing point. It also means that the second packet may follow the exact path taken by the first packet up to the passing point. Note that while it is following the previous packet, the second packet cannot arrive at the egress edge port before the first packet does, and therefore correct packet order can be maintained.

If the injection rate for this flow is increased further, the second packet will pass the first packet's ACK at a location closer to the destination edge port. It is also possible that a third, fourth, fifth, or additional packet may enter the fabric before the first packet's ACK is returned to the source edge port, depending on the data packet injection rate of this flow and the data packet-ACK round trip delay. The maximum packet rate can depend on the size of the packets and the bandwidth of the links. The round trip delay of the data packet and ACK can be an important parameter for a fabric implementation and can be used along with the maximum packet rate to calculate the maximum required number of flow channels for each link. Ideally, a design can provide a reasonable number of unallocated flow channels regardless of the traffic pattern. The demand for the number of flow channel can be high when a large number of packets arriving at an ingress edge port have different destinations and these packets have small sizes and high injection rates. In the most extreme case, each packet could be allocated a different flow channel. These flow channels are freed when the packets' ACKs are returned. Correspondingly, the number of flow channels needed can be calculated as ((Packet rate)*(Average packet to ACK round trip latency)).

Note that packet rate on a single flow channel is not to be confused with packet rate on a link. If the traffic pattern is such that many small packets are being sent to different destinations, then successive packets sent onto the link can have different destinations. This means that each packet could belong to a different flow and could be the only packet to use the corresponding flow channel. In this example, the link can experience a high packet rate, but the packet rate of individual flows can be low. Optionally, a number of ACKs (e.g., 48 ACKs) can be aggregated together into a single ACK frame for transmission over a link and protected by a Frame Check Sequence (e.g., a 32-bit FCS). For example, the ACKs can occupy 25 bits each, and there can be a 9-byte overhead to the frame. That is, the overhead per ACK on a full size frame is approximately 9/(25/8*48)*100%=6%. The logic can optimize the number of ACKs per frame so an ACK does not need to wait too long to be aggregated when the ACKs are arriving slowly. For example, the ACK aggregation logic block can use three timers to manage ACK transmission based on the activity of an outgoing link. These timers can be started when a new ACK arrives at the ACK aggregation logic block. If the outgoing link is idle, a first timer, which can for example be set at 30 ns, can be used to hold the ACK while waiting for additional ACKs to arrive. When this timer expires, all the ACK received within the corresponding time window can be aggregated into one frame and transmitted onto the outgoing link. If the outgoing link is busy, a second timer, which can for example be set at 60 ns, can be used to wait for additional ACKs. Using this second timer can allow more ACKs to be aggregated into a single frame, and this frame can be transmitted only if a predetermined number of ACKs are collected. Note that due to the Ethernet framing constrains, some numbers of ACKs in a single frame can use less wire bandwidth per ACKs than other numbers of ACKs. If no efficient number of ACKs are collected, and the outgoing link remains busy sending normal data packets, then a third timer, which can for example be set at 90 ns, can be used. Once this third timer expires, all the ACKs that have been collected can be aggregated in a frame and transmitted onto the link. By using these three timers, the system can significantly reduce the overhead of sending ACKs on the outgoing link.

In some examples, the ingress edge port of a switch can encapsulate a received data packet with a fabric header, which allows the packet to be forwarded using flow channels. FIG. 3A shows an exemplary fabric header for a data packet. The fabric header can include a flow_id field, which can identify the flow channel, and a “data_flow” field, which can indicate the progression of the entire flow.

When a data packet is delivered to its destination, at least one ACK can be generated. FIG. 3B shows an exemplary ACK packet format. An ACK packet can include a “flow_id” field, an “ack_flow” field, an “ACK type” field, and a cyclic redundancy check (CRC) field. The flow_id field can indicate the flow this ACK packet belongs to. The ack_flow field can indicate the data packet to which this ACK packet acknowledges. Recall that each switch can maintain a flow_extent value which indicates the amount of transmitted but unacknowledged data. The value of flow_extent can be derived as flow_extent=data_flow−ack_flow, wherein data_flow value is taken from the last transmitted data packet.

The ACK type field can indicate different types of ACKs. As mentioned above, during normal operation, when a data packet is delivered to the destination edge port, a regular ACK packet can be generated and sent back to the source. Correspondingly, the ACK type field in the ACK packet can indicate a normal ACK. When congestion occurs in the fabric, the ACK type field can be used to indicate various types and severity of congestion, such as a new congestion, a persistent congestion, or a severe congestion at the egress edge port that calls for rerouting of the flow. In addition, under special circumstances such as the presence of a severely congested fabric link, dropped packets, or link error, an ACK can also be generated by an intermediate switch that is not the final destination, and the ACK type field can be used to notify upstream switches of different types of network condition. Other additional fields can also be included in an ACK packet.

FIG. 3C shows the relationship between different variables used to derive and maintain state information of a flow. In this example, a switch can use the variable “total_extent” to track the total amount of unacknowledged transmitted data and data currently queued at the switch. The value of total_extent can equal the sum of flow_extent, which is the amount of transmitted and unacknowledged data, and queue_extent, which is the amount of data stored in the input queue for the corresponding flow. The variable “ack_flow” can indicate the data position that corresponds to the latest ACK for this flow. The variable “data_flow” can indicate the position of the next data packet to be transmitted, which also corresponds to the data packet stored at the head of the input queue. The variable “next_data_flow” can indicate the position of the next data packet that the switch can expect to receive from the upstream switch. Note that queue_extent=next_data_flow−data_flow, and flow_extent=data_flow−ack_flow.

In some examples, flow channel tables can be used to facilitate flow channels throughout a fabric is. Flow channel tables are data structures that store the forwarding and state information for a given flow at the port of a switch. FIG. 4A shows an example of how flow channel tables can be used to store state information associated with multiple flows. This state information can be specific to each flow and efficiently stored in a table. Assume that a source host 402 is sending data packets to a destination host 404 via a fabric. The data path traversed by the data packets can include an ingress edge switch 406, intermediate switches 408 and 430, and egress edge switch 432.

When a packet arrives on an ingress edge link 403 of switch 406, the packet's header can be analyzed by an address translate logic block 410. Address translate logic block 410 can determine the destination fabric address of the egress switch (which in this case is switch 432) based on the packet's Ethernet, IP, or HPC header information. Note that header information associated with other protocols or a combination of different protocols can also be used by address translate logic block 410. The fabric destination address determined by address translate logic block 410 can then be used to perform a lookup in an edge flow channel table (EFCT) 412. EFCT 412 can perform a lookup operation for the packet using the packet's fabric destination address and optionally additional values extracted from the packet's header, which can be referred to as a match pattern. EFCT 412 can compare the packet's match pattern against stored match patterns of all existing allocated flows. If a match is found, then this packet is part of an existing flow and the previously allocated flow ID can be returned for this packet. If no match is found, a new flow ID can be allocated for this packet, and a match pattern can be added to EFCT 412. In other words, EFCT 412 can be used to determine whether a flow channel already exists for the incoming packet, or whether a new flow channel needs to be allocated. In addition to the destination fabric address, other packet header information such as traffic class, TCP or UDP port number, and process or thread ID can be used to map or allocate flow IDs.

The flow ID obtained by EFCT 412 can then be used as an index to map to an entry in an input flow channel table (IFCT) 414. Each entry in IFCT 414 can be indexed by a flow ID and store state information for the corresponding flow. An entry in IFCT 414 can store the values of next_data_flow, data_flow, and ack_flow (see FIG. 3C) associated with a flow. In addition, an IFCT entry can store other parameters for congestion control and dynamic routing for a flow.

The flow ID can also be used to identify or allocate a flow-specific input queue in which the incoming packet can be temporarily stored. The state information for a particular queue, as well as parameters for monitoring and controlling the queue (such as threshold for detecting congestion) can be stored in the corresponding entry in IFCT 414. An input queue management logic block can determine when a packet can be dequeued from the input queue and sent to a data crossbar switch 413 based on flow-control parameters stored in the entry of IFCT 414.

When a packet is dequeued from the input queue and sent through crossbar switch 413 to an output port, the packet is sent with the input port number on which it has arrived at switch 406. When the packet reaches an output port's transmission buffer, the packet's header can be updated, based on the packet's flow ID and input port number, with a new flow ID to be used by the next-hop switch (i.e., switch 408) for the same flow. This is because each link, in each direction, can have its own set of flow channels identified by their respective flow IDs. The mapping from the incoming flow ID to the outgoing flow ID used on the next link can be done by looking up an output flow channel table (OFCT) 416. OFCT 416 can perform a lookup using a match pattern that is a combination of the local input port number corresponding to link 403 and the packet's flow ID which is produced by EFCT 412. If a match is found, then the flow has already been defined, and the packet's flow ID is updated with the value corresponding to the match pattern (this new outgoing flow ID is to be used by the downstream next-hop switch 408). If a match is not found, then a new flow channel can be allocated with a new, outgoing flow ID, which can be mapped to the input port number and the previous, incoming flow ID. An entry including the outgoing flow ID, input port number, and incoming flow ID can be stored in OFCT 416.

In the case where the packet is the first packet in the flow, a lookup in OFCT 416 would not produce any mapping. In turn, OFCT 416 can allocate for the packet a flow channel with a flow ID to be used by the input port and IFCT 418 on switch 408. This new flow channel, identified by its flow ID, can be added to the packet header for transmission onto link 417, and can be used by the link partner's (which is switch 408) IFCT 418 to access the flow channel's congestion information. As before, OFCT 424 can further generate a new flow channel if no match is found, using the match pattern of its immediate upstream input port number and flow ID associated with link 417. OFCT 424 can then allocate a new flow channel identified by a new flow ID. Note that OFCT 416 can also function as a forwarding table for ACKs of this flow in the upstream direction. After being forwarded upstream from switch 408 to switch 406, the ACK packet can be updated with the flow ID associated with edge link 403 and forwarded to the appropriate input port on switch 406 as indicated by the corresponding entry in OFCT 416. The ACK packets can be forwarded to the input port by an ACK crossbar switch 415 in the upstream direction.

Subsequently, when the packet arrives at switch 408, its flow ID can be used to identify an input queue to use and to determine an entry in IFCT 418. If the packet's flow ID has not been previously allocated by switch 408, a new input queue can be provided and a new entry in IFCT 418 can be created. From this point onward, a similar process can be performed to forward the packet across switches 408 and 430 until the packet reaches egress switch 432.

When the packet reaches switch 432, after the packet is forwarded by a data crossbar switch 423, an ACK generator logic block 420 can generate an ACK packet based on the packet's flow ID and input port number. This ACK packet can then be forwarded in the upstream direction by an ACK crossbar switch 422. At the same time, based on the ACK packet, an IFCT 421 can update the state information for the flow in the corresponding table entry. When the ACK packet reaches switch 430, an OFCT 419 can be looked up to determine the upstream flow ID and upstream input port to which the ACK packet is to be forwarded. The ACK packet can then have its flow ID updated and be forwarded to the appropriate input port in the upstream direction. As the ACK packet traverses the data path upstream in a similar way, the IFCT at each switch can update its table entry for the flow based on the ACK.

Note that the flow_extent variable can be an important parameter, because it represents the total amount of downstream packet data for a flow. A flow channel is considered free to be reallocated to another flow when the flow_extent of an entry is zero. In general, on receipt of a new packet, the input logic can make a request to send data to an output port. The selected output port can be a function of the flow_extent stored in the IFCT. If flow_extent is zero, there are no packets downstream in the flow to the destination egress edge port. As a result, the switch can use a load based adaptive route selection to choose any valid path that leads to the destination. In a multi-path network, dynamic adaptive routing can be done without the packet being reordered. If flow_extent is not zero, and if in-order delivery is required, the packet can use the same route taken by previous packets. The IFCT can have a field that stores a previous output port number, which is loaded when a packet request is made to an output port and can be used to ensure a connection to the previously used output port.

As mentioned before, the flow channels can use a match function to recognize packets belonging to an existing flow. Received Ethernet frames or other types of packets can be parsed in real time when the frame or packet is received on an ingress edge port and some fields of the packet header can be used for a lookup in a CAM or Ternary Content Addressable Memory (TCAM). If there is a match, the match address can become the flow ID used to select a flow channel. When no match occurs, the switch hardware can load the pattern that fails to match directly onto a free line of the CAM, which can be done without additional delay. As a result, any following packet can be matched to this new entry without significant amount of buffering. The free entry chosen becomes the new flow ID for the new flow channel entry. Note that no external software intervention is required for the loading of the new entry. The process can be completed autonomously by the switch hardware.

The de-allocation of flow IDs and corresponding CAM match lines can also be automatically performed by the hardware when the last ACK is returned for the flow. The de-allocation can occur in hardware with respect to potentially matching new packets, without external software intervention.

In some examples, ingress edge switch 406 can include a fine-grain flow control logic block 434, which can communicate with a network interface controller (NIC) 401 on host 402 to apply flow control on a per-flow basis. More details on find-grain flow control are provided below in conjunction with the description on congestion management.

FIG. 4B shows an example of an EFCT. In this example, an EFCT can include a data_flow field 454, an ACK_flow field 456, and optionally additional fields. The EFCT can be associated with an input port, and entries in the EFCT can be indexed by flow_ID values, such as flow_ID 452. In one embodiment, the match pattern field can reside in the match function logic block, which can include a CAM or TCAM. The match function logic block can use the match pattern to generate the flow_ID value, which in turn can be used as an index to the corresponding EFCT entry. From this EFCT's perspective, the flow_extent (i.e., data_flow−ack_flow) can include all the unacknowledged data downstream of this table, which can include the local flow_queue plus the corresponding IFCT's flow_extent value.

FIG. 4C shows an example of an IFCT. In this example, an IFCT can be associated with an input port, and can include a follow_port field 466, a next_data_flow field 468, a data_flow field 470, an ACK_flow field 472, an ep_congestion field 474, an upstream metering (UM) flag field 477, a downstream metering (DM) flag field 478, and optionally additional fields. An incoming packet's flow_ID value, such as flow_ID 464, can be used as an index to look up the output port number, which is indicated by follow_port field 466, and the state information associated with the corresponding flow. Congestion-control information associated with endpoint congestion (such as ep_congestion field 474) and (hop-by-hop credit-based flow control (such as UM flag field 477 and DM flag field 478), which is described in more detail later in this document, can also be stored in the IFCT. The IFCT can further store information related to dynamic routing associated with different flows.

FIG. 4D shows an example of an OFCT. In this example, an OFCT can be associated with an output port, and can include an input_port field 482, an input_port_flow_ID field 484 (which corresponds to a packet's existing flow_ID upon its arrival at an input port), a data_flow field 486, an ACK_flow field 488, and optionally additional fields. Data_flow field 486 and ACK_flow field 488 can be used to determine the value of flow_extent from this OFCT onward. The combination of input_port field 482 and input_port_flow_ID field 484 (which can also be referred to as “incoming flow_ID”) can be used to determine or allocate the outgoing flow_ID of a packet that is ready for transmission onto the outgoing link corresponding to this OFCT. In one embodiment, the outgoing flow_ID values, such as flow_ID 486, can be used as an index to look up entries in the OFCT.

Congestion Management

As described above, each flow at a given switch can have its own private queue of packets. This configuration facilitates separate flow control for each flow. As a result, the network can remain mostly lossless, and one flow using a link can be blocked without blocking any of the other flows using the same link. Unlike a traditional packet switched network, congestion in one part of the network can only affect the flows that are contributing to the congestion. For example, in a conventional network, the buffers before a congested link can quickly fill up with the packets causing the congestion. This in turn can force the switch to issue a pause command or use some other flow control method to prevent neighboring switches from sending packets toward the congested link. Consequently, the packets causing congestion can be stopped or slowed down, and all other packets, which may not be heading to the congested link, can also be stopped or slowed down. As a result, the congestion could spread sideways and increase the size of the saturation tree from a topological perspective.

In contrast, with flow channels, the load corresponding to flows contributing to congestion can be reduced on the links leading up to the congestion. This reduction of load can allow other flows that are sharing these links to use more link bandwidth and deliver their payload more quickly, while only the packets contributing to the congested link are slowed down.

Typically, conventional networks can operate normally provided the network load is not at or near full capacity. This can be the case for small or medium sized networks most of the time. With large or very large networks operating with multiple bandwidth-hungry applications, however, at any point in time part of the network can be saturated with traffic load. Under these circumstances, unfair packet delivery could occur even if individual switches implement locally fair policies.

FIG. 5 shows an example where an unfair share of link bandwidth can occur in a network. In this example, each of the sources A to K is trying to send a stream of packets to destination L, forming an incast scenario where multiple sources are sending packets to a single destination. Source nodes A, B, and C are coupled to switch 502; source nodes D, E, and F are coupled to switch 504; source nodes G, H, and I are coupled to switch 506; and source nodes and J and K, and destination node L are coupled to switch 508. Assume that each switch has a fair arbitration policy of selecting an equal number of packets from each of its input ports to any particular output port. However, as can be seen in FIG. 5, sources closer to the destination can receive a much higher proportion of the final link bandwidth than sources the traffic of which needs to pass through more stages of switching. Switch 508 has three sources of incoming data from nodes J, K and switch 506, and can divide the bandwidth on the outgoing link to node L equally among each source. Hence, nodes J, K can each take 33.3% of the bandwidth on the outgoing link toward destination node L.

The next nearest switch, which is switch 506, can do the same and so on. In this example, with only four stages of switches and only three or four inputs on each stage, and only with a total of 11 inputs trying to send to the destination node L, three input sources (nodes A, B, and C) only take 1/48 the bandwidth taken by two other input sources (nodes J and K) on the outgoing link toward destination node L. Hence, even with locally fair arbitration policies, nodes that are far away from the destination can suffer from very unfair treatment. A more realistic network topology can involve more switching stages, greater numbers of switch inputs, and more sources trying to send to a single destination. A moderate-sized incast could result in six orders of magnitude difference between the delivered bandwidths among different sources.

The unfairness problem described above is often caused by the fact that the arbitration policies implemented by a switch are based on the input ports. That is, the bandwidth throttling is done with a per-port granularity. In contrast, by facilitating flow channels and implementing flow-specific throttling, a network can significantly reduce the amount of unfairness among different flows. For example, in the scenario shown in FIG. 5, when the switches implement a fair per-flow bandwidth allocation policy, all the eight source nodes can take substantially equal share of the bandwidth of the edge link between switch 508 and destination node L. More specifically, assume that switch 508 is delivering 11 flows to node L. Each flow can take 1/11 or approximately 9.1% of the egress edge link's bandwidth. With flow channels the links from each of the nodes can use 1/11 of the egress edge link's bandwidth, the link from switch 502 to switch 504 has 3 flows, leaving 8/11 of the entire bandwidth available for other traffic. The next link between switch 504 and switch 506 has 6 flows, leaving 5/11 of the egress edge link's bandwidth for other traffic. The link between switch 506 and switch 508 has 9 flows, leaving 2/11 of the bandwidth on the egress edge link free.

When the arbitration is made on flows instead of ports, the bandwidth each flow achieves can be determined by the switch coupled to the link with the bottleneck. Other switches can be just forwarding packets at the rate decided by the switch coupled to the bottleneck link. In the example in FIG. 5, switch 508 can be responsible for setting the per-flow data rate of all other switches. With flow channels, global network fairness can be achieved with a locally fair arbitration policy implemented on flows in each switch.

This unfairness problem is not generally recognized or fully understood within the computer industry. Even within the HPC community that is normally able to spend more effort trying to optimize applications, hardware and systems, the focus is often directed towards minimum packet delivery latency when usually the biggest impact to scalability is caused by large occasional network latencies incurred on a small percentage of the packets. The finer the granularity of the application, the more significant this tail latency becomes.

Flow channels offer a solution to the global fairness problem that is based on the flows and knowledge that the final fabric arbitration is being made to the destination egress edge port.

Using the following method, during a saturating incast, all sources contributing to the incast can achieve similar bandwidth regardless of the size of frames or number of hops to the destination. In one embodiment, the system can allocate a substantially equal amount of buffer space on the egress edge port available for each flow contributing to an incast. The system can use this allocated buffer space to arbitrate the packets heading into the egress edge port that is the focus of an incast. This arbitration mechanism can be fair between flows, rather than between ports. This means a port receiving many flows that are each destined into the same incast egress port can gain a larger proportion of the final egress edge link's bandwidth than a port with fewer or only one flow into the incast egress port.

By equalizing the bandwidths of each flow, the worst-case tail latency (where a small proportion of communications take a large amount of time to complete) can be significantly reduced, and data transfer times can also be reduced and more predictable.

The flow channels can be used for managing delivery of data to mitigate the worst effects of an incast at any port of a network, where the ports have a plurality of inputs which are capable of contributing to the incast. This is done by restricting the amount of packet data that can be forwarded to an output port for the purposes of fair arbitration based on the packets belonging to a particular flow. This is done so that the output will select a packet to egress based on it being part of a flow, rather than it coming from a particular port, so that an ingress port with many flows can present more packet data for arbitration consideration than an ingress port with fewer flows, and in so doing will gain a larger proportion of the available egress bandwidth.

By providing a much fairer flow based arbitration policy, extreme tail latencies of individual packets can also be substantially reduced. For large system installations, controlling the maximum latencies through a network is often a major concern for architects. Often, this can only be achieved by restricting the input bandwidth into a network to a small percentage of the peak bandwidth. For example, an input bandwidth limit of 20% of the peak bandwidth can be typical for large datacenters. With flow channels and proper control mechanisms, in contrast, it is now possible to build a network that does not impose such restrictions.

Flow-based arbitration with fairness can deliver significant benefits for TCP streams. TCP is a reliable transport protocol and has been designed to deliver a high error-free data rate a network can offer. TCP attempts to measure the current state of congestion in the network with an initial round trip latency test and then inject packets into the network at the maximum rate the network can accommodate without significantly increasing that round trip latency. For a network that is predominantly filled with only TCP streams this approach can deliver good results. However, many applications and system services use UDP for their transport protocol. UDP has less control and only provides a best effort delivery, relying on the application to provide transmission control. UDP streams can be multicast and transmitted at line rate from a NIC without any level of throttling if the application so desires. In practice, applications using UDP tend to be more considerate but there are no guarantees. In a traditional network, a TCP stream sharing a link with a saturating UDP stream typically throttles its transmission rate to a low level, sometimes close to zero. It may stay at near zero for as long as the UDP stream is consuming the remaining bandwidth. However, with flow channels, the TCP and the UDP streams can be separately managed. Faced with a saturating UDP stream, the bandwidth of the TCP packets may decrease but only to the point where the TCP and UDP streams each have 50% of the shared link bandwidth. If there are more UDP or TCP streams they can also consume their fair share of the total link bandwidth. If a particular stream (TCP or UDP) is sharing with many other streams at one point in the network, it can only consume the bandwidth seen at that bottleneck on all the other links the flow uses in other places in the network. This means that even a line-rate broadcasting UDP stream can be accommodated alongside many TCP and other UDP streams without a catastrophic collapse of the whole network.

TCP has a number of controls that are usually optimized for a typical lossy Ethernet network. These controls include parameters to control the rate at which the bandwidth is increased on the slow start. Transmission between ports connected to a flow-channel-enabled fabric does not need the slow start. The flow channels can be lossless and deliver all the packets, in the right order, using the best-choice path across the network. Higher bandwidth on short-lived TCP connections across the fabric can also be provided with changes to these parameters.

FIG. 5B shows an exemplary switch that facilitates packet forwarding with flow-based global fairness. In this example, a switch 522 can include a number of input ports, such as input port 524, a crossbar switch 526, and a number of output ports such as output port 532. Each output port can be associated with an output buffer, which can be controlled by a control logic block. For example, output buffer 530 can buffer packets to be transmitted via output port 532. Arbitration of packets stored in output buffer 530 can be done by an output buffer control logic 528.

During operation, packets are received via the input ports, and forwarded to their corresponding output ports by crossbar switch 526. All the packets to be transmitted to an output port can be buffered in the output buffer coupled to that output port. In one embodiment, output buffer control logic block 528 can allocate space in output buffer 530 among all the flows to be transmitted via output port 532 based on a predetermined configuration. For example, in one scheme, every flow can receive a substantially equal amount of buffer space in output buffer 530. Output buffer control logic block 528 can further arbitrate the scheduling of transmission for the buffered packets based on the same scheme, thereby enforcing a globally fair allocation of output port 532's bandwidth among all the flows buffered in output buffer 530. Note that other bandwidth allocation schemes are also possible to facilitate differentiated service among different traffic classes.

FIG. 5C shows a flow chart of an exemplary process of forwarding packets with flow-based global fairness. During operation, a switch can receive a number of packets to be transmitted via an output port, and store these packets in the corresponding output buffer (operation 542). The system can further allocate buffer space in the output buffer to the packets based on the flows these packets belong to (operation 544). Note that the allocation can be done based on a predetermined configuration or scheme, which can be for example a fairness-based Round robin scheme. Subsequently, the switch can arbitrate the packets stored in the output buffer accordingly and dequeuer the packets for transmission (operation 546). The switch can then transmit the dequeued packets to the output port (operation 548).

In addition to fairness, another challenge faced by network architects is congestion. In general, two types of congestions can occur in a network. The first type is endpoint congestion, where an egress edge link coupled to a destination device is congested. The second type is fabric link congestion, where an intermediate fabric link is congested.

FIG. 6 shows an example of endpoint congestion. In this example, two source hosts 602 and 604 are sending data to a destination host 606. Traffic from source hosts 602 and 604 converges at edge switch 610, and an egress edge link 608 between switch 610 and host 606 can become congested. This congestion scenario can typically occur with incast, where multiple sources are sending traffic to a single destination. Congestion can occur when egress edge link reaches its full data rate capacity, or when destination host 606 cannot process all the incoming packets at a sufficiently fast rate. In any case, the output transmission buffer on switch 610 that is coupled to link 608 can experience an increase in its stored data amount when endpoint congestion occurs.

A switch can detect and mitigate endpoint congestion by monitoring the output buffer on an egress edge link and by sending ACKs with congestion information to upstream switches and source nodes. More specifically, the output buffer coupled to an egress edge link can monitor the state of the buffer and detect congestion when certain criteria are met. When a packet arrives at or leaves an output buffer, the output buffer can compute three congestion-detection parameters, such as: (1) the amount of data stored in the buffer, (2) the number of packets stored in the buffer, and (3) the rate of change of buffer depth (amount of data stored in the buffer). Three threshold values can be set respectively for these three monitored parameters, although more or less can be set. Congestion is considered to be present when at least one of these parameters exceeds the corresponding threshold.

When congestion is detected, the switch can generate and transmit an endpoint-congestion-notification ACK corresponding to the packet that has just entered the output buffer. The ACK can include a value indicating the severity of the congestion. Note that this endpoint-congestion-notification ACK is not intended to notify upstream switches of the successful delivery of the packet, but to inform them of the presence and degree of congestion at the egress edge link. (In fact when this endpoint-congestion-notification ACK is sent, the packet may still be stored in the output buffer waiting to be transmitted onto the egress edge link.) This fast, explicit congestion notification mechanism allows the switches to act quickly on a specific flow contributing to the congestion.

In addition, the output buffer can update the congestion-detection parameters when a packet is dequeued and transmitted onto the egress edge link. If no congestion is present, a regular ACK is generated and sent, which can clear any previous congestion notifications received by the upstream switches operating on the corresponding flow. If congestion is present, the ACK can be marked with a flag, which allows the ACK to notify the switches of persistent congestion at the egress edge link as well as the successful delivery of the packet.

FIG. 7A shows a flow chart of an exemplary process of generating an explicit endpoint-congestion-notification ACK. During operation, the system can continuously monitor an egress edge link's output buffer. The system can then receive a packet at the output buffer (operation 702). Upon receipt of the packet, the system can compute the three congestion parameters (total amount of data, total number of packets, and rate of change of buffer depth) for the output buffer (operation 704). The system can further determine whether any of the parameters exceeds a corresponding threshold (operation 706). If at least one parameter exceeds the threshold, congestion is considered to be present. Accordingly, the system can generate and send an explicit endpoint-congestion-notification ACK packet corresponding to the packet's flow to the upstream switches (operation 708). If no congestion is detected, the system can return to normal operation.

FIG. 7B shows an exemplary endpoint congestion management logic block. In this example, an endpoint congestion management logic block 730 can include an output buffer monitor 732, a congestion parameter computation logic block 734, and an endpoint-congestion-notification ACK generation logic block 736. During operation, output buffer monitor 732 can monitor the state of an output buffer associated with an egress edge link. Based on the state of the monitored output buffer, congestion parameter computation logic block 734 can compute the three congestion parameters (see operation 704 in the flow chart in FIG. 7A). When one of these parameters exceeds the corresponding threshold, endpoint-congestion-notification ACK generation logic block 736 can generate an endpoint-congestion-notification ACK and transmit the ACK to the upstream switch.

FIG. 8 shows a flow chart showing of exemplary process of generating an ACK in response to a packet being dequeued from an output buffer. In this example, the system first dequeues a packet from the output buffer (operation 802). The system can then compute the three congestion parameters (total amount of data, total number of packets, and rate of change of buffer depth) for the output buffer (operation 804). The system can determine whether any of the parameters exceeds a corresponding threshold (operation 806). If at least one parameter exceeds the threshold, congestion is considered to be present. Accordingly, the system can generate an ACK packet with a marked flag indicating persisting congestion (operation 808). If no congestion is detected, the system can generate a regular ACK packet (operation 809). The system can subsequently send the ACK packet to the upstream switches (operation 810), and transmit the dequeued data packet onto the egress edge link (operation 812).

Note that the endpoint congestion management logic block shown in FIG. 7B can also perform the operations described by the flow chart shown in FIG. 8. In other words, endpoint congestion management logic block 730 can potentially general endpoint-congestion-notification ACKs upon the arrival of a packet at the output buffer as well as the departure of the packet from the output buffer.

As an endpoint-congestion-notification ACK traverses the fabric, the IFCT's of the switches along the path can apply bandwidth limitations to the flow corresponding to the ACK. Effectively, the fabric can slow down the delivery of that flow in a distributed way at each switch along the data path. When an endpoint-congestion-notification ACK passes an IFCT its value can be stored in the flow's table entry as an ep_congestion value, which can be used to select a desired maximum bandwidth for the flow. Each value of ep_congestion can have a corresponding set of high, target, and drop watermark values. For high levels of congestion, when ep_congestion has a high value, the watermark values can have lower values, so that the congestion can be mitigated more aggressively. For low levels of congestion, when ep_congestion has a low value, a different set of greater high, target, and drop watermark values can be used for higher flow bandwidth. For example, a table indexed by the ep_congestion value can be used. For each ep_congestion value, the table can indicate a corresponding set of high, target, and drop watermark values. The entries of this table can be predetermined, so that when an endpoint-congestion-notification ACK is received, the switch can use the ep_congestion value to perform a lookup in this table, and apply the three corresponding watermark values to the identified flow.

In some cases, if the source is injecting data in a greedy manner, only slowing down the forwarding inside the network might not be sufficient to fully remove the congestion. To address this problem, an ingress edge switch can be configured to instruct the source device (which typically resides outside the fabric) to limit data injection on a fine-grain, per-flow basis. This switch-to-host flow control mechanism can be referred to as Fine Gran Flow Control (FGFC).

In particular, especially in an HPC environment, an end host or computing node could have a large number of cores running numerous threads, processes, or virtual machines, each of which could be injecting their own stream of data into the network through a common physical network interface controller (NIC). When congestion is present, a per-port based flow control can only throttle the overall data rate over a single port on the NIC, which can be 40 Gb/s or more. Pushing back on the total data rate on the entire port can cause unfairness to flows that are not contributing to congestion. FGFC can extend the concept of the individual flows or group of associated flows to their ultimate source, which can be a single thread executing on one of the cores.

To slow down data injection from the source, an FGFC logic block on an ingress edge switch (for example, FGFC logic block 434 in edge switch 406 in FIG. 4A) can use a pause-credit hybrid method to throttle incoming data associated a particular flow or group of flows. A pause-based method typically involves a receiving end issuing a pause command to the transmitter end, which in response can stop transmission until further notice. With a credit-based method, the receiving end can issue transmission credits to the transmitting end, which allows the transmitter to send more data but only up to the amount specified by the credit value. This mechanism allows the receiving end to control more precisely its input buffer depth to avoid overflow while allowing transmission to continue. FGFC can use a hybrid method, in which upon detection of congestion the ingress edge switch can issue a FGFC frame for one or more flows with a set timer value to the end host NIC (such as NIC 401 on end host 402 in FIG. 4A). After the FGFC frame is received, the ingress edge switch may turn on a credit-based flow control mode. In response, the NIC can throttle the transmission data rate for the corresponding flow(s) based on the received credit, while allowing other flows to be transmitted at normal data rate. After the predetermined timer expires, the end host NIC can revert to normal transmission for the throttled flow(s), unless another pause command is received. Note that a throttled flow can be identified by any field derived from a packet. A throttled flow can be specific to a single process or thread executed on the end host.

FGFC can implement the control communication between an edge switch and an end host NIC using an Ethernet frame with an Organizationally Unique Identifier (OUI) extended Ether_Type field. These frames can indicate one or more of the following: (1) the protocol used by the flow being controlled; (2) an identifier to indicate the source (e.g., application, process, or thread) generating the packets that need to be throttled; (3) a pause time value for which the flow control is to last (which can prevent a lockup if subsequent FGFC frames are lost due to errors), and (4) a credit value, which can be zero, to indicate the number of frames or amount of data that can be sent during the pause period.

Note that the identifier for indicating the source flow subject to flow control can be different based on the protocol associated with the flow. For layer-2 Ethernet virtual local area network (VLAN) traffic, the identifier can include the VLAN number. For IPv4 traffic, the identifier can include a source/destination IP address pair, a UDP or TCP/IP 5-tuple that includes UDP or TCP port numbers, or an optional flow label. For IPv6 traffic, the identifier can include one or more IPv6 addresses or an IPv6 flow label. For proprietary HPC protocol traffic, the identifier can include a process or thread ID. In general, this identifier is also stored in the EFCT of the edge switch, since it is used to map the corresponding traffic to a flow ID.

To trigger FGFC, the IFCT of an ingress edge switch can monitor its flow-specific input queues. For each queue, the corresponding IFCT entry can indicate three watermark values: high, target, and drop, which can be used to measure the queue depth. In some examples, these watermark values can be included as additional fields in the IFCT as shown in FIG. 4C, or can be stored in a separate table and linked by a field in the IFCT. When the queue depth is less than the target value, no FGFC is necessary. When the queue depth reaches the target watermark value, the IFCT can communicate with an FGFC logic block to initiate FGFC with an end host's NIC. When the queue depth reduces to below the drop watermark value, FGFC can be stopped and normal transmission for the flow can be resumed.

FIG. 9A shows a flow chart of an exemplary FGFC process. During operation, at an ingress edge switch, the system can monitor the flow-specific input queues (operation 902). The system can further determine, for a respective flow, whether FGFC is currently turned on (operation 904). If FGFC is currently turned on for this flow, the system can then determine whether the queue depth is below the drop watermark (operation 906). If the queue depth has not reduced to below the drop watermark, the system can continue the credit based transmission in the FGFC mode (operation 912). If the queue depth has reduced to below the drop watermark, the system can revert to normal transmission for the flow (operation 914). Referring back to operation 904, if FGFC is currently not turned on, the system can determine whether the queue depth is greater than the target watermark (operation 908). If so, the system can initiate FGFC for the flow (operation 910). The FGFC logic block in the edge switch can obtain flow identifying information (e.g., VLAN tag, TCP/IP 5-tuple, thread ID, etc.) from the EFCT entry corresponding to the flow and send an FGFC Ethernet frame to the NIC on the end host. Subsequently, the system can continue to monitor the input queues (operation 902). If the queue depth is not greater than the target watermark, the system can continue regular data transmission (operation 914)

To facilitate FGFC, a NIC can be configured to process the FGFC Ethernet frame, so that the NIC can communicate to the application or process on an end host that is generating the data. Parsing of the FGFC Ethernet frame and communication to the application or process can be done in software, hardware, or a combination of both. FIG. 9B shows an example of a FGFC-enabled NIC. In this example, a NIC 930 can include a processor 932, a memory 934, a transmitter 936, a receiver 938, a FGFC logic block 940, and a communication logic block 942. During operation, transmitter 936 and receiver 938 can perform communication to and from an edge switch via an edge link. Communication logic block 942 can perform communication via a data bus (such as a Peripheral Component Interconnect Express (PCIe) bus) with the central processing unit of the end host in which NIC 930 resides. Processor 932 and memory 934, which are internal to NIC 930, can perform local processing of the data. During operation, FGFC logic block 940 can work with an edge switch to apply FGFC on a per-flow basis. In addition, FGFC logic block 940 can communicate via communication logic block 942 with the end host's central processing unit to throttle and data injection of an individual application or process corresponding to the specific flow subject to FGFC, thereby controlling the amounted of data injected into the fabric.

As mentioned above, two types of congestions can occur in a network. A first type is endpoint congestion, and a second type is fabric link congestion. FIG. 10 shows an example of fabric link congestion. In this example, two intermediate switches 1002 and 1006 are in communication via a fabric link 1004. Multiple source/destination pairs can be sending traffic via fabric link 1004. As a result, fabric link 1004 can experience congestion, although the links leading up to and away from fabric link 1004 might not be congested. Fabric link 1004 can appear to be a “hot spot” when such congestion occurs.

To mitigate fabric link congestion, a switch can apply dynamic per-flow credit-based flow control. At a switch, if an input queue starts to fill up, and the queue_extent value for this flow reaches a predetermined threshold, the switch can generate a special ACK to notify the upstream switch's IFCT of the congestion. This special per-hop ACK can be referred to as “HeadroomACK.” Upon receiving the HeadroomACK, the upstream switch's IFCT can start a credit based flow control with the downstream switch. In the downstream IFCT entry, a flag Upstream Metering (UM) can be set to indicate that the data transmission from the upstream switch is now metered based on the credits. The HeadroomACK packet can also include a credit value.

When the upstream switch receives a HeadroomACK, a flag called Downstream Metered (DM) can be set in the corresponding entry of the IFCT. The IFCT can also store a signed headroom field in the IFCT entry with the credit value carried by the HeadroomACK (i.e., the headroom value indicates the number of credits). This headroom field can represent the amount of data that can be forwarded to the downstream switch. This establishes a credit based flow control for the corresponding flow. If the upstream IFCT receives a HeadroomACK while the DM flag in the flow's entry is already set, the credit value carried by the HeadroomACK can be added to the existing headroom value.

New packets received by the upstream IFCT can be blocked if the headroom value is not greater than zero (i.e., there is no credit available). These packets can fill this flow's input queue and may in turn cause the IFCT to initiate per-flow credit based flow control with its upstream IFCT, and so on. If the headroom value is greater than zero, a packet stored in the input queue can be dequeued and forwarded to the downstream switch, and the headroom value can be decremented by the size of the forwarded packet, which may cause the headroom value to become zero or negative.

With the flow restricted from sending new packets to the downstream IFCT, the downstream IFCT's input queue can start to drain at some rate depending on its downstream congestion. As described above, each flow's input queue can have three queue-depth watermark values, namely high, target, and drop, which can be used to manage credit-based flow control. The target watermark can be approximately the ideal queue depth for the desired flow bandwidth. It indicates sufficient buffering is available for transmitting data downstream. When there is congestion, the credit-based flow control mechanism can attempt to keep the flow's queue_extent value approximately at this target watermark.

If the queue_extent value is between the high watermark and drop watermark, and is greater than the target watermark, when a packet is forwarded, slightly less than this packet's size of credit can be returned with a HeadroomACK to the upstream switch. If the queue_extent value does not exceed the target watermark, when a packet is forwarded, slightly more than this packet's size of credit can be returned with the HeadroomACK to the upstream switch.

If the queue_extent depth is greater than the high watermark, no credit is returned when packets are forwarded. This mechanism can bring the queue_extent value down more quickly and is usually used when congestion is detected for the first time. If the congestion clears, the flow's input queue can start to empty more quickly. When the queue depth is less than the drop watermark, the credit-based flow control can be switched off. This can done by clearing the UM flag in the IFCT entry and returning a HeadroomACK with the maximum credit value to the upstream switch. When received by the upstream IFCT the HeadroomACK clears the entry's DM flag and flow control against the headroom value is turned off.

Note that in a typical network topology there can be a number of switches and between two endpoints there can be multiple data paths. In a multi-path network, it is possible to use various methods to control fabric link congestion. For example, the injection limits, described later in this document, can control the maximum total amount of data in the entire fabric. This means that if a particular fabric link is overloaded, a flow can use a different data path that does not go through the congested link. It is possible to detect an overloaded link and generate “reroute” ACKs for a set of flows. The reroute ACKs can temporarily block the flow in an upstream switch, and when all the ACKs for that flow have been returned, the flow can be unblocked and become free to use a different path across the fabric. A dynamic load-based adaptive routing mechanism can then direct the lead packet to use a different uncongested fabric link. In turn the load across the entire fabric can become more balanced.

FIG. 11 shows a flow chart of an example process of applying credit-based flow control on a congested fabric link. During operation, a switch system can monitor its flow-specific input queues (operation 1102). The system can determine whether an entry in its IFCT has a UM flag set (operation 1104). If the UM flag is set, which means that credit-based flow control is on, the system can further determine whether the queue_extent value is less than the drop watermark value (operation 1106). If the queue_extent value is less than the drop watermark value, the system can clear the UM flag, turn off the credit-based flow control, and resume normal data transmission (operation 1014). If the queue_extent value is greater than the drop watermark value, the system can continue the credit-based flow control (operation 1106). Referring back to operation 1104, if the UM flag is not set, which means the system is in regular transmission mode, the system can determine whether the queue_extent value is greater than the target watermark value (operation 1108). If so, the system can initiate credit-based flow control and send a HeadroomACK to the upstream switch (operation 1110). If the queue_extent value is not greater than the target watermark value, the system can continue with regular data transmission (operation 1112).

In general, a flow channel switch can use a combination of several congestion detection and control mechanisms. For example, different degrees of endpoint congestion can be reported using the endpoint-congestion-notification ACK that can be returned from the final fabric egress edge port. This ACK type can be used to manage the bandwidth of flows into a significantly congested egress edge port. The system can also use a per-hop credit-based flow control to manage fabric link congestion. This per-hop congestion management mechanism can be effective against low to moderate levels of congestion, because the response time can be much shorter than the network-wise round trip delay.

If the congestion is severe, perhaps caused by a wide incast, the system can also apply a per-flow injection limit. A flow's injection limit can be determined based on the ep_congestion value. The injection limit can be compared with the flow_extent value in all IFCTs the flow passes through. If the flow_extent is greater than this limit the IFCT can block the forwarding of packets from the input queue for this flow. This mechanism can reduce the rate of forwarding of packets over an entire flow to as little as a single packet.

The system can also protect unrelated traffic from extreme congestion caused by incasts with a large number of contributors. In this case, the ep_congestion value can be set to a high value and the average amount of data of a flow can be reduced to a small fraction of a packet. This can be achieved by only releasing the next packet of an individual flow into the fabric from the ingress edge port's IFCT after a programmable delay has elapsed since when the ACK of the previous packet has been received.

In addition to per-flow injection limits, the system can measure the amount of data that has been injected into the fabric on a per-ingress-port basis, and set injection limits to impose a cap on the total amount of data a port can inject into the fabric. Since every ingress port can apply this injection limit, the system can control the maximum amount of data allowed inside the fabric. Limiting the total amount of data into the fabric can ensure that buffer exhaustion does not occur where bandwidth is scarce. As a result, traffic which is not using the paths with reduced bandwidth are not affected.

To facilitate per-port injection limit, an IFCT can maintain a total traffic count. Each time a packet is injected into the fabric from the edge port the total count can be incremented. When a flow's ACK is returned, the total traffic count can be decremented. Once all the ACKs of all the flows of an ingress port have been returned (i.e., when the sum of the flow_extent values for all the flows becomes zero), the total traffic count can be set to zero.

FIG. 12 shows an exemplary edge switching system that facilitates flow channels (which, for example, can correspond to switch 406 in FIG. 4A). In this example, a switch 1202 can include a number of communication ports, such as port 1220. Each port can include a transmitter and a receiver. Switch 1202 can also include a processor 1204, a storage device 1206, and a flow channel switching logic block 1208. Flow channel switching module 1208 can be coupled to all the communication ports and can further include a crossbar switch 1210, an EFCT logic block 1212, an IFCT logic block 1214, and an OFCT logic block 1216.

Crossbar switch 1210 can include one or more crossbar switch chips, which can be configured to forward data packets and control packets (such as ACK packets) among the communication ports. EFCT logic block 1212 can process packets received from an edge link and map the received packets to respective flows based on one or more header fields in the packets. In addition, EFCT logic block 1212 can assemble FGFC Ethernet frames, which can be communicated to an end host to control the amount of data injected by individual processes or threads. IFCT logic block 1214 can include the IFCT, and perform various flow control methods in response to control packets, such as endpoint-congestion-notification ACKs and fabric-link credit-based flow control ACKs. OFCT logic block 1216 can include a memory unit that stores the OFCT and communicate with another switch's IFCT logic block to update a packet's flow ID when the packet is forwarded to a next-hop switch.

FIG. 13 shows an exemplary intermediary switching system that facilitates flow channels (which, for example, can correspond to switches 408 and 430 in FIG. 4A). In this example, a switch 1302 can include a number of communication ports, such as port 1320. Each port can include a transmitter and a receiver. Switch 1302 can also include a processor 1304, a storage device 1306, and a flow channel switching logic block 1308. Flow channel switching module 1308 can be coupled to all the communication ports and can further include a crossbar switch 1310, an EFCT logic block 1312, an IFCT logic block 1314, and an OFCT logic block 1316.

Crossbar switch 1310 can include one or more crossbar switch chips, which can be configured to forward data packets and control packets (such as ACK packets) among the communication ports. EFCT logic block 1312 can process packets received from an edge link and map the received packets to respective flows based on one or more header fields in the packets. In addition, EFCT logic block 1312 can assemble FGFC Ethernet frames, which can be communicated to an end host to control the amount of data injected by individual processes or threads. IFCT logic block 1314 can include the IFCT, and perform various flow control methods in response to control packets, such as endpoint-congestion-notification ACKs and fabric-link credit-based flow control ACKs. OFCT logic block 1316 can include a memory unit that stores the OFCT and communicate with another switch's IFCT logic block to update a packet's flow ID when the packet is forwarded to a next-hop switch.

In summary, the present disclosure describes a data-driven intelligent networking system that can accommodate dynamic traffic with fast, effective congestion control. The system can maintain state information of individual packet flows, which can be set up or released dynamically based on injected data. A packet flow can be mapped to their layer-2, layer-3, or other protocol-specific header information. Each flow can be provided with a flow-specific input queue upon arriving at a switch. Packets of a respective flow are acknowledged after reaching the egress point of the network, and the acknowledgement packets are sent back to the ingress point of the flow along the same data path in the reverse direction. As a result, each switch can obtain state information of each flow and perform flow control of a per-flow basis. Such flow control allows the network to be better utilized while providing versatile traffic-engineering and congestion control capabilities.

The methods and processes described above can be performed by hardware logic blocks, modules, or apparatus. The hardware logic blocks, modules, or apparatus can include, but are not limited to, application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), dedicated or shared processors that execute a piece of code at a particular time, and other programmable-logic devices now known or later developed. When the hardware logic blocks, modules, or apparatus are activated, they perform the methods and processes included within them.

The methods and processes described herein can also be embodied as code or data, which can be stored in a storage device or computer-readable storage medium. When a processor reads and executes the stored code or data, the processor can perform these methods and processes.

The foregoing descriptions of embodiments of the present invention have been presented for purposes of illustration and description only. They are not intended to be exhaustive or to limit the present invention to the forms disclosed. Accordingly, many modifications and variations will be apparent to practitioners skilled in the art. Additionally, the above disclosure is not intended to limit the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A switch, comprising: an output port;an output buffer coupled to the output port that buffers packets of packet flows to be transmitted via the output port, wherein a respective packet flow includes packets with one or more common header fields and is identified by a flow identifier that is unique within an input port via which the packet flow is received;an output buffer control logic block that allocates bandwidth of the output port based on a predetermined configuration among the packet flows, wherein the packet flows share the bandwidth of the output port with fairness based on the predetermined configuration;an arbitration logic block that: arbitrates among the number of packet flows based on the bandwidth allocated to a respective packet flow, andselects a packet flow based on the arbitration for forwarding a packet from the output buffer; andan input queue logic block that: maintains state information for the respective packet flow by monitoring acknowledgment packets for packets of the respective packet flow transmitted via the output port, anddeallocates the flow identifier from the respective packet flow in response to receiving an acknowledgement packet for a last transmitted packet of the respective packet flow.
2. The switch of claim 1, wherein while allocating bandwidth on the output port among the packet flows, the output buffer control logic block allocates space in the output buffer among the packet flows based on the predetermined configuration.
3. The switch of claim 1, wherein the output port is an edge port to be coupled to an end host.
4. The switch of claim 1, wherein the output port is a fabric port to be coupled to another switch.
5. The switch of claim 1, wherein the predetermined configuration indicates that the bandwidth on the output port is to be shared among the packet flows based on a fairness-based round-robin scheme.
6. The switch of claim 1, wherein at least one of the packet flows corresponds to a Transmission Control Protocol (TCP) session.
7. The switch of claim 1, wherein at least one of the packet flows corresponds to a User Datagram Protocol (UDP) session.
8. A method comprising: buffering, in an output buffer associated with an output port of a switch, packets in packet flows to be transmitted via the output port, wherein a respective packet flow includes packets with one or more common header fields and is identified by a flow identifier that is unique within an input port of the switch via which the packet flow is received;allocating bandwidth of the output port based on a predetermined configuration among the packet flows, wherein the packet flows share the bandwidth of the output port with fairness based on the predetermined configuration;arbitrating among the packet flows based on the bandwidth allocated to a respective packet flow;selecting a packet flow based on the arbitration for forwarding a packet from the output buffer;maintaining state information for the respective packet flow by monitoring acknowledgment packets for packets of the respective packet flow transmitted via the output port, anddeallocating the flow identifier from the respective packet flow in response to receiving an acknowledgement packet for a last transmitted packet of the respective packet flow.
9. The method of claim 8, wherein allocating bandwidth on the output port among the packet flows comprises allocating space in the output buffer among the packet flows based on the predetermined configuration.
10. The method of claim 8, wherein the output port is an edge port to be coupled to an end host.
11. The method of claim 8, wherein the output port is a fabric port to be coupled to another switch.
12. The method of claim 8, wherein at least one of the packet flows corresponds to a Transmission Control Protocol (TCP) session.
13. The method of claim 8, wherein at least one of the packet flows corresponds to a User Datagram Protocol (UDP) session.
14. The method of claim 8, wherein the predetermined configuration indicates that the bandwidth on the output port is to be shared among the packet flows based on a fairness-based round-robin scheme.
15. A computing system, comprising: a processor;an output port;a non-transitory computer-readable storage medium storing instructions that when executed by the processor cause the computing system to perform a method, the method comprising: buffering, in an output buffer associated with the output port, packets in a packet flows to be transmitted via the output port, wherein a respective packet flow includes packets with one or more common header fields and is identified by a flow identifier that is unique within an input port via which the packet flow is receive;allocating, bandwidth of the output port based on a predetermined configuration among the packet flows, wherein the packet flows share the bandwidth of the output port with fairness based on the predetermined configuration;arbitrating among the packet flows based on the bandwidth allocated to a respective packet flow;selecting a packet flow based on the arbitration for forwarding a packet from the output buffer;maintaining state information for the respective packet flow by monitoring acknowledgment packets for packets of the respective packet flow transmitted via the output port, anddeallocating the flow identifier from the respective packet flow in response to receiving an acknowledgement packet for a last transmitted packet of the respective packet flow.
16. The computing system of claim 15, wherein while allocating bandwidth on the output port among the packet flows, the method further comprises allocating space in the output buffer among the packet flows based on the predetermined configuration.
17. The computing system of claim 15, wherein the output port is an edge port to be coupled to an end host.
18. The computing system of claim 15, wherein the output port is a fabric port to be coupled to another computing system.
19. The computing system of claim 15, wherein the predetermined configuration indicates that the bandwidth on the output port is to be shared among the packet flows based on a fairness-based round-robin scheme.
20. The computing system of claim 15, wherein at least one of the packet flows corresponds to a Transmission Control Protocol (TCP) session.
21. The computing system of claim 15, wherein at least one of the packet flows corresponds to a User Datagram Protocol (UDP) session.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a 371 National Stage Entry of PCT/US2020/024266, filed on Mar. 23, 2020, which claims the benefit of and priority to U.S. Provisional Patent Application No. 62/852,203, filed on May 23, 2019; U.S. Provisional Patent Application No. 62/852,273, filed on May 23, 2019; and U.S. Provisional Patent Application No. 62/852,289, filed on May 23, 2019; the contents of which are incorporated herein by reference in their entirety.

PCT Information

Filing Document	Filing Date	Country	Kind
PCT/US2020/024266	3/23/2020	WO

Publishing Document	Publishing Date	Country	Kind
WO2020/236285	11/26/2020	WO	A

US Referenced Citations (425)

Number	Name	Date	Kind
4807118	Lin et al.	Feb 1989	A
5138615	Lamport et al.	Aug 1992	A
5457687	Newman	Oct 1995	A
5937436	Watkins	Aug 1999	A
5960178	Cochinwala et al.	Sep 1999	A
5970232	Passint et al.	Oct 1999	A
5983332	Watkins	Nov 1999	A
6112265	Harriman et al.	Aug 2000	A
6230252	Passint et al.	May 2001	B1
6246682	Roy et al.	Jun 2001	B1
6493347	Sindhu et al.	Dec 2002	B2
6545981	Garcia et al.	Apr 2003	B1
6633580	Toerudbakken et al.	Oct 2003	B1
6674720	Passint et al.	Jan 2004	B1
6714553	Poole et al.	Mar 2004	B1
6728211	Peris et al.	Apr 2004	B1
6732212	Sugahara et al.	May 2004	B2
6735173	Lenoski et al.	May 2004	B1
6894974	Aweva et al.	May 2005	B1
7023856	Washabaugh et al.	Apr 2006	B1
7133940	Blightman et al.	Nov 2006	B2
7218637	Best et al.	May 2007	B1
7269180	Bly et al.	Sep 2007	B2
7305487	Blumrich et al.	Dec 2007	B2
7337285	Tanoue	Feb 2008	B2
7397797	Alfieri et al.	Jul 2008	B2
7430559	Lomet	Sep 2008	B2
7441006	Biran et al.	Oct 2008	B2
7464174	Ngai	Dec 2008	B1
7483442	Torudbakken et al.	Jan 2009	B1
7562366	Pope et al.	Jul 2009	B2
7593329	Kwan et al.	Sep 2009	B2
7596628	Aloni et al.	Sep 2009	B2
7620791	Wentzlaff et al.	Nov 2009	B1
7633869	Morris et al.	Dec 2009	B1
7639616	Manula et al.	Dec 2009	B1
7734894	Wentzlaff et al.	Jun 2010	B1
7774461	Tanaka et al.	Aug 2010	B2
7782869	Chitlur Srinivasa	Aug 2010	B1
7796579	Bruss	Sep 2010	B2
7856026	Finan et al.	Dec 2010	B1
7933282	Gupta et al.	Apr 2011	B1
7953002	Opsasnick	May 2011	B2
7975120	Sabbatini, Jr. et al.	Jul 2011	B2
8014278	Subramanian et al.	Sep 2011	B1
8023521	Woo et al.	Sep 2011	B2
8050180	Judd	Nov 2011	B2
8077606	Mark	Dec 2011	B1
8103788	Miranda	Jan 2012	B1
8160085	Voruganti et al.	Apr 2012	B2
8175107	Yalagandula et al.	May 2012	B1
8249072	Sugumar et al.	Aug 2012	B2
8281013	Mundkur et al.	Oct 2012	B2
8352727	Chen et al.	Jan 2013	B2
8353003	Noehring et al.	Jan 2013	B2
8443151	Tang et al.	May 2013	B2
8473783	Andrade et al.	Jun 2013	B2
8543534	Alves et al.	Sep 2013	B2
8619793	Lavian et al.	Dec 2013	B2
8626957	Blumrich et al.	Jan 2014	B2
8650582	Archer et al.	Feb 2014	B2
8706832	Blocksome	Apr 2014	B2
8719543	Kaminski et al.	May 2014	B2
8811183	Anand et al.	Aug 2014	B1
8948175	Bly et al.	Feb 2015	B2
8971345	McCanne et al.	Mar 2015	B1
9001663	Attar et al.	Apr 2015	B2
9053012	Northcott et al.	Jun 2015	B1
9088496	Vaidya et al.	Jul 2015	B2
9094327	Jacobs et al.	Jul 2015	B2
9178782	Matthews et al.	Nov 2015	B2
9208071	Talagala et al.	Dec 2015	B2
9218278	Talagala et al.	Dec 2015	B2
9231876	Mir et al.	Jan 2016	B2
9231888	Bogdanski et al.	Jan 2016	B2
9239804	Kegel et al.	Jan 2016	B2
9269438	Nachimuthu et al.	Feb 2016	B2
9276864	Pradeep	Mar 2016	B1
9436651	Underwood et al.	Sep 2016	B2
9455915	Sinha et al.	Sep 2016	B2
9460178	Bashyam et al.	Oct 2016	B2
9479426	Munger et al.	Oct 2016	B2
9496991	Plamondon et al.	Nov 2016	B2
9544234	Markine	Jan 2017	B1
9548924	Pettit et al.	Jan 2017	B2
9594521	Blagodurov et al.	Mar 2017	B2
9635121	Mathew et al.	Apr 2017	B2
9742855	Shuler et al.	Aug 2017	B2
9762488	Previdi et al.	Sep 2017	B2
9762497	Kishore et al.	Sep 2017	B2
9830273	Bk et al.	Nov 2017	B2
9838500	Ilan et al.	Dec 2017	B1
9853900	Mula et al.	Dec 2017	B1
9887923	Chorafakis et al.	Feb 2018	B2
10003544	Liu et al.	Jun 2018	B2
10009270	Stark et al.	Jun 2018	B1
10031857	Menachem et al.	Jul 2018	B2
10050896	Yang et al.	Aug 2018	B2
10061613	Brooker et al.	Aug 2018	B1
10063481	Jiang et al.	Aug 2018	B1
10089220	Mckelvie et al.	Oct 2018	B1
10169060	Vincent et al.	Jan 2019	B1
10178035	Dillon	Jan 2019	B2
10200279	Aljaedi	Feb 2019	B1
10218634	Aldebert et al.	Feb 2019	B2
10270700	Burnette et al.	Apr 2019	B2
10305772	Zur et al.	May 2019	B2
10331590	Macnamara et al.	Jun 2019	B2
10353833	Hagspiel et al.	Jul 2019	B2
10454835	Contavalli et al.	Oct 2019	B2
10498672	Graham et al.	Dec 2019	B2
10567307	Fairhurst et al.	Feb 2020	B2
10728173	Agrawal et al.	Jul 2020	B1
10802828	Volpe et al.	Oct 2020	B1
10817502	Talagala et al.	Oct 2020	B2
11128561	Matthews et al.	Sep 2021	B1
11271869	Agrawal et al.	Mar 2022	B1
11416749	Bshara et al.	Aug 2022	B2
11444886	Stawitzky et al.	Sep 2022	B1
20010010692	Sindhu et al.	Aug 2001	A1
20010047438	Forin	Nov 2001	A1
20020174279	Wynne et al.	Nov 2002	A1
20030016808	Hu et al.	Jan 2003	A1
20030041168	Musoll	Feb 2003	A1
20030110455	Baumgartner et al.	Jun 2003	A1
20030174711	Shankar	Sep 2003	A1
20030200363	Futral	Oct 2003	A1
20030223420	Ferolito	Dec 2003	A1
20040008716	Stiliadis	Jan 2004	A1
20040059828	Hooper et al.	Mar 2004	A1
20040095882	Hamzah et al.	May 2004	A1
20040133634	Luke et al.	Jul 2004	A1
20040223452	Santos et al.	Nov 2004	A1
20050021837	Haselhorst et al.	Jan 2005	A1
20050047334	Paul et al.	Mar 2005	A1
20050088969	Carlsen et al.	Apr 2005	A1
20050091396	Nilakantan et al.	Apr 2005	A1
20050100011	Chiruvolu et al.	May 2005	A1
20050108444	Flauaus et al.	May 2005	A1
20050108518	Pandya	May 2005	A1
20050152274	Simpson	Jul 2005	A1
20050182854	Pinkerton et al.	Aug 2005	A1
20050270974	Mayhew	Dec 2005	A1
20050270976	Yang et al.	Dec 2005	A1
20060023705	Zoranovic et al.	Feb 2006	A1
20060067347	Naik et al.	Mar 2006	A1
20060075480	Noehring et al.	Apr 2006	A1
20060174251	Pope et al.	Aug 2006	A1
20060203728	Kwan et al.	Sep 2006	A1
20070061433	Reynolds et al.	Mar 2007	A1
20070070901	Aloni et al.	Mar 2007	A1
20070198804	Moyer	Aug 2007	A1
20070211746	Oshikiri et al.	Sep 2007	A1
20070242611	Archer et al.	Oct 2007	A1
20070268825	Corwin	Nov 2007	A1
20080013453	Chiang et al.	Jan 2008	A1
20080013549	Okagawa et al.	Jan 2008	A1
20080071757	Ichiriu et al.	Mar 2008	A1
20080084864	Archer et al.	Apr 2008	A1
20080091915	Moertl et al.	Apr 2008	A1
20080147881	Krishnamurthy et al.	Jun 2008	A1
20080159138	Shepherd et al.	Jul 2008	A1
20080253289	Naven et al.	Oct 2008	A1
20090003212	Kwan et al.	Jan 2009	A1
20090010157	Holmes et al.	Jan 2009	A1
20090013175	Elliott	Jan 2009	A1
20090055496	Garg et al.	Feb 2009	A1
20090092046	Naven et al.	Apr 2009	A1
20090141621	Fan et al.	Jun 2009	A1
20090198958	Arimilli et al.	Aug 2009	A1
20090259713	Blumrich et al.	Oct 2009	A1
20090285222	Hoover et al.	Nov 2009	A1
20100061241	Sindhu et al.	Mar 2010	A1
20100169608	Kuo et al.	Jul 2010	A1
20100172260	Kwan et al.	Jul 2010	A1
20100183024	Gupta	Jul 2010	A1
20100220595	Petersen	Sep 2010	A1
20100274876	Kagan et al.	Oct 2010	A1
20100302942	Shankar et al.	Dec 2010	A1
20100316053	Miyoshi et al.	Dec 2010	A1
20110051724	Scott et al.	Mar 2011	A1
20110066824	Bestler	Mar 2011	A1
20110072179	Lacroute	Mar 2011	A1
20110099326	Jung et al.	Apr 2011	A1
20110110383	Yang et al.	May 2011	A1
20110128959	Bando et al.	Jun 2011	A1
20110158096	Leung et al.	Jun 2011	A1
20110158248	Vorunganti et al.	Jun 2011	A1
20110164496	Loh et al.	Jul 2011	A1
20110173370	Jacobs et al.	Jul 2011	A1
20110264822	Ferguson et al.	Oct 2011	A1
20110276699	Pedersen	Nov 2011	A1
20110280125	Jayakumar	Nov 2011	A1
20110320724	Mejdrich et al.	Dec 2011	A1
20120093505	Yeap et al.	Apr 2012	A1
20120102506	Hopmann et al.	Apr 2012	A1
20120117423	Andrade et al.	May 2012	A1
20120137075	Vorbach	May 2012	A1
20120144064	Parker et al.	Jun 2012	A1
20120144065	Parker et al.	Jun 2012	A1
20120147752	Ashwood-Smith et al.	Jun 2012	A1
20120170462	Sinha	Jul 2012	A1
20120170575	Mehra	Jul 2012	A1
20120213118	Lindsay et al.	Aug 2012	A1
20120250512	Jagadeeswaran et al.	Oct 2012	A1
20120287821	Godfrey et al.	Nov 2012	A1
20120297083	Ferguson et al.	Nov 2012	A1
20120300669	Zahavi	Nov 2012	A1
20120314707	Epps et al.	Dec 2012	A1
20130010636	Regula	Jan 2013	A1
20130039169	Schlansker et al.	Feb 2013	A1
20130060944	Archer et al.	Mar 2013	A1
20130103777	Kagan et al.	Apr 2013	A1
20130121178	Mainaud et al.	May 2013	A1
20130136090	Liu et al.	May 2013	A1
20130182704	Jacobs et al.	Jul 2013	A1
20130194927	Yamaguchi et al.	Aug 2013	A1
20130203422	Masputra et al.	Aug 2013	A1
20130205002	Wang et al.	Aug 2013	A1
20130208593	Nandagopal	Aug 2013	A1
20130246552	Underwood et al.	Sep 2013	A1
20130290673	Archer et al.	Oct 2013	A1
20130301645	Bogdanski et al.	Nov 2013	A1
20130304988	Totolos et al.	Nov 2013	A1
20130311525	Neerincx et al.	Nov 2013	A1
20130329577	Suzuki	Dec 2013	A1
20130336164	Yang et al.	Dec 2013	A1
20140019661	Hormuth et al.	Jan 2014	A1
20140032695	Michels et al.	Jan 2014	A1
20140036680	Lih et al.	Feb 2014	A1
20140064082	Yeung et al.	Mar 2014	A1
20140095753	Crupnicoff et al.	Apr 2014	A1
20140098675	Frost et al.	Apr 2014	A1
20140119367	Han et al.	May 2014	A1
20140122560	Ramey et al.	May 2014	A1
20140129664	McDaniel et al.	May 2014	A1
20140133292	Yamatsu et al.	May 2014	A1
20140136646	Tamir et al.	May 2014	A1
20140169173	Naouri et al.	Jun 2014	A1
20140185621	Decusatis et al.	Jul 2014	A1
20140189174	Ajanovic et al.	Jul 2014	A1
20140207881	Nussle et al.	Jul 2014	A1
20140211804	Makikeni et al.	Jul 2014	A1
20140226488	Shamis et al.	Aug 2014	A1
20140241164	Cociglio et al.	Aug 2014	A1
20140258438	Ayoub	Sep 2014	A1
20140301390	Scott et al.	Oct 2014	A1
20140307554	Basso et al.	Oct 2014	A1
20140325013	Tamir et al.	Oct 2014	A1
20140328172	Kumar et al.	Nov 2014	A1
20140347997	Bergamasco et al.	Nov 2014	A1
20140362698	Arad	Dec 2014	A1
20140369360	Carlstrom	Dec 2014	A1
20140379847	Williams	Dec 2014	A1
20150003247	Mejia et al.	Jan 2015	A1
20150006849	Xu et al.	Jan 2015	A1
20150009823	Ganga et al.	Jan 2015	A1
20150026361	Matthews et al.	Jan 2015	A1
20150029848	Jain	Jan 2015	A1
20150055476	Decusatis et al.	Feb 2015	A1
20150055661	Boucher et al.	Feb 2015	A1
20150067095	Gopal et al.	Mar 2015	A1
20150089495	Persson et al.	Mar 2015	A1
20150103667	Elias et al.	Apr 2015	A1
20150124826	Edsall et al.	May 2015	A1
20150146527	Kishore et al.	May 2015	A1
20150154004	Aggarwal	Jun 2015	A1
20150161064	Pope	Jun 2015	A1
20150180782	Rimmer et al.	Jun 2015	A1
20150186318	Kim et al.	Jul 2015	A1
20150193262	Archer et al.	Jul 2015	A1
20150195388	Snyder et al.	Jul 2015	A1
20150208145	Parker et al.	Jul 2015	A1
20150220449	Stark et al.	Aug 2015	A1
20150237180	Swartzentruber et al.	Aug 2015	A1
20150244617	Nakil et al.	Aug 2015	A1
20150244804	Warfield et al.	Aug 2015	A1
20150261434	Kagan et al.	Sep 2015	A1
20150263955	Talaski et al.	Sep 2015	A1
20150263994	Haramaty et al.	Sep 2015	A1
20150288626	Aybay	Oct 2015	A1
20150365337	Pannell	Dec 2015	A1
20150370586	Cooper et al.	Dec 2015	A1
20160006664	Sabato et al.	Jan 2016	A1
20160012002	Arimilli et al.	Jan 2016	A1
20160028613	Haramaty et al.	Jan 2016	A1
20160065455	Wang et al.	Mar 2016	A1
20160094450	Ghanwani et al.	Mar 2016	A1
20160134518	Callon et al.	May 2016	A1
20160134535	Callon	May 2016	A1
20160134559	Abel et al.	May 2016	A1
20160134573	Gagliardi et al.	May 2016	A1
20160142318	Beecroft	May 2016	A1
20160154756	Dodson et al.	Jun 2016	A1
20160182383	Pedersen	Jun 2016	A1
20160205023	Janardhanan	Jul 2016	A1
20160226797	Aravinthan et al.	Aug 2016	A1
20160254991	Eckert et al.	Sep 2016	A1
20160259394	Ragavan	Sep 2016	A1
20160283422	Crupnicoff et al.	Sep 2016	A1
20160285545	Schmidtke et al.	Sep 2016	A1
20160285677	Kashyap et al.	Sep 2016	A1
20160294694	Parker et al.	Oct 2016	A1
20160294926	Zur et al.	Oct 2016	A1
20160301610	Amit et al.	Oct 2016	A1
20160344620	G. Santos et al.	Nov 2016	A1
20160381189	Caulfield et al.	Dec 2016	A1
20170024263	Verplanken	Jan 2017	A1
20170039063	Gopal et al.	Feb 2017	A1
20170041239	Goldenberg et al.	Feb 2017	A1
20170048144	Liu	Feb 2017	A1
20170054633	Underwood et al.	Feb 2017	A1
20170091108	Arellano et al.	Mar 2017	A1
20170097840	Bridgers	Apr 2017	A1
20170103108	Datta et al.	Apr 2017	A1
20170118090	Pettit et al.	Apr 2017	A1
20170118098	Littlejohn et al.	Apr 2017	A1
20170153852	Ma et al.	Jun 2017	A1
20170177541	Berman et al.	Jun 2017	A1
20170220500	Tong	Aug 2017	A1
20170237654	Turner et al.	Aug 2017	A1
20170237671	Rimmer et al.	Aug 2017	A1
20170242753	Sherlock et al.	Aug 2017	A1
20170250914	Caulfield et al.	Aug 2017	A1
20170251394	Johansson et al.	Aug 2017	A1
20170270051	Chen et al.	Sep 2017	A1
20170272331	Mendle	Sep 2017	A1
20170272370	Ganga et al.	Sep 2017	A1
20170286316	Doshi et al.	Oct 2017	A1
20170289066	Haramaty et al.	Oct 2017	A1
20170295098	Watkins et al.	Oct 2017	A1
20170324664	Xu et al.	Nov 2017	A1
20170371778	Mckelvie et al.	Dec 2017	A1
20180004705	Menachem et al.	Jan 2018	A1
20180019948	Patwardhan et al.	Jan 2018	A1
20180026878	Zahavi et al.	Jan 2018	A1
20180077064	Wang	Mar 2018	A1
20180083868	Cheng	Mar 2018	A1
20180097645	Rajagopalan et al.	Apr 2018	A1
20180097912	Chumbalkar et al.	Apr 2018	A1
20180113618	Chan et al.	Apr 2018	A1
20180115469	Erickson et al.	Apr 2018	A1
20180131602	Civanlar et al.	May 2018	A1
20180131678	Agarwal et al.	May 2018	A1
20180150374	Ratcliff	May 2018	A1
20180152317	Chang et al.	May 2018	A1
20180152357	Natham et al.	May 2018	A1
20180173557	Nakil et al.	Jun 2018	A1
20180183724	Callard	Jun 2018	A1
20180191609	Caulfield et al.	Jul 2018	A1
20180198736	Labonte et al.	Jul 2018	A1
20180212876	Bacthu et al.	Jul 2018	A1
20180212902	Steinmacher-Burow	Jul 2018	A1
20180219804	Graham et al.	Aug 2018	A1
20180225238	Karguth et al.	Aug 2018	A1
20180234343	Zdornov et al.	Aug 2018	A1
20180254945	Bogdanski et al.	Sep 2018	A1
20180260324	Marathe et al.	Sep 2018	A1
20180278540	Shalev et al.	Sep 2018	A1
20180287928	Levi et al.	Oct 2018	A1
20180323898	Dods	Nov 2018	A1
20180335974	Simionescu et al.	Nov 2018	A1
20180341494	Sood et al.	Nov 2018	A1
20190007349	Wang et al.	Jan 2019	A1
20190018808	Beard et al.	Jan 2019	A1
20190036771	Sharpless et al.	Jan 2019	A1
20190042337	Dinan et al.	Feb 2019	A1
20190042518	Marolia et al.	Feb 2019	A1
20190044809	Willis et al.	Feb 2019	A1
20190044827	Ganapathi et al.	Feb 2019	A1
20190044863	Mula et al.	Feb 2019	A1
20190044872	Ganapathi et al.	Feb 2019	A1
20190044875	Murty et al.	Feb 2019	A1
20190052327	Motozuka et al.	Feb 2019	A1
20190058663	Song	Feb 2019	A1
20190068501	Schneider et al.	Feb 2019	A1
20190081903	Kobayashi et al.	Mar 2019	A1
20190095134	Li	Mar 2019	A1
20190104057	Goel et al.	Apr 2019	A1
20190104206	Goel et al.	Apr 2019	A1
20190108106	Aggarwal et al.	Apr 2019	A1
20190108332	Glew et al.	Apr 2019	A1
20190109791	Mehra et al.	Apr 2019	A1
20190121781	Kasichainula	Apr 2019	A1
20190140979	Levi et al.	May 2019	A1
20190146477	Cella et al.	May 2019	A1
20190171612	Shahar et al.	Jun 2019	A1
20190196982	Rozas et al.	Jun 2019	A1
20190199646	Singh et al.	Jun 2019	A1
20190253354	Caulfield et al.	Aug 2019	A1
20190280978	Schmatz et al.	Sep 2019	A1
20190294575	Dennison et al.	Sep 2019	A1
20190306134	Shanbhogue et al.	Oct 2019	A1
20190332314	Zhang et al.	Oct 2019	A1
20190334624	Bernard	Oct 2019	A1
20190356611	Das et al.	Nov 2019	A1
20190361728	Kumar et al.	Nov 2019	A1
20190379610	Srinivasan et al.	Dec 2019	A1
20200036644	Belogolovy et al.	Jan 2020	A1
20200084150	Burstein et al.	Mar 2020	A1
20200145725	Eberle et al.	May 2020	A1
20200177505	Li	Jun 2020	A1
20200177521	Blumrich et al.	Jun 2020	A1
20200259755	Wang	Aug 2020	A1
20200272579	Humphrey et al.	Aug 2020	A1
20200274832	Humphrey et al.	Aug 2020	A1
20200334195	Chen et al.	Oct 2020	A1
20200349098	Caulfield et al.	Nov 2020	A1
20210081410	Chavan et al.	Mar 2021	A1
20210152494	Johnsen et al.	May 2021	A1
20210263779	Haghighat et al.	Aug 2021	A1
20210334206	Colgrove et al.	Oct 2021	A1
20210377156	Michael et al.	Dec 2021	A1
20210409351	Das et al.	Dec 2021	A1
20220131768	Ganapathi et al.	Apr 2022	A1
20220166705	Froese	May 2022	A1
20220200900	Roweth	Jun 2022	A1
20220210058	Bataineh et al.	Jun 2022	A1
20220217078	Ford et al.	Jul 2022	A1
20220217101	Yefet et al.	Jul 2022	A1
20220245072	Roweth et al.	Aug 2022	A1
20220278941	Shalev et al.	Sep 2022	A1
20220309025	Chen et al.	Sep 2022	A1
20230035420	Sankaran et al.	Feb 2023	A1
20230046221	Pismenny et al.	Feb 2023	A1

Foreign Referenced Citations (32)

Number	Date	Country
101729609	Jun 2010	CN
102932203	Feb 2013	CN
110324249	Oct 2019	CN
110601888	Dec 2019	CN
0275135	Jul 1988	EP
2187576	May 2010	EP
2219329	Aug 2010	EP
2947832	Nov 2015	EP
3445006	Feb 2019	EP
2003-244196	Aug 2003	JP
3459653	Oct 2003	JP
10-2012-0062864	Jun 2012	KR
10-2012-0082739	Jul 2012	KR
10-2014-0100529	Aug 2014	KR
10-2015-0026939	Mar 2015	KR
10-2015-0104056	Sep 2015	KR
10-2017-0110106	Oct 2017	KR
10-1850749	Apr 2018	KR
2001069851	Sep 2001	WO
0247329	Jun 2002	WO
2003019861	Mar 2003	WO
2004001615	Dec 2003	WO
2005094487	Oct 2005	WO
2007034184	Mar 2007	WO
2009010461	Jan 2009	WO
2009018232	Feb 2009	WO
2014092780	Jun 2014	WO
2014137382	Sep 2014	WO
2014141005	Sep 2014	WO
2018004977	Jan 2018	WO
2018046703	Mar 2018	WO
2019072072	Apr 2019	WO

Non-Patent Literature Citations (76)

Entry
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024266, dated Jul. 9, 2020, 10 pages.
Prakash, P., et al.; “The TCP Outcast Problem: Exposing Unfairness in Data Center Networks”; 2011; 15 pages.
Ramakrishnan et al, RFC 3168, “The addition of Explicit Congestion Notification (ECN) to IP”, Sep. 2001 (Year: 2001).
International Search Report and Written Opinion received for PCT Patent Application No. PCT/US20/24340, dated Oct. 26, 2020, 9 pages.
International Search Report and Written Opinion received for PCT Patent Application No. PCT/US20/24342, dated Oct. 27, 2020, 10 pages.
International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2020/024192, dated Oct. 23, 2020, 9 pages.
International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2020/024221, dated Oct. 26, 2020, 9 pages.
International Search Report cited in PCT/US2020/024170 dated Dec. 16, 2020; 3 pages.
Maabi, S., et al.; “ERFAN: Efficient reconfigurable fault-tolerant deflection routing algorithm for 3-D Network-on-Chip”; Sep. 6-9, 2016.
Maglione-Mathey, G., et al.; “Scalable Deadlock-Free Deterministic Minimal-Path Routing Engine for InfiniBand-Based Dragonfly Networks”; Aug. 21, 2017; 15 pages.
Mamidala, A. R., et al.; “Efficient Barrier and Allreduce on Infiniband clusters using multicast and adaptive algorithms”; Sep. 20-23, 2004; 10 pages.
Mammeri, Z; “Reinforcement Learning Based Routing in Networks: Review and Classification of Approaches”; Apr. 29, 2019; 35 pages.
Mollah; M. A., et al.; “High Performance Computing Systems. Performance Modeling, Benchmarking, and Simulation: 8th International Workshop”; Nov. 13, 2017.
Open Networking Foundation; “OpenFlow Switch Specification”; Mar. 26, 2015; 283 pages.
Ramakrishnan, K., et al.; “The Addition of Explicit Congestion Notification (ECN) to IP”; Sep. 2001; 63 pages.
Roth, P. C., et al.; “MRNet: A Software-Based Multicast/Reduction Network for Scalable Tools1”; Nov. 15-21, 2003; 16 pages.
Silveira, J., et al.; “Preprocessing of Scenarios for Fast and Efficient Routing Reconfiguration in Fault-Tolerant NoCs”; Mar. 4-6, 2015.
Tsunekawa, K.; “Fair bandwidth allocation among LSPs for AF class accommodating TCP and UDP traffic in a Diffserv-capable MPLS network”; Nov. 17, 2005; 9 pages.
Underwood, K.D., et al.; “A hardware acceleration unit for MPI queue processing”; Apr. 18, 2005; 10 pages.
Wu, J.; “Fault-tolerant adaptive and minimal routing in mesh-connected multicomputers using extended safety levels”; Feb. 2000; 11 pages.
Xiang, D., et al.; “Fault-Tolerant Adaptive Routing in Dragonfly Networks”; Apr. 12, 2017; 15 pages.
Xiang, D., et al; “Deadlock-Free Broadcast Routing in Dragonfly Networks without Virtual Channels”, submission to IEEE transactions on Parallel and Distributed Systems, 2015, 15 pages.
Extended European Search Report and Search Opinion received for EP Application No. 20809930.9, dated Mar. 2, 2023, 9 pages.
Extended European Search Report and Search Opinion received for EP Application No. 20810784.7, dated Mar. 9, 2023, 7 pages.
Awerbuch, B., et al.; “An On-Demand Secure Routing Protocol Resilient to Byzantine Failures”; Sep. 2002; 10 pages.
Belayneh L.W., et al.; “Method and Apparatus for Routing Data in an Inter-Nodal Communications Lattice of a Massively Parallel Computer System by Semi-Randomly Varying Routing Policies for Different Packets”; 2019; 3 pages.
Bhatele, A., et al.; “Analyzing Network Health and Congestion in Dragonfly-based Supercomputers”; May 23-27, 2016; 10 pages.
Blumrich, M.A., et al.; “Exploiting Idle Resources in a High-Radix Switch for Supplemental Storage”; Nov. 2018; 13 pages.
Chang, F., et al.; “PVW: Designing Vir PVW: Designing Virtual World Ser orld Server Infr er Infrastructur astructure”; 2010; 8 pages.
Chang, F., et al; “PVW: Designing Virtual World Server Infrastructure”; 2010; 8 pages.
Chen, F., et al.; “Requirements for RoCEv3 Congestion Management”; Mar. 21, 2019; 8 pages.
Cisco Packet Tracer; “packet-tracer;—ping”; https://www.cisco.com/c/en/us/td/docs/security/asa/asa-command-reference/I-R/cmdref2/p1.html; 2017.
Cisco; “Understanding Rapid Spanning Tree Protocol (802.1w)”; Aug. 1, 2017; 13 pages.
Eardley, ED, P; “Pre-Congestion Notification (PCN) Architecture”; Jun. 2009; 54 pages.
Escudero-Sahuquillo, J., et al.; “Combining Congested-Flow Isolation and Injection Throttling in HPC Interconnection Networks”; Sep. 13-16, 2011; 3 pages.
Hong, Y.; “Mitigating the Cost, Performance, and Power Overheads Induced by Load Variations in Multicore Cloud Servers”; Fall 2013; 132 pages.
Huawei; “The Lossless Network For Data Centers”; Nov. 7, 2017; 15 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024248, dated Jul. 8, 2020, 11 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US20/024332, dated Jul. 8, 2020, 13 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US20/24243, dated Jul. 9, 2020, 10 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US20/24253, dated Jul. 6, 2020, 12 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US20/24256, dated Jul. 7, 2020, 11 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US20/24257, dated Jul. 7, 2020, 10 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US20/24258, dated Jul. 7, 2020, 9 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US20/24259, dated Jul. 9, 2020, 13 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US20/24260, dated Jul. 7, 2020, 11 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US20/24268, dated Jul. 9, 2020, 11 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US20/24269, dated Jul. 9, 2020, 11 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US20/24339, dated Jul. 8, 2020, 11 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024125, dated Jul. 10, 2020, 5 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024129, dated Jul. 10, 2020, 11 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024237, dated Jul. 14, 2020, 5 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024239, dated Jul. 14, 2020, 11 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024241, dated Jul. 14, 2020, 13 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024242, dated Jul. 6, 2020, 11 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024244, dated Jul. 13, 2020, 10 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024245, dated Jul. 14, 2020, 11 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024246, dated Jul. 14, 2020, 10 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024250, dated Jul. 14, 2020, 12 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024254, dated Jul. 13, 2020, 10 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024262, dated Jul. 13, 2020, 10 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024270, dated Jul. 10, 2020, 13 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024271, dated Jul. 9, 2020, 10 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024272, dated Jul. 9, 2020, 10 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024276, dated Jul. 13, 2020, 9 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024304, dated Jul. 15, 2020, 11 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024311, dated Jul. 17, 2020, 8 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024321, dated Jul. 9, 2020, 9 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024324, dated Jul. 14, 2020, 10 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024327, dated Jul. 10, 2020, 15 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/24158, dated Jul. 6, 2020, 18 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/24251, dated Jul. 6, 2020, 11 pages.
International Search Report and Written Opinion received for PCT Application No. PCT/US2020/24267, dated Jul. 6, 2020, 9 pages.
International Search Report and Written Opinion received for PCT Patent Application No. PCT/US20/24303, dated Oct. 21, 2020, 9 pages.
Albuquerque. C. et al., “Network Border Patrol: Preventing Congestion Collapse and Promoting Fairness in the Internet,” IEEE, vol. 12, No. 1, 2004, pp. 173-186.
Extended European Search Report and Search Opinion received for EP Application No. 20810785.4, dated May 2, 2023, 9 pages.

Related Publications (1)

	Number	Date	Country
	20220210092 A1	Jun 2022	US

Provisional Applications (3)

Number	Date	Country
62852273	May 2019	US
62852203	May 2019	US
62852289	May 2019	US

System and method for facilitating global fairness in a network

Information

Patent Number

Date Filed

Date Issued

Inventors

Original Assignees

Examiners

Agents

CPC

Field of Search

CPC

International Classifications

Abstract