Patent Grant
11962490
As network-enabled devices and applications become progressively more ubiquitous, various types of traffic as well as the ever-increasing network load continue to demand more performance from the underlying network architecture. For example, applications such as high-performance computing (HPC), media streaming, and Internet of Things (IOT) can generate different types of traffic with distinctive characteristics. As a result, in addition to conventional network performance metrics such as bandwidth and delay, network architects continue to face challenges such as scalability, versatility, and efficiency.
The present disclosure, in accordance with one or more various embodiments, is described in detail with reference to the following figures. The figures are provided for purposes of illustration only and merely depict typical or example embodiments.
The figures are not exhaustive and do not limit the present disclosure to the precise form disclosed.
Large networks are composed from many separate switches connected with may data links. Traditional networks split data into manageable chunks called packets or frames. This allows many separate and different communications to share the bandwidth of a single link. In particular, a single large data transfer for one communication will not prevent many other small communications from completing. The large communication is broken into many separate packets and its packets are time multiplexed with the packets of other small and large communications. This approach allows a single shared network resource to perform many concurrent communications and significantly reduces the maximum latency of small communications in the presence of large communications.
However, sharing resources between many completely different communications only works well provided no one communication can exhaust any of the shared resources of the other communications. It is also essential that access to the shared resources remains fair and appropriate to the importance of each of the communications taking place. Traffic classification has traditionally been utilized to implement data-transit specific network attributes. These traditional classifications consider throughput, bandwidth allocation and latency issues. However, high-performance computing (HPC) has several unique network behaviors, workflows, functions and characteristics not matched by these traditional classifications.
In order to address the characteristics specific to HPC applications, the disclosed embodiments describe a switch device that allows for the determination of data flow and network properties, such as routing and re-routing directives, data-flow separation, ordered or unordered data delivery, lossy or lossless transit, telemetry gathering and traffic shaping rules. Additional traffic classes, beyond those typically employed by conventional per class traffic routing approaches, are used. These traffic classes are based upon the aforementioned HPC-related determinations, and operate simultaneously over the same physical network infrastructure.
As will be described in detail, the disclosed per traffic class routing functionality can be enabled, configured and tuned across the network on a packet-by-packet basis by marking packets within a data flow. Further, the packet-by-packet approach allows for in-band control of data flows, which is set up by the data packets themselves. This type in-band control of data within the network, based upon these type of HPC-related characteristics, has not previously been considered. In other words, the systems and techniques disclosed herein allow for considerations outside of latency and bandwidth, including routing behavior, application data flow separation, and congestion control on a per traffic class basis. Ultimately, the per traffic class routing according to the embodiments, results in a fine-grained control of network data transit.
The present disclosure describes systems and methods that can accommodate exascale computing, e.g., perform data-intensive tasks such as simulations, data analytics, artificial intelligence workloads at exascale speeds. In particular, an HPC network or interconnect fabric is provided that may be Ethernet-compatible, able to connect to third-party data storage, and can be built using a switch component that is extremely high bandwidth, e.g., on the order to 12.8 Tb/s/dir per switch with, e.g., 64 200 Gbps ports that support large network creation with very low diameter (e.g., only three network hops). Moreover, low latency can be achieved through novel congestion control mechanisms, adaptive routing, and the use of traffic classes that allows for flexibility in terms of bandwidth shaping, priority, and routing policy.
Regarding adaptive routing, the techniques and system described herein can achieve dynamic routing of flows by leveraging the identification and management of flow channels. When routing data packets between a source node and a destination, typical routing techniques have been either static or adaptive (e.g., dynamic). In one example of adaptive routing, local routing decisions are made dynamically based upon load information and other factors. In current systems, adaptive routing can causes congestion to spread. That said, certain data flows can be identified as sources of congestion, while other data flows can be identified simply as victims of congestion. In the adaptive routing techniques dealing with persistent flows, as disclosed herein, the victim flows are allowed to continue making traditional routing decisions, whereas the routing of flows causing congestion will be limited. As alluded to above, this capability is enabled by the identification and management of flow channels.
Typically, traffic can be injected into switch fabric 100 via an ingress port of an edge switch, and leave switch fabric 100 via an egress port of another (or the same) edge switch. An ingress edge switch can group injected data packets into flows, which can be identified by flow ID's. The concept of a flow is not limited to a particular protocol or layer (such as layer-2 or layer-3 in the Open System Interface (OSI) reference model). For example, a flow can be mapped to traffic with a particular source Ethernet address, traffic between a source IP address and destination IP address, traffic corresponding to a TCP or UDP port/IP 5-tuple (source and destination IP addresses, source and destination TCP or UDP port numbers, and IP protocol number), or traffic produced by a process or thread running on an end host. In other words, a flow can be configured to map to data between any physical or logic entities. The configuration of this mapping can be done remotely or locally at the ingress edge switch.
Upon receiving injected data packets, the ingress edge switch can assign a flow ID to the flow. This flow ID can be included in a special header, which the ingress edge switch can use to encapsulate the injected packets. Furthermore, the ingress edge switch can also inspect the original header fields of an injected packet to determine the appropriate egress edge switch's address, and include this address as a destination address in the encapsulation header. Note that the flow ID can be a locally significant value specific to a link, and this value can be unique only to a particular input port on a switch. When the packet is forwarded to the next-hop switch, the packet enters another link, and the flow-ID can be updated accordingly. As the packets of a flow traverses multiple links and switches, the flow IDs corresponding to this flow can form a unique chain. That is, at every switch, before a packet leaves the switch, the packet's flow ID can be updated to a flow ID used by the outgoing link. This up-stream-to-down-stream one-to-one mapping between flow ID's can begin at the ingress edge switch and end at the egress edge switch. Because the flow ID's only need to be unique within an incoming link, a switch can accommodate a large number of flows. For example, if a flow ID is 11 bits long, an input port can support up to 2048 flows. Furthermore, the match pattern (one or more header fields of a packet) used to map to a flow can include a greater number of bits. For instance, a 32-bit long match pattern, which can include multiple fields in a packet header, can map up 2{circumflex over ( )}32 different header field patterns. If a fabric has N ingress edge ports, a total number of N*2{circumflex over ( )}32 identifiable flows can be supported.
A switch can assign every flow a separate, dedicated input queue. This configuration allows the switch to monitor and manage the level of congestion of individual flows, and prevent head-of-queue blocking which could occur if shared buffer were used for multiple flows. When a packet is delivered to the destination egress switch, the egress switch can generate and send back an acknowledgement (ACK) in the upstream direction along the same data path to the ingress edge switch. As this ACK packet traverses the same data path, the switches along the path can obtain the state information associated with the delivery of the corresponding flow by monitoring the amount of outstanding, unacknowledged data. This state information can then be used to perform flow-specific traffic management to ensure the health of the entire network and fair treatment of the flows. As explained in more detail below, this per-flow queuing, combined with flow-specific delivery acknowledgements, can allow the switch fabric to implement effective, fast, and accurate congestion control. In turn, the switch fabric can deliver traffic with significantly improved network utilization without suffering from congestion.
Flows can be set up and released dynamically, or “on the fly,” based on demand. Specifically, a flow can be set up (e.g., the flow-ID to packet header mapping is established) by an ingress edge switch when a data packet arrives at the switch and no flow ID has been previously assigned to this packet. As this packet travels through the network, flow IDs can be assigned along every switch the packet traverses, and a chain of flow IDs can be established from ingress to egress. Subsequent packets belonging to the same flow can use the same flow IDs along the data path. When packets are delivered to the destination egress switch and ACK packets are received by the switches along the data path, each switch can update its state information with respect to the amount of outstanding, unacknowledged data for this flow. When a switch's input queue for this flow is empty and there is no more unacknowledged data, the switch can release the flow ID (i.e., release this flow channel) and re-use the flow-ID for other flows. This data-driven dynamic flow setup and teardown mechanism can obviate the need for centralized flow management, and allows the network to respond quickly to traffic pattern changes.
Note that the network architecture described herein is different from software-defined networks (SDN's), which typically uses the OpenFlow protocol. In SDN, switches are configured by a central network controller, and packets are forwarded based one or more fields in the layer-2 (data link layer, such as Ethernet), layer-3 (network layer, such as IP), or layer-4 (transport layer, such as TCP or UDP) headers. In SDN such header-field lookup is performed at every switch in the network, and there is no fast flow ID-based forwarding as is done in the networks described herein. Furthermore, because the OpenFlow header-field lookup is done using ternary content-addressable memory (TCAM), the cost of such lookups can be high. Also, because the header-field mapping configuration is done by the central controller, the setup and tear-down of each mapping relationship is relatively slow and could require a fair amount of control traffic. As a result, an SDN network's response to various network situations, such as congestion, can be slow. In contrast, in the network described herein, the flows can be set up and torn down dynamically based on traffic demand; and packets can be forwarded by a fixed-length flow ID. In other words, flow channels can be data driven and managed (i.e., set up, monitored, and torn down) in a distributed manner, without the intervention of a central controller. Furthermore, the flow ID-based forwarding can reduce the amount of TCAM space used and as a result a much greater number of flows can be accommodated.
Referring to the example shown in
Note that the operations described above can be performed substantially at line speed with little buffering and delay when the first packet is received. After the first packet is processed and scheduled for transmission, subsequent packets from the same flow can be processed by switch 110 even faster because the same flow ID is used. In addition, the design of the flow channels can be such that the allocation, matching, and deallocation of flow channels can have substantially the same cost. For example, a conditional allocation of a flow channel based on a lookup match and a separate, independent deallocation of another flow channel can be performed concurrently in nearly every clock cycle. This means that generating and controlling the flow channels can add nearly no additional overhead to the regular forwarding of packets. The congestion control mechanism, on the other hand, can improve the performance of some applications by more than three orders of magnitude.
At each switch along the data path (which includes switches 110, 106, and 104), a dedicated input buffer can be provided for this flow, and the amount of transmitted but unacknowledged data can be tracked. When the first packet reaches switch 104, switch 104 can determine that the destination fabric address in the packet's fabric header matches its own address. In response, switch 104 can decapsulate the packet from the fabric header, and forward the decapsulated packet to host 116. Furthermore, switch 104 can generate an ACK packet and send this ACK packet back to switch 110. As this ACK packet traverses the same data path, switches 106 and 110 can each update their own state information for the unacknowledged data for this flow.
In general, congestion within a network can cause the network buffers to fill. When a network buffer is full, the traffic trying to pass through the buffer ideally should be slowed down or stopped. Otherwise, the buffer could overflow and packets could be dropped. In conventional networks, congestion control is typically done end-to-end at the edge. The core of the network is assumed to function only as “dumb pipes,” the main purpose of which is to forward traffic. Such network design often suffers from slow responses to congestions, because congestion information often cannot be sent to the edge devices quickly, and the resulting action taken by the edge devices cannot always be effective in removing the congestion. This slow response in turn limits the utilization of the network, because to keep the network free of congestion the network operator often needs to limit the total amount of traffic injected into the network. Furthermore, end-to-end congestion control usually is only effective provided that the network is not already congested. Once the network is heavily congested, end-to-end congestion control would not work, because the congestion notification messages can be congested themselves (unless a separate control-plane network that is different from the data-plane network is used for sending congestion control messages).
In contrast, the flow channels can prevent such congestion from growing within the switch fabric. The flow channel mechanism can recognize when a flow is experiencing some degree of congestion, and in response can slow down or stop new packets of the same flow from entering the fabric. In turn, these new packets can be buffered in a flow channel queue on the edge port and are only allowed into the fabric when packets for the same flow leave the fabric at the destination edge port. This process can limit the total buffering requirements of this flow within the fabric to an amount that would not cause the fabric buffers to become too full.
With flow channels, the switches have a reasonably accurate state information on the amount of outstanding in-transit data within the fabric. This state information can be aggregated for all the flows on an ingress edge port. This means that the total amount of data injected by an ingress edge port can be known. Consequently, the flow channel mechanism can set a limit on the total amount of data in the fabric. When all edge ports apply this limit action, the total amount of packet data in the entire fabric can be well controlled, which in turn can prevent the entire fabric from being saturated. The flow channels can also slow the progress of an individual congested flow within the fabric without slowing down other flows. This feature can keep packets away from a congestion hot spot while preventing buffers from becoming full and ensuring free buffer space for unrelated traffic.
Crossbar switch 210 can include one or more crossbar switch chips, which can be configured to forward data packets and control packets (such as ACK packets) among the communication ports. EFCT logic block 212 can process packets received from an edge link and map the received packets to respective flows based on one or more header fields in the packets. In addition, EFCT logic block 212 can assemble FGFC Ethernet frames, which can be communicated to an end host to control the amount of data injected by individual processes or threads. IFCT logic block 214 can include the IFCT, and perform various flow control methods in response to control packets, such as endpoint-congestion-notification ACKs and fabric-link credit-based flow control ACKs. OFCT logic block 216 can include a memory unit that stores the OFCT and communicate with another switch's IFCT logic block to update a packet's flow ID when the packet is forwarded to a next-hop switch.
In one embodiment, switch 202 is an application-specific integrated circuit (ASIC) that can provide 64 network ports that can operate at either 100 Gbps or 200 Gbps for an aggregate throughput of 12.8 Tbps. Each network edge port may be able to support IEEE 802.3 Ethernet, and Optimized-IP based protocols as well as Portals, an enhanced frame format that provides support for higher rates of small messages. Ethernet frames can be bridged based on their L2 address or they can be routed based on their L3 (1Pv4//1Pv6) address. Optimized-IP frames may only have an L3 (IPv4/1Pv6) header, and are routed. Specialized NIC support can be used for the Portals enhanced frame format, and can map directly onto the fabric format of network 100, e.g., a fabric format that provides certain control and status fields to support a multi-chip fabric when switches/switch chips, such as switches 102, 104, 106, 108, and 110 are connected and communicate with each other. As alluded to above, a congestion control mechanism based on flow channels can be used by such switches, and can also achieve high transmission rates for small packets (e.g., more than 1.2 billion packets per second per port) to accommodate the needs of HPC applications.
Switch 202 can provide system-wide Quality of Service (QoS) classes, along with the ability to control how network bandwidth is allocated to different classes of traffic, and to different classes of applications, where a single privileged application may access more than one class of traffic. Where there is contention for network bandwidth, arbiters select packets to forward based on their traffic class and the credits available to that class. Network 100 can support minimum and maximum bandwidths for each traffic class. If a class does not use its minimum bandwidth, other classes may use the unused bandwidth, but no class can get more than its maximum allocated bandwidth. The ability to manage bandwidth provides the opportunity to dedicate network resources, as well as CPUs and memory bandwidth to a particular application.
In addition to support for QoS classes, switch 202 effectuates flow channel-based congestion control, and can reduce the number of network hops, e.g., in a network having a dragonfly topology, from five network hops to three. The design of switch 202, described in greater detail below, can reduce network cost and power consumption, and may further facilitate use of innovative adaptive routing algorithms that improve application performance. A fabric created by a plurality of switches, such as a plurality of switches 202 may also be used in constructing Fat-Tree networks, for example when building a storage subsystem for integration with third-party networks and software. Further still, the use of switch 202 enables fine-grain adaptive routing while maintaining ordered packet delivery. In some embodiments, switch 202 may be configured to send the header of a packet from an input port to an output port before the full data payload arrives, thereby allowing output port load metrics to reflect future loads, thereby improving adaptive routing decisions made by switch 202.
Crossbar switch 210 may comprise separate, distributed crossbars routing data/data elements between input and output ports. In some embodiments, and as illustrated in
Request crossbar 210a is used to send requests from an input to a targeted output age queue. Grant crossbar 210b is used to return a grant back to the input to satisfy a request. In particular, grant crossbar 210b returns a pointer indicating where a packet is within an input buffer. It should be noted that a grant is returned when there is space in the output for the corresponding packet. Grant crossbar 201b may also optionally return a credit for requested space in the output. It should be noted that grants are returned when there is a landing spot for a packet at the output, e.g., an output port 220c, so packets cannot be blocked (though they can face transient contention for resources).
It should be understood that in accordance with various embodiments, a credit protocol may be used to guarantee that there is a landing space for a request at the output. Accordingly, a credit crossbar 210c may be used to return credit for requested space in the output.
A data crossbar 210d is used to move granted packets from an input buffer to a targeted output buffer. An Ack crossbar 210e is used to propagate Ack packets from output ports 220c to input ports 220b. Acks are steered in accordance with a state kept in an output flow channel table.
It should be understood that data crossbar 210d moves multi-clock packets with both headers and data, while the other four crossbars (request crossbar 210a, grant crossbar 210b, credit crossbar 210c, and Ack crossbar 210e) move only single-clock packet headers. All five crossbars use the same architecture with row buses and column buses within an 8×4 matrix of 32 dual-port tiles (as described below).
Referring back to
Each crossbar switch 210 has sixteen input ports 220b, one for each port in its row, and eight output ports 220c, one for each port in its column. Row buses can be driven from each source in a row to all eight crossbars in that row (one-to-all). Arbitration can be performed at the crossbar from the sixteen row buses in that row to the eight column buses in a given column. Buffering can be provided at each 16×8 crossbar for each of the row buses in order to absorb packets during times when there is contention for a column bus. In some embodiments, a non-jumbo packet is kept off a row bus unless there is room for the entire packet in the targeted crossbar input buffer. Due to area constraints, jumbo packets are allowed to go even if there is not sufficient space (crossbar input buffer only sized to sink a non-jumbo packet) with the row bus being blocked until the packet wins arbitration and space is freed as it is moved onto a column bus.
Column buses are driven from a given crossbar to each destination port within a column (all-to-all). Each destination may have another level of arbitration between the column buses from the four rows. With sixteen row buses driving eight crossbars, each feeding eight column buses, there is a 4× speedup between rows and columns. Each row has identical connections with the one-to-all row bus connections for a single row shown in row buses. Each tile will have a one (request, grant, credit) or a two (data, ack) clock delay per tile depending on the crossbar. This gives a maximum seven or fourteen clock delay to get between the leftmost and rightmost columns. Credit returns routed through credit crossbar 210c may have a one clock delay per tile, and therefore, can take a maximum of seven clocks to complete transmission.
It should be noted that each column may have identical connections with the all-to-all column bus connections for a single column, and there may be a two clock delay per tile, resulting in a six clock delay to get from the top row to the bottom row. It should also be understood that both row and column buses both use the aforementioned credit-based protocol to determine when they are able to send. In the case of row buses, the source port maintains credit counts for the input buffers of the crossbars within that row. For the data crossbar, care is needed to determine when a packet is allowed to go on a row bus. If grants targeting a particular crossbar input buffer all go through a single queue, space for the packet at the head of the queue is required before starting the packet transfer. If the grants are distributed across multiple queues, in order to prevent small packets from locking out large packets, a packet transfer does not start unless there is space for an entire max sized packet in the buffer. In this way, once a packet transfer on a row bus starts, it will not stop until the entire packet has been transferred. Accordingly, crossbar input buffers are configured to be large enough to handle the maximum packet size plus additional space to cover the worst case round trip (packet send to credit return). This will not be the case for jumbo packets. To save on buffering area, the crossbar input buffers are only deep enough to handle a non-jumbo sized MTU (1500 bytes) with a jumbo packet being allowed to block a row bus while waiting to gain access to the targeted column bus.
For column buses, each crossbar maintains credit counts for the input buffers at each destination port in that column. Unlike row buses, there is no requirement that credits be available for a maximum-sized packet before starting transfer of that packet on a column bus. Individual words of the packet will move as credits become available. Therefore, the input buffer at the destination for each column bus needs to only be big enough to cover the worst case round trip (packet to credit).
A fair round-robin arbitration may be used between the sixteen sources for each destination. For the data crossbar 210d, once a source wins arbitration, it keeps control of the destination column bus until the entire packet has been sent. Each output grants a limited amount of packet payload so it is expected that contention for a given column bus should be fairly limited when larger packets are involved. Because of this, a round-robin arbitration is expected to be sufficient even with possibly large differences in packet size among requesters.
Parts of switch 202 associated with output functions generally operate on frames within the switch fabric format, and have a fabric header, even, for example, for a frame arriving and leaning on an Ethernet port within a single switch 202.
Age queue output control is responsible for accepting requests from all of the input port, e.g., input ports 220b, via request crossbar 210a, buffering the requests, arbitrating between them by traffic class using a traffic shaper, and passing the requests to the OFCT 216 to be granted via grant crossbar 210b. Age queue buffering is managed to allow each input to have enough space to flow while also allowing an input with multiple flows targeting a given output to take more space. In particular, an age queue space is managed by output control. The age queue/output control may also be responsible for managing access to the link either using credit-based flow control for a connected input buffer or pause-based flow control for non-fabric links. When a packet is released by the age queue, it is committed to being put on the link. Additionally the age queue has a path allowing packets initiated on a given port e.g., one of input ports 220b (such as maintenance or reduction packets), to arbitrate for resources on the given port.
Requests come into the output control block via a column bus from each row of matrix 30. Each column bus feeds an independent FIFO (e.g., first-in-first-out shift register or buffer) with space in the FIFO managed via credits. The FIFOs may be sized (24 deep) to cover a round-trip plus additional space to allow requests to be moved out of the crossbars 210a-210e and prevent head-of-line blocking. Prior to writing into a FIFO, a request may be checked for a valid error correcting code (ECC). If the ECC check has either a multi bit error (MBE) or a single bit error (SBE) in the destination field (i.e. it has been routed to the wrong port), the request is considered to be an invalid request, and is discarded with an error being flagged.
Least recently used (LRU) arbitration may be performed between column bus FIFOs to choose which FIFO gets forwarded to age queue management. As requests are removed from each FIFO, credits are returned to the corresponding crossbar. The row with which an incoming column bus corresponds can be dependent both on where in the matrix the tile is located, and as well as which half of the tile the block is in.
The output buffer (OBUF) makes requests to the output control block for sending reduction and maintenance packets across a link. These requests may be given the highest priority. A FIFO with 8 locations can be used to buffer these reduction/maintenance packet requests while they wait for resources. Reduction packets need not use flow channels, and maintenance packets may use loopback to create a flow so that checking for flow channel availability or flowing through the OFCT to create a grant is not needed. Reduction and maintenance packets also need not use any space in the output buffer so that no check of space is required. Rather, a check for the link partner input butter may be performed. If allowed, a shaping queue (SQ) or virtual channel (VC) can be granted, blocking any grants from the age queue path from being granted during that cycle.
The size of the next request to be processed from the output buffer is checked against max_frame_size. If it exceeds this setting, the request is not processed and an error flag is set. This will result in the output buffer request path being blocked until a warm reset is performed. The error flag will stay set until the reset is done. The condition can also be released by increasing the setting of max_frame_size to a value above the size of the stuck output buffer request. The size used in the comparison may be the size indicated in the output buffer request (which may include a 4-byte frame checksum (FCS) used on the wire).
Each input may be given the same fixed allocation of age queue space. This age queue space is large enough to reserve a location for each SQ/VC with enough additional space to cover a request/credit round-trip. It is up to the input to manage the space it is given across its SQs/VCs. This allocation (fixed_al/oc) is programmable via a control and status register (CSR) in each input queue (INQ), and can be, e.g., in the range of 64-96 locations. The remaining age queue space (8K−64*fixed_al/oc) may be shared space that is available to all inputs. The shared space can be managed by the output with it moving incoming requests from static to shared space as they arrive if there is room in the shared space, subject to per-input limits. When moving a request to the shared space, a credit is returned, e.g., immediately, via credit crossbar 210c, with the request marked in the age queue as being in the shared space.
When a request is granted, if it is marked as using the shared space, the shared space is credited. If it is not marked as using shared space, the request is considered to have used the static space, and a credit is returned to the input with the grant.
Due to conflicts in credit crossbar 210c, it is possible that credits may not be sent every clock period. Accordingly, a FIFO provides buffering for these transient disruptions. Space in this FIFO is required before taking a request from the request crossbar. A FIFO with a depth of 32 locations can be used to limit the chances of it ever backing up into request crossbar 210a. The shared space may have limits for how much space any input (from an input port 220b) can take. These limits can be set as a percentage of the available space. For instance, if the limit is set to 50%, if one input port is active, it has access to 50% of the buffer space, with two active input ports, each gets 37.5% ((space_used_by_I+space_left*0.5)/2=(50%+50%*0.5)/2), with three active input ports, each gets 29.2% ((space_used_by_2+space_left*0.5)/3=(75%+25%*0.5)/3), and so on. Additionally, the total space used by the active input ports can be limited to the given total (50%, 75%, 87.5%). Thus, the space allocated to each of input port 220b may vary dynamically by how many inputs ports are currently active. The addition of an active input port causes other active inputs ports to give up their space which is then taken by the new input.
Given that division is not something easily done in hardware, the aforementioned age queue credit management function can be implemented as a lookup table with 64 entries. The number of inputs currently active in the age queues indexes the lookup table. The values in the lookup table reflect the limit of the number of shared space locations any input can take along with the total space they can consume as a whole. Thus, it is up to software to program the values in the lookup table according to how much total shared space there is and what percentage each input port is allowed to take. As more input ports 220b become active, each input port 220b is allowed less space, and the total space available increases. Incoming requests from input ports 220b that are above this limit, or in total, exceed the total space limit, are not allowed to take more shared space. In order to track the number of active input ports 220b in the age queues, a set of 64 counters (one for each input port) is used. These count up when a request is put in the age queues and count down as they are taken out (i.e., granted). A count of the number of non-zero counts is used as an index into the lookup table. In addition, in order to manage the shared space, an additional set of 64 counters may be used to track the current usage of the shared space by each input. There may also be a single counter that can be used to track overall shared space usage. These counters are compared against the current quotas to determine if a request is allowed to use the shared space or not. Counters can be, e.g., 13 bits wide, to provide sufficient coverage of the maximum value of an object that may be somewhat less than 8K.
Age queues may use a single storage RAM 321 that has 8K locations in it. These locations can be dynamically allocated to 32 separate queues (one for each SQ/VC) with each consisting of a linked-list of locations within the storage RAM 321. This gives each SQ/VC the ability to take more space as needed.
An age queue can be created with a front pointer pointing to the front of the queue, and a next pointer for each location pointing the next item in the queue. The last location in the queue may be indicated by a back pointer. Items are taken from the front of the queue and inserted at the back of the queue. In addition to the above data structures, each queue has a FIFO of entries at its head. These FIFOs may ensure that a queue can sustain a request every clock with a multi-clock read access time from the request RAM. When a new request arrives, if the head FIFO for that queue is not full, it bypasses the request RAM, and can be written directly into the head FIFO. Once requests for a given age queue are being written to the request RAM subsequent requests are also written to the request RAM to maintain order. The bypass path can be used again once there are no more requests for that age queue in the request RAM and there is room are the head FIFO. When a request is read from a head FIFO, and there are corresponding requests queued in the request RAM, a dequeue is initiated. One head FIFO 328 may be read at a time, such that a single dequeue operation can be initiated each clock period. Logic may be included to handle the various race conditions between an ongoing or imminent enqueue operation and a head FIFO being read.
The aforementioned ECC protection used in the age queue RAM can be extended to the FIFOs 328 to protect the data path flops. The resulting structure may include 8K flops (32 queues×5 deep×SQ-bits wide). When generating the ECC, the age queue number can be included in the calculation (but not stored) as an extra check of the free list management. When the ECC is checked, the request can be considered to be in error if there is an MBE or there is an SBE in the queue number bits.
A free list RAM can be a simple FIFO which is initialized with pointers to all 8K entries whenever a reset is performed. A count can be maintained to keep track of how may entries are valid within the free list. As entries are taken, they are popped off the front of the FIFO and used. As entries are returned, they are pushed onto the back of the FIFO. Some number of entries, e.g., three entries,) at the head of the free list can be kept in flops so they are available for quick access. As with the head FIFOs for the age queues, ECC is carried through the flops to provide protection. The resulting structure may have minimal flops (57=3 deep×19-bits wide).
In order to support full performance for small packets, age queues support both an enqueue and a dequeue every clock period. The operations across the data structures for an enqueue operation are discussed below, and can differ depending on whether the queue being written is empty or not.
In some cases, a simultaneous enqueue and dequeue to a specific queue is easily handled as they are using and updating separate fields. Some specialized scenarios may arise, e.g., when a dequeue operation empties the age queue. In order to handle this scenario, a dequeue occurs first logically, followed by an enqueue operation. Accordingly, an empty flag is seen as being set when the queue is emptied by the dequeue operation, and then cleared due to the enqueue operation.
The arbitration alluded to above can be performed among requests that are permitted to be granted subject to input buffer management, output buffer management, and flow channel quotas. Arbitration can also be halted if there are no credits for the OFCT input FIFO. In some embodiments, arbitration may be performed at two levels. First, traffic shaping arbitration can be used to arbitrate between the SQs. A Deficit Round-robin arbitration can be used to arbitrate between VCs within a given SQ. Traffic shaping arbitration may use a series of token buckets to control the bandwidth of each SQ as follows: eight leaf buckets, one for each SQ; four branch buckets; and a single head bucket.
Arbitration can be divided into three groups with a first group having the highest priority, followed by a second group, which in turn is followed by a third group. For the first and second groups, arbitration may be handled in the same way among eligible SQs. A ×8 round-robin arbitration can be performed between the SQs for each of the eight priority levels (eight parallel round-robin arbitrations). A fixed arbitration can be performed between priority levels. For example, group 3 arbitration has no priorities, and therefore is simply a single ×8 round-robin arbitration.
For arbitration in the first group, the priority for each comes from the setting in the leaf buckets. For arbitration in the second group, priority comes from the setting in the branches of the leaf buckets. In all cases, the buckets which are checked to be eligible for that group, are also the buckets from which packet size tokens are obtained if that request wins arbitration.
Regarding age queue selection, packets can be classified in order to select the SQ to which their request is forwarded. This allows traffic associated with an application to be shaped differently from traffic originating from a different application or a different traffic class. This can be useful on the edge ports which connect to a NIC in that the applications will have been configured to use a share of the resources on the node, and similarly will be granted a proportion of the network bandwidth. In accordance with one embodiment, this classification is performed by classifying the packets into a traffic class identifier (FTAG), e.g., a 4-bit code that is part of the fabric frame header, and a VLAN ID (VNI) as the packet ingresses into the fabric. The FTAG and VNI may then be used as the packet egresses the fabric to select the shaping queue.
A configuration register can be used to map FTAGs to SQs. This configuration matches the corresponding configuration in the in queue. When the output buffer requests or returns link partner credits, it converts a given FTAG to an SQ. For packet injection, the FTAG is found in R_TF_OBUF_CFG_PFG_TX_CTRL. For test generation, the FTAG is found in the test control register. When the reduction engine (RED) requests a credit return, the FTAG is found in ret_cdtJtag. When a reduction frame is removed from the output stream and link partner credits need to be returned, the FTAG is found in the frame header.
Regarding the SQs discussed herein, each age queue may have 32 SQs that are addressed by {SQ VC}. The 3-bit SQ can be considered a shaping function, and the VC selects one of four queues within that shaping function. For Ethernet egress (edge) ports, the VC is not needed for deadlock avoidance. Accordingly, all 32 SQs can be available. In such a scenario, the SQ 330 can be selected by adding the SQ base from R_TF_OBUF_CFG_FTAG_SQ_MAP to the lower bits of the VNI. The 5-bit sum defines the {SQ,VC} to send to the age queue. It should be noted that when injecting frames on an egress port, a VNI is not available, and therefore, an SQ base can be directly used. For fabric links, the SQ is taken from the upper three bits of the SQ base. The VC can be taken from the frame header when returning credits for reduction frames, or from the appropriate control CSR (R_TF_OBUF_CFG_TEST_CTRL or R_TF_OBUF_CFG_PFG_TX_CTRL) when injecting frames.
A link partner input buffer management can depend on the type of device to which the link is attached. Devices such as switch 202 may use credit-based flow control where each credit represents a cell of storage in the input buffer. Other devices may use standard Ethernet pause or priority pause-based flow control. Requests which are marked to terminate locally (lac term set) need not consider link partner input buffer flow control and need not update any associated counters. Link partner space need not be considered when the link is in the draining state.
For credit-based flow control, the link partner input buffer can be divided into eight buffer classes. Each SQ can be assigned to one of these 8 buffer classes. Credits are maintained for each of the buffer classes with each credit representing 32 bytes of storage in the link partner input buffer. In order to allow credit-based flow control to work with various devices (switch, enhanced NIC), each of which may have different cell sizes, the cell size is a programmable value in units of 32 bytes.
There may be two sets of VCs with each SQ assigned to one set. A maximum frame size worth of space can be reserved for each VC, and each VC set can have a different maximum frame size. The remainder of the link partner input buffer is shared dynamic space usable by any SQ/VC, subject to per VC and buffer class limits.
The size that comes with the request represents the size of the packet on the wire which includes a 4-byte FCS. This gets converted to an internal 2-byte FCS at the link partner before writing the packet to the link partner input buffer so the crediting needs to account for this difference, which can be a factor at the boundary of the cell size. For instance, for a 96 byte cell, a size that is 97 or 98 will take a single cell. In order to know when this happens, the request includes a correction term which is calculated as: req.len_correct=(byte_len % 16)==1 or 2.
Further validation of this term is required to convert it to whatever the cell size boundary may be. It will be valid when the length just exceeds the cell size. With this, the validated fen_correct term can be determined by: len_correct=(((16-byte size) % (2*32-byte cell size))==1) & req. len_correct
An example of how these values work for a few cell and packet sizes is illustrated in the table below:
The size that comes with the request uses 8-byte units and the link partner input buffer cell size is a multiple of 32 bytes (32*y where y=cell size from CSR). First, the 8-byte size is converted to a 16-byte size (ROUNDUP((8-byte size)/2)). Also, the cell size is converted to 16 byte units (2*y). Mathematically, the number of cells a request will use can be calculated by: ROUNDDN(((16-byte size)+2*y−1−len_correct)/(2*y))=# of cells
While a divide operation is possible in hardware, due to timing reasons, a divide operation cannot be done in the critical path of the arbitration. Instead, an alternate credit management is used. That is, credits are maintained in units of 32 bytes. When a request wins arbitration, the number of credits taken is adjusted by the maximum error term (2*y−1) using the calculation: ROUNDDN(((16-byte size)+2*y−1)/2)=Maximum 32 byte credits needed. Because this calculation overestimates the credit required for the packet, on the following clock, a modulo operation (X=(16-byte size) MOD 2*y, y=32-byte cell size from CSR) van be performed to determine what the actual remainder is. This value along with the len_correct term are used to adjust the credit counter. The formula used to create the adjustment value (adf_val) for Xis: If (X==0) adj_val=y−1 else if (X==1 and fen_correct) adj_val=y else adj_val=ROUNDDN((X−1)/2)
The table below illustrates a request credit example for 96 byte cells showing the values used across several packet lengths for the 96 byte cells of the switch input buffer (y=3).
If a request is filtered before being forwarded to the link partner input buffer, the output buffer logic returns the SQ and VC so they can be used to return the credits to the appropriate credit counters. No size is required since the packet size is always the same, the length of a reduction frame (69-byte or 16-byte size=5).
The local (master) side of the link maintains a count of the number of packets sent from each VC across both sets (8 total), a count of amount of packet (in 3 2-byte quantities) sent to each VC (4), and a count of the amount of packet (in 32-byte quantities) sent for each buffer class(8). The link partner (slave) side of the link maintains the same set of counts with them being sent over the link periodically. The difference between the master and slave counts is a count of the number of packets in the link partner input buffer from each VC across both sets and a count of the amount of space (in 32-byte quantities) currently occupied by each VC and each buffer class. A count is also maintained of the total amount of space used across all packets. A summary of the counters is as follows: master_vcx_cnt[4]/slave_vcx_cnt[4]—master and slave counts of the number of packets sent to each VC in set X; master_vcy_cnt[4]/slave_vcy_cnt[4]—master and slave counts of the number of packets sent to each VC in set Y; master_bc_cnt[8]/slave_bc_cnt[8]—master and slave counts of the amount of space occupied by each buffer class in units of 32-bytes; master_vc_cnt[4]/slave_vc_cnt[4]—master and slave counts of the amount of space occupied by each VC in units of 32-bytes; master-tot-cnt/slave-tot—cnt—master and slave counts of the total amount of space occupied in units of 32-bytes.
All counters are set to zero on a warm reset. They are also forced to zero when the link is in the draining state or when the DBG_RESET CSR bit to clear their state is set. The output buffer filter will steer a reduction packet to something other than the path to the link partner input buffer. In this case, a signal can be returned along with the SQ and VC of the packet. Again, the length is not required as the size of these packets is fixed. This information is used to adjust the appropriate master credit counts.
A request is allowed to participate in arbitration if either its VC count is 0 (indicating its one statically assigned slot is available) or there is space for a max sized frame in the dynamic space (subject to the targeted buffer class and VC limits). There can be a single programmable value for max frame size which is used across all VCs and SQs. The request validation for input buffer space can be addressed using credit-based flow control.
Credit-based flow control can be used to divide a dynamic space in two ways, each independent of each other: first, based on a limit of how much dynamic space each of the four VCs can take; and second, based on a limit to how much dynamic space each of the eight buffer classes can take. In both cases, the limits are set as a percentage of the available space. For a given packet, space should be made available in both its targeted VC and buffer class. For instance, if each space has its limit set to 50%, if one is active, it has access to 50% of the buffer space, with two active, each space gets 37.5% ((50+50*0.5)/2), with three active, each space gets 29.2% ((75+25*0.5)/3), and so on. Also, the total space used by those spaces that are active can be limited to the given total (50%, 75%, 87.5%). Accordingly, the space allocated to each varies dynamically by how many are currently active. When an additional one goes active it causes others that are active to give up some of their space which is then taken by the new one.
Like the division function discussed above, this function is implemented as a lookup table. For the VC space in this example, there are 16 entries with each entry specifying the space available to each VC along with the total space available across all VCs. For the buffer classes, there may be 256 entries with each entry specifying the space available to each buffer class along with the total space available across all buffer classes. Space for each is expressed in 2048-byte units. The depth of each table is sufficient to cover all combinations of active members (VCs or buffer classes), with each being able to have an independent setting for their percentages. With this, it is up to software to program the values in the table according to how much total dynamic space there is and what percentage each is allowed to take across all possible combinations. As more become active, each is allowed less space and the total available increases. Requests for spaces that are above this limit, or in total above the total limit, are not allowed to take more dynamic space.
A VC or buffer class is considered active either if it has a request in an age queue, or if it has outstanding credits for link partner input buffer space. As an example, consider there are only 4 spaces (16 entry table) with percentages set as SPACE0(50%), SPACE1(40%), SPACE2(30%), SPACE3(10%), and a total dynamic space of 16 KB. This results in the values, in quantities of 16-bytes presented in the buffer space example table below.
As an example, the values in the row for index 7 are calculated as: Total %=0.5+(1−0.5)*0.4+(1−0.5−(1−0.5)*0.4)*0.3=0.79; SPACEO=(0.5/(0.5+0.4+0.3))*0.79*1024=337; SPACEI=(0.4/(0.5+0.4+0.3))*0.79*1024=270; SPACE2=(0.3/(0.5+0.4+0.3))*0.79*1024=202; Total=337+270+202=809
As noted above, and referring back to
Routing in the switch fabric may be controlled by a fabric routing function (FRF) implemented in switch 202. An example FRF component 500 is illustrated in
In particular, FRF component 500 determines a preferred port with preferred ports discriminator 502B to forward each frame presented at the input port based on: a received frame's destination fabric address (DFA); the frame's current routing state (where the frame is along its path, and the path(s) it took to reach its current routing state); the switch fabric routing algorithm and configuration; and load metrics associated with the output port (the aforementioned preferred port to which the frame is to be forwarded) using busy ports filters.
FRF component 500 may include a routing algorithm table 508 that may be embodied as a software configurable table that determines valid choices based on the frame's current routing state. Valid choices are decisions such as whether a local minimal, global minimal, local non-minimal, or global non-minimal path is allowed to be chosen for the frame's next hop. The routing state includes information such as the VC the frame was received on, and whether it is in the source, the destination, or an intermediate group. The routing algorithm table 508, along with the adaptive selection function or logic 512A (described below), also determines the VC to be used for the frame's next hop.
Frame routing with unicast DFAs will be described as an example. However, it should be noted that the DFA of the routing request can either be in unicast of multicast format. The unicast format can include a 9-bit global ID field (global_id), a 5-bit switch ID field (switch_id), and a 6-bit endpoint ID field (endpoint_id). The global ID can uniquely identify a group within the network. Specifically, it identifies the final group to which the frame must be delivered. The switch ID uniquely identifies a switch within the group identified by global ID. The endpoint ID field, together with the global ID and switch ID identify the endpoint, connected to the edge of the network fabric, to which the frame is to be delivered. This field is mapped to a port or set of ports on the switch identified by global ID and switch ID.
The multicast format includes a 13-bit multicast ID field (multicast_id). This field is mapped by FRF component 500 to a set of ports on the current switch to which the frame is to be forwarded.
From this information, FRF component 500 determines an updated routing state for the frame, which is then carried within the frame. For example, to effectuate routing in a dragonfly topology, a frame's current state may be gleaned from the frame's VC (discussed above). Based on algorithmic switch fabric routing rules specified for the switch fabric (the selection of which is described below), FRF component 500 determines a particular VC to be used for the frame's next hop to avoid any deadlocks. Additional routing state information can be provided depending on where the frame is along its path, e.g., whether the frame is in its source group, in an intermediate group, or in its destination group. It should be noted that FRF component 500 performs port filtering (described in greater detail below) using permitted ports filter, operational ports filter, busy ports filters, etc. to determine if a preferred port to which a frame is to be forwarded is currently faulty, busy, absent, etc.
FRF component 500 receives load measurements associated with its output port from switch 202. Load information for the FRF component 500's input port is also received from a neighboring switch. In some embodiments, FRF component 500 exchanges its input port and output port load information with all other FRF component instances within a switch, e.g., switch 202, and with neighboring switches in the switch fabric. In this way, each FRF component instance of each switch in the switch fabric is aware of summary load information for all neighboring switches.
It should be noted that FRF component 500 can support frame multicasting. When a multicast DFA is received, FRF component 500 determines a set of ports to which the frame associated with the multicast DFA should be forwarded. The set of ports can be determined by accessing a lookup table that maps software-configured multicast fabric addresses to output ports. This avoids problems associated with duplicate multicast frame copies.
Per Traffic Class Routing
Traditionally, Traffic Classes (TCs) and Quality of Service (QoS) are mechanisms employed to guarantee predictable runtimes and improve application performance by providing service level guarantees based on allocation of network resources and properties such as traffic priority, bandwidth share, or maximum latency period. Typically, these mechanisms relate to latency and bandwidth characteristics. As alluded to above, the per traffic class routing techniques disclosed herein affect characteristics outside of latency and bandwidth, including routing behavior, application data flow separation, and congestion control on a per traffic class basis. Accordingly, the disclosed techniques enable independent, fine grained control of network data transit, capable of directly supporting HPC application, service and workflow requirements in addition to more traditional Ethernet and TCP network characteristics.
For example, TCs are currently established in the Ethernet space, but many of the classifications and behaviors are best suited to Internet or Data Center applications and traffic patterns. These classifications are largely achieved by affecting network hardware device packet queues, buffers and the arbitration over them, to determine how data packets are progressed from ingress to egress.
The disclosed techniques support already established classifications, while specifically focusing on HPC applications, HPC-related characteristics, and HPC-related requirements for particular network behaviors and workflows. To better support HPC applications and services, mechanisms have been built into the switches (shown in
In addition to network data flow based classifications, HPC application specific mappings are supported by this new model. Classifications of inter-process communications within an HPC application can be performed in a number of ways, including but not limited to:
According to the embodiments, an application's traffic flow is identified and associated with particular TCs by marking packets within a flow on egress from a node's network interface into the network. A field in the packet header carries a Code Point, which is a bitmap indicating one or more preferred network behaviors. During classification by the network, traffic is divided into several classes, each being treated in a specifically different manner, subject to different routing, precedence, shaping, and scheduling rules.
Network implements network traffic classification, shaping, and control policy in a similar way to the Differentiated Services and Precedence models in which packets belonging to an application's data flows are tagged by setting bits in reserved fields of the packet header, called the Differentiated Service Code Point (DSCP) and Precedence Code Point (PCP) respectively. These Code Point bits are used in classification by the network in order to choose from a number of network-wide, pre-defined traffic classifications, mapping Code Points to the network actions that implement a particular behavior. Both established Ethernet and TCP TCs and HPC specific classifications are supported using this model.
In traditional systems, standard DSCPs invoke the corresponding standard behaviors, the mapping being performed in a consistent manner across the network, spanning from network ingress to egress. These common mappings poorly define the specialized behavior typical of an HPC application. Thus, the disclosed techniques use HPC-related TCs to identify traffic flows concerned with low-latency, transactional operations such as synchronization barriers or reductions; with traffic that was part of a bulk data transfer; or traffic associated with a particular application or user that required preferential network access.
In addition to these HPC application traffic classifications, Network-enabled fabrics can specify, configure, and control many of the performance differentiating features designed into the Switch ASIC. Adaptive routing bias, in-order/out-of-order delivery, and lossy or lossless delivery marking can be associated with particular TCs.
Referring now to
Generally, Traffic Classes 430 map well to the operational requirements of HPC applications. In addition, these classifications allow a way to specify relative application priority, making a distinction between time or business critical applications and other jobs. Several individual classes can be included into the traffic classes 430 employed by switch 430. In the example, the individual traffic classes are shown to include: low latency class 431; dedicated access class 432; bulk data class 433; best efforts class 434; and scavenger class 435. However, it should be appreciated that the individual traffic classes 431-435 shown are for purposes of discussion and are not intended to be limiting. Therefore, other types of HPC-related traffic classes can be employed in accordance to the embodiments disclosed herein, as deemed appropriate.
The low Latency class 431 supports low latency, low jitter data patterns. Low latency can be typically caused by transactional data exchanges, barrier synchronizations, and collective operations. Specified maximum latencies are guaranteed by the low latency class 431. Guarantees of service in the low latency class 431 will typically require accompanying bandwidth caps and packet size limitations so as to not consume excessive bandwidth at high priority.
The dedicated access class 432 provides a category that will operate at the highest priority. For example, dedicated access class 432 can include a high bandwidth allocation, maximum guaranteed latency, and the highest scheduling and arbitration priority. The dedicated access class 432 can have absolute priority over all other classes.
The bulk data class 433 can be used primarily for I/O and is there to separate sustained data transfer from other application inter-process communications.
The best efforts class 434 can serve as a default, shared class. The best efforts class 434 can carry the traffic for a number of applications executing concurrently over the same network infrastructure. Even though the best efforts class 434 is shared, network capacity and resource allocation are distributed fairly across the applications.
The scavenger class 435 can be used for data that is desired, but does not have strict delivery requirements. An example of example of traffic that may be included in the scavenger class 435 is monitoring data, especially for application specific monitoring such as with performance tools. Global monitoring could also be done this way for data that is too voluminous for out-of-band transport. Using the scavenger class 435 makes sure that such communication does not interfere with communication doing “real” work (e.g., high priority).
In general, a particular Traffic Class's per-hop behavior (PHB), defining how packets or frames are forwarded, can change at each network router, by reclassification of a packet's Code Point. Logically, the complete switch fabric, made of many interconnected switch ASICs, can be viewed as a single logical network entity. As such, classification according to the embodiments can be performed at network fabric ingress and is carried unchanged to Network Fabric egress. Thus, switch 402 performing the traffic classification, can be an ingress into the switch fabric.
Fabric Specific Tag for Traffic Classification
On ingress to the fabric, the switch 402 can receive a packet or frame, and parse that packet. Thereafter, either the DSCP of the header or the PCP of an Ethernet's VLAN fields is used as the primary source to generate a fabric specific tag (Ftag) indicating the traffic class that is assigned to that packet. If the DSCP is being used, there is an implicit DSCP to PCP mapping also defined. Subsequently, the Ftag can be used as a direct or indirect mapping into switch resources, such as input buffer space allocation and traffic shaping queues where subsequent traffic arbitration is performed. Other fields of the header can also be used to influence the final Ftag value. The Ftag can be a 4 bit field, which is ultimately used to control the behavior as the packet crosses the whole network fabric. As alluded to above, the FTag value is calculated when a frame enters the fabric from an Ethernet port and then is preserved, unchanged in the fabric header, as the frame traverses the Network fabric.
In an example, as Ftag progresses through the fabric, the 16 values of FTag can be mapped onto 8 values of Shaping Queue (SQ). The SQ value is passed over the request crossbar to the Age queues in the output port. The shaping function controls the arbitration performed in AGEQ.
There is a subsequent mapping from SQ to Buffer Class (BC) in the AGEQ. The BC is used to divide up the available space in the link partner's input buffer. This additional mapping allows some aggregation of the input buffer for low latency classes that do not need much bandwidth and hence little buffer space.
As there is a fixed mapping of FTag to SQ and a subsequent fixed mapping of SQ to BC, there is an implicit fixed mapping directly from FTag to BC. It is a requirement of the management software to ensure these mappings are completely consistent across the whole network fabric for all active FTags, SQs and BCs. It is possible for an FTag to be made inactive so that the software can reorganize these mappings on a busy network but this will be a heavyweight operation.
This FTag is then added to the fabric header as the packet is processed by all the functional blocks of all the switches it passes through before it egresses back onto another Ethernet link. For some transport protocols, a reverse mapping from FTag back to DSCP or from FTag back to PCP may be required. In this case, the reverse mapping can be performed in the final Switch's EEG block. On egress from the fabric, the internal Ftag can be translated back into a DSCP or PCP if the network fabric edge port is connected to a device supporting Portals or Ethernet, thereby preserving the ingress Code Point end-to-end.
The forwarding behavior (or PHB) of the network fabric can be determined by the Ftag. The Ftag can be used to implement a number of QoS categories, supporting both observable and non-observable network behavior, with the ultimate goal of providing predictable, scalable, high performance application execution for a range of application workloads and traffic. QoS categories that may be based on the Ftag, and thus the traffic class indicated by the Ftag, can include:
Routing in the switch fabric may be controlled by a fabric routing function (FRF) implemented in switch 202. An example FRF component 500 is illustrated in
In particular, FRF component 500 determines a preferred port with preferred ports discriminator 502B to forward each frame presented at the input port based on: a received frame's destination fabric address (DFA); the frame's current routing state (where the frame is along its path, and the path(s) it took to reach its current routing state); the switch fabric routing algorithm and configuration; and load metrics associated with the output port (the aforementioned preferred port to which the frame is to be forwarded) using busy ports filters.
FRF component 500 may include a routing algorithm table 508 that may be embodied as a software configurable table that determines valid choices based on the frame's current routing state. Valid choices are decisions such as whether a local minimal, global minimal, local non-minimal, or global non-minimal path is allowed to be chosen for the frame's next hop. The routing state includes information such as the VC the frame was received on, and whether it is in the source, the destination, or an intermediate group. The routing algorithm table 508, along with the adaptive selection function or logic 512A (described below), also determines the VC to be used for the frame's next hop.
Frame routing with unicast DFAs will be described as an example. However, it should be noted that the DFA of the routing request can either be in unicast of multicast format. The unicast format can include a 9-bit global ID field (global_id), a 5-bit switch ID field (switch_id), and a 6-bit endpoint ID field (endpoint_id). The global ID can uniquely identify a group within the network. Specifically, it identifies the final group to which the frame must be delivered. The switch ID uniquely identifies a switch within the group identified by global ID. The endpoint ID field, together with the global ID and switch ID identify the endpoint, connected to the edge of the network fabric, to which the frame is to be delivered. This field is mapped to a port or set of ports on the switch identified by global ID and switch ID.
The multicast format includes a 13-bit multicast ID field (multicast_id). This field is mapped by FRF component 500 to a set of ports on the current switch to which the frame is to be forwarded.
From this information, FRF component 500 determines an updated routing state for the frame, which is then carried within the frame. For example, to effectuate routing in a dragonfly topology, a frame's current state may be gleaned from the frame's VC (discussed above). Based on algorithmic switch fabric routing rules specified for the switch fabric (the selection of which is described below), FRF component 500 determines a particular VC to be used for the frame's next hop to avoid any deadlocks. Additional routing state information can be provided depending on where the frame is along its path, e.g., whether the frame is in its source group, in an intermediate group, or in its destination group. It should be noted that FRF component 500 performs port filtering (described in greater detail below) using permitted ports filter, operational ports filter, busy ports filters, etc. to determine if a preferred port to which a frame is to be forwarded is currently faulty, busy, absent, etc.
FRF component 500 receives load measurements associated with its output port from switch 202. Load information for the FRF component 500's input port is also received from a neighboring switch. In some embodiments, FRF component 500 exchanges its input port and output port load information with all other FRF component instances within a switch, e.g., switch 202, and with neighboring switches in the switch fabric. In this way, each FRF component instance of each switch in the switch fabric is aware of summary load information for all neighboring switches.
It should be noted that FRF component 500 can support frame multicasting. When a multicast DFA is received, FRF component 500 determines a set of ports to which the frame associated with the multicast DFA should be forwarded. The set of ports can be determined by accessing a lookup table that maps software-configured multicast fabric addresses to output ports. This avoids problems associated with duplicate multicast frame copies.
Routing Using Traffic Classes
Referring now to
As described throughout, a switch provides system wide traffic classes, and with them the ability for per traffic class controls. The per traffic class controls can govern how traffic is managed in the network. For instance, how a packet is routed and an amount of bandwidth that is allocated can be directly based on its assigned traffic class. As an example, traffic shaping operates on traffic classes. Where there is contention for network bandwidth, the arbiters select packets to forward based on their traffic class and the credits available to that class. Additionally, the network supports minimum and maximum bandwidths for each traffic class. The ability to manage bandwidth provides the opportunity to dedicate network resources to an application as well as CPUs and memory bandwidth.
Particularly, a per traffic class control includes a routing policy that considers the traffic class of a packet in making routing decisions.
The process 600 begins at operation 606, where a plurality of packets can be received by at switch fabric ingress port. For example, data traffic can enter a network as a result of an various HPC applications communicating to destinations via the switch fabric. The data traffic can enter the switch fabric at a ingress port, for instance as a plurality of packets received by an ingress edge switch. Different applications can generate different types of data, the data having distinct characteristics. Thus, different flows from among the plurality of packets received in operation 606 (corresponding to the different types of data) may have characters belonging to several individual traffic classes. In other words, data traffic can comprise data having different traffic classes.
Next, at operation 608, a traffic class for a packet (from the plurality of packets) is determined. Operation 608 can involve the switch parsing the packet, and analyzing a field including the PCP (and/or DSCP) in the packet header. The PCP can be indicative of the application that generated the packet, or indicate a characteristics related to the its network behavior. In some embodiments, values (e.g., bits) of the PCP are mapped to certain traffic classes defined and known by the switch. Thus, the based on the PCP within the packet's header, the switch can use this mapping to determine the corresponding traffic class for the packet. The traffic classes are based on HPC-related characteristics, and can include: low latency class; dedicated class; bulk data class; best efforts class; and scavenger class. Each of the traffic classes are described in greater detail in reference to
As previously described, the ingress edge switch can perform classification, and assign traffic classes to the plurality of packets upon entering the switch fabric (that is maintained until the point of egress). Thus, the operation 608 does not need to be performed at each switch encountered in the fabric by that packet. After a traffic class is determined for the packet, the process 600 can proceed to operation 610
Subsequently, at operation 610, a fabric specific tag can be generated for the packet based on the determined traffic class. The fabric specific tag indicates the particular traffic class the packet has been assigned. The fabric specific tag can be a Ftag, as described in detail above. In some cases, generating the Ftag involves a translation of the PCP into the Ftag value. Thus, the fabric specific tag serves as a marking in the packet that is recognizable within the fabric, and used to further classify the packet into its correspondingly assigned traffic class. In other words, within the network fabric, the packet's traffic class is identified by an FTAG value.
Thereafter, at operation 612, a check may be performed to determine whether any routing directives for the packet are dependent on traffic classification vis-à-vis the fabric specific tag. As described in detail regarding
In operation 614, the FRF can determine a particular routing directive that corresponds to the particular traffic class of the packet (as indicated by its Ftag). As an example, the FTAG can be used as a bias value in adaptive routing. According to adaptive routing, the bias value can vary depending on the type path to which it is being applied (non-preferred minimal, preferred minimal, and non-minimal), the traffic class of the packet being routed, and where the packet is along its path. For example, packets in a low-latency traffic class can be more strongly biased toward minimal paths than packets in other traffic classes.
Once a packet is tagged in previous operation 610, there are a number of per traffic class routing controls that can implemented in the switch. The traffic class of a packet can be used in operation 614 to conduct:
In some embodiments, operation 614 can perform additional operations based on traffic class in addition to (or in leui of) routing, such as: data-flow separation, ordered or unordered data delivery, lossy or lossless transit, telemetry gathering and traffic shaping rules.
Referring back to operation 612, if it determined that there are no routing directives that are defined per traffic class, then the operation can continue to operation 616 and route the packets based on other routing policies employed by the switch. For example, the packet may be routed based on adaptive routing techniques that are not dependent on the packet's traffic class.
The computer system 700 also includes a main memory 706, such as a random access memory (RAM), cache and/or other dynamic storage devices, coupled to bus 702 for storing information and instructions to be executed by processor 704. Main memory 706 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 704. Such instructions, when stored in storage media accessible to processor 704, render computer system 700 into a special-purpose machine that is customized to perform the operations specified in the instructions.
The computer system 700 further includes a read only memory (ROM) 708 or other static storage device coupled to bus 702 for storing static information and instructions for processor 704. A storage device 710, such as a magnetic disk, optical disk, or USB thumb drive (Flash drive), etc., is provided and coupled to bus 702 for storing information and instructions.
The computer system 700 may be coupled via bus 702 to a display 712, such as a liquid crystal display (LCD) (or touch screen), for displaying information to a computer user. An input device 714, including alphanumeric and other keys, is coupled to bus 702 for communicating information and command selections to processor 704. Another type of user input device is cursor control 716, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 804 and for controlling cursor movement on display 712. In some embodiments, the same direction information and command selections as cursor control may be implemented via receiving touches on a touch screen without a cursor.
The computing system 700 may include a user interface module to implement a GUI that may be stored in a mass storage device as executable software codes that are executed by the computing device(s). This and other modules may include, by way of example, components, such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables.
In general, the word “component,” “engine,” “system,” “database,” data store,” and the like, as used herein, can refer to logic embodied in hardware or firmware, or to a collection of software instructions, possibly having entry and exit points, written in a programming language, such as, for example, Java, C or C++. A software component may be compiled and linked into an executable program, installed in a dynamic link library, or may be written in an interpreted programming language such as, for example, BASIC, Perl, or Python. It will be appreciated that software components may be callable from other components or from themselves, and/or may be invoked in response to detected events or interrupts. Software components configured for execution on computing devices may be provided on a computer readable medium, such as a compact disc, digital video disc, flash drive, magnetic disc, or any other tangible medium, or as a digital download (and may be originally stored in a compressed or installable format that requires installation, decompression or decryption prior to execution). Such software code may be stored, partially or fully, on a memory device of the executing computing device, for execution by the computing device. Software instructions may be embedded in firmware, such as an EPROM. It will be further appreciated that hardware components may be comprised of connected logic units, such as gates and flip-flops, and/or may be comprised of programmable units, such as programmable gate arrays or processors.
The computer system 700 may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and/or program logic which in combination with the computer system causes or programs computer system 700 to be a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system 700 in response to processor(s) 704 executing one or more sequences of one or more instructions contained in main memory 706. Such instructions may be read into main memory 706 from another storage medium, such as storage device 710. Execution of the sequences of instructions contained in main memory 706 causes processor(s) 704 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.
The term “non-transitory media,” and similar terms, as used herein refers to any media that store data and/or instructions that cause a machine to operate in a specific fashion. Such non-transitory media may comprise non-volatile media and/or volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device 710. Volatile media includes dynamic memory, such as main memory 706. Common forms of non-transitory media include, for example, a floppy disk, a flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge, and networked versions of the same.
Non-transitory media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between non-transitory media. For example, transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus 702. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.
The computer system 700 also includes a communication interface 718 coupled to bus 702. Network interface 718 provides a two-way data communication coupling to one or more network links that are connected to one or more local networks. For example, communication interface 718 may be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, network interface 718 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN (or WAN component to communicated with a WAN). Wireless links may also be implemented. In any such implementation, network interface 718 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.
A network link typically provides data communication through one or more networks to other data devices. For example, a network link may provide a connection through local network to a host computer or to data equipment operated by an Internet Service Provider (ISP). The ISP in turn provides data communication services through the world wide packet data communication network now commonly referred to as the “Internet.” Local network and Internet both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link and through communication interface 718, which carry the digital data to and from computer system 700, are example forms of transmission media.
The computer system 700 can send messages and receive data, including program code, through the network(s), network link and communication interface 718. In the Internet example, a server might transmit a requested code for an application program through the Internet, the ISP, the local network and the communication interface 718.
The received code may be executed by processor 704 as it is received, and/or stored in storage device 710, or other non-volatile storage for later execution.
Each of the processes, methods, and algorithms described in the preceding sections may be embodied in, and fully or partially automated by, code components executed by one or more computer systems or computer processors comprising computer hardware. The one or more computer systems or computer processors may also operate to support performance of the relevant operations in a “cloud computing” environment or as a “software as a service” (SaaS). The processes and algorithms may be implemented partially or wholly in application-specific circuitry. The various features and processes described above may be used independently of one another, or may be combined in various ways. Different combinations and sub-combinations are intended to fall within the scope of this disclosure, and certain method or process blocks may be omitted in some implementations. The methods and processes described herein are also not limited to any particular sequence, and the blocks or states relating thereto can be performed in other sequences that are appropriate, or may be performed in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The performance of certain of the operations or processes may be distributed among computer systems or computers processors, not only residing within a single machine, but deployed across a number of machines.
As used herein, a circuit might be implemented utilizing any form of hardware, software, or a combination thereof. For example, one or more processors, controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logical components, software routines or other mechanisms might be implemented to make up a circuit. In implementation, the various circuits described herein might be implemented as discrete circuits or the functions and features described can be shared in part or in total among one or more circuits. Even though various features or elements of functionality may be individually described or claimed as separate circuits, these features and functionality can be shared among one or more common circuits, and such description shall not require or imply that separate circuits are required to implement such features or functionality. Where a circuit is implemented in whole or in part using software, such software can be implemented to operate with a computing or processing system capable of carrying out the functionality described with respect thereto, such as computer system 700.
As used herein, the term “or” may be construed in either an inclusive or exclusive sense. Moreover, the description of resources, operations, or structures in the singular shall not be read to exclude the plural. Conditional language, such as, among others, “can,” “could,” “might,” or “may,” unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements and/or steps.
Terms and phrases used in this document, and variations thereof, unless otherwise expressly stated, should be construed as open ended as opposed to limiting. Adjectives such as “conventional,” “traditional,” “normal,” “standard,” “known,” and terms of similar meaning should not be construed as limiting the item described to a given time period or to an item available as of a given time, but instead should be read to encompass conventional, traditional, normal, or standard technologies that may be available or known now or at any time in the future. The presence of broadening words and phrases such as “one or more,” “at least,” “but not limited to” or other like phrases in some instances shall not be read to mean that the narrower case is intended or required in instances where such broadening phrases may be absent.
Under 35 U.S.C. 119, this claims the benefit and right of priority of U.S. Provisional Patent Application No. 62/852,273, filed May 23, 2019, entitled “Network Switch,” U.S. Provisional Patent Application No. 62/852,203, filed May 23, 2019, entitled “Network Interface Controller,” and U.S. Provisional Patent Application No. 62/852,289, filed May 23, 2019, entitled “Network Computer System,” the disclosures of which are incorporated by reference herein.
The invention(s) described herein were made with U.S. Government support under one or more of the contracts set forth below. The U.S. Government has certain rights in these inventions. Contract TitleCustomer/AgencyContract ReferenceFastForward-2Lawrence Livermore NationalSubcontract B609229 under primeSecurity, LLC/Dept of Energycontract DE-AC52-07NA27344BeePresentMaryland Procurement OfficeH98230-15-D-0020; Delivery Order 003SeaBiscuitMaryland Procurement OfficeII98230-14-C-0758PathForwardLawrence Livermore NationalSubcontract B620872 under primeSecurity, LLC/Dept of Energycontract DE-AC52-07NA27344DesignForwardThe Regents of the UniversitySubcontract 7078453 under primeof California/Dept of Energycontract DE-AC02-05CII11231DesignForward-2The Regents of the UniversitySubcontract 7216357 under primeof California/Dept of Energycontract DE-AC02-05CII11231
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2020/024340 | 3/23/2020 | WO |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2020/236301 | 11/26/2020 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 4807118 | Lin et al. | Feb 1989 | A |
| 5138615 | Lamport et al. | Aug 1992 | A |
| 5457687 | Newman | Oct 1995 | A |
| 5937436 | Watkins | Aug 1999 | A |
| 5960178 | Cochinwala et al. | Sep 1999 | A |
| 5970232 | Passint et al. | Oct 1999 | A |
| 5983332 | Watkins | Nov 1999 | A |
| 6112265 | Harriman et al. | Aug 2000 | A |
| 6230252 | Passint et al. | May 2001 | B1 |
| 6246682 | Roy et al. | Jun 2001 | B1 |
| 6493347 | Sindhu et al. | Dec 2002 | B2 |
| 6545981 | Garcia et al. | Apr 2003 | B1 |
| 6633580 | Toerudbakken et al. | Oct 2003 | B1 |
| 6674720 | Passint et al. | Jan 2004 | B1 |
| 6714553 | Poole et al. | Mar 2004 | B1 |
| 6728211 | Peris et al. | Apr 2004 | B1 |
| 6732212 | Sugahara et al. | May 2004 | B2 |
| 6735173 | Lenoski et al. | May 2004 | B1 |
| 6894974 | Aweva et al. | May 2005 | B1 |
| 7023856 | Washabaugh et al. | Apr 2006 | B1 |
| 7133940 | Blightman et al. | Nov 2006 | B2 |
| 7218637 | Best et al. | May 2007 | B1 |
| 7269180 | Bly et al. | Sep 2007 | B2 |
| 7305487 | Blumrich et al. | Dec 2007 | B2 |
| 7337285 | Tanoue | Feb 2008 | B2 |
| 7397797 | Alfieri et al. | Jul 2008 | B2 |
| 7430559 | Lomet | Sep 2008 | B2 |
| 7441006 | Biran et al. | Oct 2008 | B2 |
| 7464174 | Ngai | Dec 2008 | B1 |
| 7483442 | Torudbakken et al. | Jan 2009 | B1 |
| 7562366 | Pope et al. | Jul 2009 | B2 |
| 7593329 | Kwan et al. | Sep 2009 | B2 |
| 7596628 | Aloni et al. | Sep 2009 | B2 |
| 7620791 | Wentzlaff et al. | Nov 2009 | B1 |
| 7633869 | Morris et al. | Dec 2009 | B1 |
| 7639616 | Manula et al. | Dec 2009 | B1 |
| 7734894 | Wentzlaff et al. | Jun 2010 | B1 |
| 7774461 | Tanaka et al. | Aug 2010 | B2 |
| 7782869 | Chitlur Srinivasa | Aug 2010 | B1 |
| 7796579 | Bruss | Sep 2010 | B2 |
| 7856026 | Finan et al. | Dec 2010 | B1 |
| 7933282 | Gupta et al. | Apr 2011 | B1 |
| 7953002 | Opsasnick | May 2011 | B2 |
| 7975120 | Sabbatini, Jr. et al. | Jul 2011 | B2 |
| 8014278 | Subramanian et al. | Sep 2011 | B1 |
| 8023521 | Woo et al. | Sep 2011 | B2 |
| 8050180 | Judd | Nov 2011 | B2 |
| 8077606 | Litwack | Dec 2011 | B1 |
| 8103788 | Miranda | Jan 2012 | B1 |
| 8160085 | Voruganti et al. | Apr 2012 | B2 |
| 8175107 | Yalagandula et al. | May 2012 | B1 |
| 8249072 | Sugumar et al. | Aug 2012 | B2 |
| 8281013 | Mundkur et al. | Oct 2012 | B2 |
| 8352727 | Chen et al. | Jan 2013 | B2 |
| 8353003 | Noehring et al. | Jan 2013 | B2 |
| 8443151 | Tang et al. | May 2013 | B2 |
| 8473783 | Andrade et al. | Jun 2013 | B2 |
| 8543534 | Alves et al. | Sep 2013 | B2 |
| 8619793 | Lavian et al. | Dec 2013 | B2 |
| 8626957 | Blumrich et al. | Jan 2014 | B2 |
| 8650582 | Archer et al. | Feb 2014 | B2 |
| 8706832 | Blocksome | Apr 2014 | B2 |
| 8719543 | Kaminski et al. | May 2014 | B2 |
| 8811183 | Anand et al. | Aug 2014 | B1 |
| 8948175 | Bly et al. | Feb 2015 | B2 |
| 8971345 | McCanne et al. | Mar 2015 | B1 |
| 9001663 | Attar et al. | Apr 2015 | B2 |
| 9053012 | Northcott et al. | Jun 2015 | B1 |
| 9088496 | Vaidya et al. | Jul 2015 | B2 |
| 9094327 | Jacobs et al. | Jul 2015 | B2 |
| 9178782 | Matthews et al. | Nov 2015 | B2 |
| 9208071 | Talagala et al. | Dec 2015 | B2 |
| 9218278 | Talagala et al. | Dec 2015 | B2 |
| 9231876 | Mir et al. | Jan 2016 | B2 |
| 9231888 | Bogdanski et al. | Jan 2016 | B2 |
| 9239804 | Kegel et al. | Jan 2016 | B2 |
| 9269438 | Nachimuthu et al. | Feb 2016 | B2 |
| 9276864 | Pradeep | Mar 2016 | B1 |
| 9436651 | Underwood et al. | Sep 2016 | B2 |
| 9455915 | Sinha et al. | Sep 2016 | B2 |
| 9460178 | Bashyam et al. | Oct 2016 | B2 |
| 9479426 | Munger et al. | Oct 2016 | B2 |
| 9496991 | Plamondon et al. | Nov 2016 | B2 |
| 9544234 | Markine | Jan 2017 | B1 |
| 9548924 | Pettit et al. | Jan 2017 | B2 |
| 9594521 | Blagodurov et al. | Mar 2017 | B2 |
| 9635121 | Mathew et al. | Apr 2017 | B2 |
| 9742855 | Shuler et al. | Aug 2017 | B2 |
| 9762488 | Previdi et al. | Sep 2017 | B2 |
| 9762497 | Kishore et al. | Sep 2017 | B2 |
| 9830273 | Bk et al. | Nov 2017 | B2 |
| 9838500 | Ilan et al. | Dec 2017 | B1 |
| 9853900 | Mula et al. | Dec 2017 | B1 |
| 9887923 | Chorafakis et al. | Feb 2018 | B2 |
| 10003544 | Liu et al. | Jun 2018 | B2 |
| 10009270 | Stark et al. | Jun 2018 | B1 |
| 10031857 | Menachem et al. | Jul 2018 | B2 |
| 10050896 | Yang et al. | Aug 2018 | B2 |
| 10061613 | Brooker et al. | Aug 2018 | B1 |
| 10063481 | Jiang et al. | Aug 2018 | B1 |
| 10089220 | McKelvie et al. | Oct 2018 | B1 |
| 10169060 | Vincent et al. | Jan 2019 | B1 |
| 10178035 | Dillon | Jan 2019 | B2 |
| 10200279 | Aljaedi | Feb 2019 | B1 |
| 10218634 | Aldebert et al. | Feb 2019 | B2 |
| 10270700 | Burnette et al. | Apr 2019 | B2 |
| 10305772 | Zur et al. | May 2019 | B2 |
| 10331590 | Macnamara et al. | Jun 2019 | B2 |
| 10353833 | Hagspiel et al. | Jul 2019 | B2 |
| 10454835 | Contavalli et al. | Oct 2019 | B2 |
| 10498672 | Graham et al. | Dec 2019 | B2 |
| 10567307 | Fairhurst et al. | Feb 2020 | B2 |
| 10652112 | Raney | May 2020 | B2 |
| 10728173 | Agrawal et al. | Jul 2020 | B1 |
| 10802828 | Volpe et al. | Oct 2020 | B1 |
| 10817502 | Talagala et al. | Oct 2020 | B2 |
| 11128561 | Matthews et al. | Sep 2021 | B1 |
| 11271869 | Agrawal et al. | Mar 2022 | B1 |
| 11416749 | Bshara et al. | Aug 2022 | B2 |
| 11444886 | Stawitzky et al. | Sep 2022 | B1 |
| 20010010692 | Sindhu et al. | Aug 2001 | A1 |
| 20010047438 | Forin | Nov 2001 | A1 |
| 20020174279 | Wynne et al. | Nov 2002 | A1 |
| 20030016808 | Hu et al. | Jan 2003 | A1 |
| 20030041168 | Musoll | Feb 2003 | A1 |
| 20030110455 | Baumgartner et al. | Jun 2003 | A1 |
| 20030174711 | Shankar | Sep 2003 | A1 |
| 20030200363 | Futral | Oct 2003 | A1 |
| 20030223420 | Ferolito | Dec 2003 | A1 |
| 20040008716 | Stiliadis | Jan 2004 | A1 |
| 20040059828 | Hooper et al. | Mar 2004 | A1 |
| 20040095882 | Hamzah et al. | May 2004 | A1 |
| 20040133634 | Luke et al. | Jul 2004 | A1 |
| 20040223452 | Santos et al. | Nov 2004 | A1 |
| 20050021837 | Haselhorst et al. | Jan 2005 | A1 |
| 20050047334 | Paul et al. | Mar 2005 | A1 |
| 20050088969 | Carlsen et al. | Apr 2005 | A1 |
| 20050091396 | Nilakantan et al. | Apr 2005 | A1 |
| 20050108444 | Flauaus et al. | May 2005 | A1 |
| 20050108518 | Pandya | May 2005 | A1 |
| 20050152274 | Simpson | Jul 2005 | A1 |
| 20050182854 | Pinkerton et al. | Aug 2005 | A1 |
| 20050270974 | Mayhew | Dec 2005 | A1 |
| 20050270976 | Yang et al. | Dec 2005 | A1 |
| 20060023705 | Zoranovic et al. | Feb 2006 | A1 |
| 20060067347 | Naik et al. | Mar 2006 | A1 |
| 20060075480 | Noehring et al. | Apr 2006 | A1 |
| 20060174251 | Pope et al. | Aug 2006 | A1 |
| 20060203728 | Kwan et al. | Sep 2006 | A1 |
| 20070061433 | Reynolds et al. | Mar 2007 | A1 |
| 20070070901 | Aloni et al. | Mar 2007 | A1 |
| 20070198804 | Moyer | Aug 2007 | A1 |
| 20070211746 | Oshikiri et al. | Sep 2007 | A1 |
| 20070242611 | Archer et al. | Oct 2007 | A1 |
| 20070268825 | Corwin et al. | Nov 2007 | A1 |
| 20080013453 | Chiang et al. | Jan 2008 | A1 |
| 20080013549 | Okagawa et al. | Jan 2008 | A1 |
| 20080071757 | Ichiriu et al. | Mar 2008 | A1 |
| 20080084864 | Archer et al. | Apr 2008 | A1 |
| 20080091915 | Moertl et al. | Apr 2008 | A1 |
| 20080147881 | Krishnamurthy et al. | Jun 2008 | A1 |
| 20080159138 | Shepherd et al. | Jul 2008 | A1 |
| 20080253289 | Naven et al. | Oct 2008 | A1 |
| 20090003212 | Kwan et al. | Jan 2009 | A1 |
| 20090010157 | Holmes et al. | Jan 2009 | A1 |
| 20090013175 | Elliott | Jan 2009 | A1 |
| 20090055496 | Garg et al. | Feb 2009 | A1 |
| 20090092046 | Naven et al. | Apr 2009 | A1 |
| 20090141621 | Fan et al. | Jun 2009 | A1 |
| 20090198958 | Arimilli et al. | Aug 2009 | A1 |
| 20090259713 | Blumrich et al. | Oct 2009 | A1 |
| 20090285222 | Hoover et al. | Nov 2009 | A1 |
| 20100061241 | Sindhu et al. | Mar 2010 | A1 |
| 20100169608 | Kuo et al. | Jul 2010 | A1 |
| 20100172260 | Kwan et al. | Jul 2010 | A1 |
| 20100183024 | Gupta | Jul 2010 | A1 |
| 20100220595 | Petersen | Sep 2010 | A1 |
| 20100274876 | Kagan et al. | Oct 2010 | A1 |
| 20100302942 | Shankar et al. | Dec 2010 | A1 |
| 20100316053 | Miyoshi et al. | Dec 2010 | A1 |
| 20110051724 | Scott et al. | Mar 2011 | A1 |
| 20110066824 | Bestler | Mar 2011 | A1 |
| 20110072179 | Lacroute et al. | Mar 2011 | A1 |
| 20110099326 | Jung et al. | Apr 2011 | A1 |
| 20110110383 | Yang et al. | May 2011 | A1 |
| 20110128959 | Bando et al. | Jun 2011 | A1 |
| 20110158096 | Leung et al. | Jun 2011 | A1 |
| 20110158248 | Vorunganti et al. | Jun 2011 | A1 |
| 20110164496 | Loh et al. | Jul 2011 | A1 |
| 20110173370 | Jacobs et al. | Jul 2011 | A1 |
| 20110264822 | Ferguson et al. | Oct 2011 | A1 |
| 20110276699 | Pedersen | Nov 2011 | A1 |
| 20110280125 | Jayakumar | Nov 2011 | A1 |
| 20110320724 | Mejdrich et al. | Dec 2011 | A1 |
| 20120093505 | Yeap et al. | Apr 2012 | A1 |
| 20120102506 | Hopmann et al. | Apr 2012 | A1 |
| 20120117423 | Andrade et al. | May 2012 | A1 |
| 20120137075 | Vorbach | May 2012 | A1 |
| 20120144064 | Parker et al. | Jun 2012 | A1 |
| 20120144065 | Parker et al. | Jun 2012 | A1 |
| 20120147752 | Ashwood-Smith et al. | Jun 2012 | A1 |
| 20120170462 | Sinha | Jul 2012 | A1 |
| 20120170575 | Mehra | Jul 2012 | A1 |
| 20120213118 | Lindsay et al. | Aug 2012 | A1 |
| 20120250512 | Jagadeeswaran et al. | Oct 2012 | A1 |
| 20120287821 | Godfrey et al. | Nov 2012 | A1 |
| 20120297083 | Ferguson et al. | Nov 2012 | A1 |
| 20120300669 | Zahavi | Nov 2012 | A1 |
| 20120314707 | Epps et al. | Dec 2012 | A1 |
| 20130010636 | Regula | Jan 2013 | A1 |
| 20130039169 | Schlansker et al. | Feb 2013 | A1 |
| 20130060944 | Archer et al. | Mar 2013 | A1 |
| 20130103777 | Kagan et al. | Apr 2013 | A1 |
| 20130121178 | Mainaud et al. | May 2013 | A1 |
| 20130136090 | Liu et al. | May 2013 | A1 |
| 20130182704 | Jacobs et al. | Jul 2013 | A1 |
| 20130194927 | Yamaguchi et al. | Aug 2013 | A1 |
| 20130203422 | Masputra et al. | Aug 2013 | A1 |
| 20130205002 | Wang | Aug 2013 | A1 |
| 20130208593 | Nandagopal | Aug 2013 | A1 |
| 20130246552 | Underwood et al. | Sep 2013 | A1 |
| 20130290673 | Archer et al. | Oct 2013 | A1 |
| 20130301645 | Bogdanski et al. | Nov 2013 | A1 |
| 20130304988 | Totolos et al. | Nov 2013 | A1 |
| 20130311525 | Neerincx et al. | Nov 2013 | A1 |
| 20130329577 | Suzuki et al. | Dec 2013 | A1 |
| 20130336164 | Yang et al. | Dec 2013 | A1 |
| 20140019661 | Hormuth et al. | Jan 2014 | A1 |
| 20140032695 | Michels et al. | Jan 2014 | A1 |
| 20140036680 | Lih et al. | Feb 2014 | A1 |
| 20140064082 | Yeung et al. | Mar 2014 | A1 |
| 20140095753 | Crupnicoff et al. | Apr 2014 | A1 |
| 20140098675 | Frost et al. | Apr 2014 | A1 |
| 20140119367 | Han et al. | May 2014 | A1 |
| 20140122560 | Ramey et al. | May 2014 | A1 |
| 20140129664 | McDaniel et al. | May 2014 | A1 |
| 20140133292 | Yamatsu et al. | May 2014 | A1 |
| 20140136646 | Tamir et al. | May 2014 | A1 |
| 20140169173 | Naouri et al. | Jun 2014 | A1 |
| 20140185621 | Decusatis et al. | Jul 2014 | A1 |
| 20140189174 | Ajanovic et al. | Jul 2014 | A1 |
| 20140207881 | Nussle et al. | Jul 2014 | A1 |
| 20140211804 | Makikeni et al. | Jul 2014 | A1 |
| 20140226488 | Shamis et al. | Aug 2014 | A1 |
| 20140241164 | Cociglio et al. | Aug 2014 | A1 |
| 20140258438 | Ayoub | Sep 2014 | A1 |
| 20140301390 | Scott et al. | Oct 2014 | A1 |
| 20140307554 | Basso | Oct 2014 | A1 |
| 20140325013 | Tamir et al. | Oct 2014 | A1 |
| 20140328172 | Kumar et al. | Nov 2014 | A1 |
| 20140347997 | Bergamasco et al. | Nov 2014 | A1 |
| 20140362698 | Arad | Dec 2014 | A1 |
| 20140369360 | Carlstrom | Dec 2014 | A1 |
| 20140379847 | Williams | Dec 2014 | A1 |
| 20150003247 | Mejia et al. | Jan 2015 | A1 |
| 20150006849 | Xu et al. | Jan 2015 | A1 |
| 20150009823 | Ganga et al. | Jan 2015 | A1 |
| 20150026361 | Matthews et al. | Jan 2015 | A1 |
| 20150029848 | Jain | Jan 2015 | A1 |
| 20150055476 | Decusatis et al. | Feb 2015 | A1 |
| 20150055661 | Boucher et al. | Feb 2015 | A1 |
| 20150067095 | Gopal et al. | Mar 2015 | A1 |
| 20150089495 | Persson et al. | Mar 2015 | A1 |
| 20150103667 | Elias et al. | Apr 2015 | A1 |
| 20150124826 | Edsall et al. | May 2015 | A1 |
| 20150146527 | Kishore et al. | May 2015 | A1 |
| 20150154004 | Aggarwal | Jun 2015 | A1 |
| 20150161064 | Pope | Jun 2015 | A1 |
| 20150180782 | Rimmer et al. | Jun 2015 | A1 |
| 20150186318 | Kim et al. | Jul 2015 | A1 |
| 20150193262 | Archer et al. | Jul 2015 | A1 |
| 20150195388 | Snyder, II | Jul 2015 | A1 |
| 20150208145 | Parker et al. | Jul 2015 | A1 |
| 20150220449 | Stark et al. | Aug 2015 | A1 |
| 20150237180 | Swartzentruber et al. | Aug 2015 | A1 |
| 20150244617 | Nakil et al. | Aug 2015 | A1 |
| 20150244804 | Warfield et al. | Aug 2015 | A1 |
| 20150261434 | Kagan et al. | Sep 2015 | A1 |
| 20150263955 | Talaski et al. | Sep 2015 | A1 |
| 20150263994 | Haramaty et al. | Sep 2015 | A1 |
| 20150288626 | Aybay | Oct 2015 | A1 |
| 20150365337 | Pannell | Dec 2015 | A1 |
| 20150370586 | Cooper et al. | Dec 2015 | A1 |
| 20160006664 | Sabato et al. | Jan 2016 | A1 |
| 20160012002 | Arimilli et al. | Jan 2016 | A1 |
| 20160028613 | Haramaty et al. | Jan 2016 | A1 |
| 20160065455 | Wang et al. | Mar 2016 | A1 |
| 20160094450 | Ghanwani et al. | Mar 2016 | A1 |
| 20160134518 | Callon et al. | May 2016 | A1 |
| 20160134535 | Callon | May 2016 | A1 |
| 20160134559 | Abel et al. | May 2016 | A1 |
| 20160134573 | Gagliardi et al. | May 2016 | A1 |
| 20160142318 | Beecroft | May 2016 | A1 |
| 20160154756 | Dodson et al. | Jun 2016 | A1 |
| 20160182383 | Pedersen | Jun 2016 | A1 |
| 20160205023 | Janardhanan | Jul 2016 | A1 |
| 20160226797 | Aravinthan et al. | Aug 2016 | A1 |
| 20160254991 | Eckert et al. | Sep 2016 | A1 |
| 20160259394 | Ragavan | Sep 2016 | A1 |
| 20160283422 | Crupnicoff et al. | Sep 2016 | A1 |
| 20160285545 | Schmidtke et al. | Sep 2016 | A1 |
| 20160285677 | Kashyap et al. | Sep 2016 | A1 |
| 20160294694 | Parker et al. | Oct 2016 | A1 |
| 20160294926 | Zur et al. | Oct 2016 | A1 |
| 20160301610 | Amit et al. | Oct 2016 | A1 |
| 20160344620 | G. Santos et al. | Nov 2016 | A1 |
| 20160381189 | Caulfield et al. | Dec 2016 | A1 |
| 20170024263 | Verplanken | Jan 2017 | A1 |
| 20170039063 | Gopal et al. | Feb 2017 | A1 |
| 20170041239 | Goldenberg et al. | Feb 2017 | A1 |
| 20170048144 | Liu | Feb 2017 | A1 |
| 20170054633 | Underwood et al. | Feb 2017 | A1 |
| 20170091108 | Arellano et al. | Mar 2017 | A1 |
| 20170097840 | Bridgers | Apr 2017 | A1 |
| 20170103108 | Datta et al. | Apr 2017 | A1 |
| 20170118090 | Pettit et al. | Apr 2017 | A1 |
| 20170118098 | Littlejohn et al. | Apr 2017 | A1 |
| 20170153852 | Ma et al. | Jun 2017 | A1 |
| 20170177541 | Berman et al. | Jun 2017 | A1 |
| 20170220500 | Tong | Aug 2017 | A1 |
| 20170237654 | Turner et al. | Aug 2017 | A1 |
| 20170237671 | Rimmer et al. | Aug 2017 | A1 |
| 20170242753 | Sherlock et al. | Aug 2017 | A1 |
| 20170250914 | Caulfield et al. | Aug 2017 | A1 |
| 20170251394 | Johansson et al. | Aug 2017 | A1 |
| 20170270051 | Chen et al. | Sep 2017 | A1 |
| 20170272331 | Lissack | Sep 2017 | A1 |
| 20170272370 | Ganga et al. | Sep 2017 | A1 |
| 20170286316 | Doshi et al. | Oct 2017 | A1 |
| 20170289066 | Haramaty et al. | Oct 2017 | A1 |
| 20170295098 | Watkins et al. | Oct 2017 | A1 |
| 20170324664 | Xu et al. | Nov 2017 | A1 |
| 20170371778 | McKelvie et al. | Dec 2017 | A1 |
| 20180004705 | Menachem et al. | Jan 2018 | A1 |
| 20180019948 | Patwardhan et al. | Jan 2018 | A1 |
| 20180026878 | Zahavi et al. | Jan 2018 | A1 |
| 20180077064 | Wang | Mar 2018 | A1 |
| 20180083868 | Cheng | Mar 2018 | A1 |
| 20180097645 | Rajagopalan et al. | Apr 2018 | A1 |
| 20180097912 | Chumbalkar et al. | Apr 2018 | A1 |
| 20180113618 | Chan et al. | Apr 2018 | A1 |
| 20180115469 | Erickson et al. | Apr 2018 | A1 |
| 20180131602 | Civanlar et al. | May 2018 | A1 |
| 20180131678 | Agarwal et al. | May 2018 | A1 |
| 20180150374 | Ratcliff | May 2018 | A1 |
| 20180152317 | Chang et al. | May 2018 | A1 |
| 20180152357 | Natham et al. | May 2018 | A1 |
| 20180173557 | Nakil et al. | Jun 2018 | A1 |
| 20180183724 | Callard et al. | Jun 2018 | A1 |
| 20180191609 | Caulfield et al. | Jul 2018 | A1 |
| 20180198736 | Labonte et al. | Jul 2018 | A1 |
| 20180212876 | Bacthu et al. | Jul 2018 | A1 |
| 20180212902 | Steinmacher-Burow | Jul 2018 | A1 |
| 20180219804 | Graham et al. | Aug 2018 | A1 |
| 20180225238 | Karguth et al. | Aug 2018 | A1 |
| 20180234343 | Zdornov et al. | Aug 2018 | A1 |
| 20180254945 | Bogdanski et al. | Sep 2018 | A1 |
| 20180260324 | Marathe et al. | Sep 2018 | A1 |
| 20180278540 | Shalev et al. | Sep 2018 | A1 |
| 20180287928 | Levi et al. | Oct 2018 | A1 |
| 20180323898 | Dods | Nov 2018 | A1 |
| 20180335974 | Simionescu et al. | Nov 2018 | A1 |
| 20180341494 | Sood et al. | Nov 2018 | A1 |
| 20190007349 | Wang et al. | Jan 2019 | A1 |
| 20190018808 | Beard et al. | Jan 2019 | A1 |
| 20190036771 | Sharpless et al. | Jan 2019 | A1 |
| 20190042337 | Dinan et al. | Feb 2019 | A1 |
| 20190042518 | Marolia | Feb 2019 | A1 |
| 20190044809 | Willis et al. | Feb 2019 | A1 |
| 20190044827 | Ganapathi et al. | Feb 2019 | A1 |
| 20190044863 | Mula et al. | Feb 2019 | A1 |
| 20190044872 | Ganapathi et al. | Feb 2019 | A1 |
| 20190044875 | Murty et al. | Feb 2019 | A1 |
| 20190052327 | Motozuka et al. | Feb 2019 | A1 |
| 20190058663 | Song | Feb 2019 | A1 |
| 20190068501 | Schneider et al. | Feb 2019 | A1 |
| 20190081903 | Kobayashi et al. | Mar 2019 | A1 |
| 20190095134 | Li | Mar 2019 | A1 |
| 20190104057 | Goel et al. | Apr 2019 | A1 |
| 20190104206 | Goel et al. | Apr 2019 | A1 |
| 20190108106 | Aggarwal et al. | Apr 2019 | A1 |
| 20190108332 | Glew et al. | Apr 2019 | A1 |
| 20190109791 | Mehra et al. | Apr 2019 | A1 |
| 20190121781 | Kasichainula | Apr 2019 | A1 |
| 20190140979 | Levi et al. | May 2019 | A1 |
| 20190146477 | Cella et al. | May 2019 | A1 |
| 20190171612 | Shahar et al. | Jun 2019 | A1 |
| 20190196982 | Rozas et al. | Jun 2019 | A1 |
| 20190199646 | Singh | Jun 2019 | A1 |
| 20190253354 | Caulfield et al. | Aug 2019 | A1 |
| 20190280978 | Schmatz et al. | Sep 2019 | A1 |
| 20190294575 | Dennison et al. | Sep 2019 | A1 |
| 20190306134 | Shanbhogue et al. | Oct 2019 | A1 |
| 20190332314 | Zhang et al. | Oct 2019 | A1 |
| 20190334624 | Bernard | Oct 2019 | A1 |
| 20190356611 | Das et al. | Nov 2019 | A1 |
| 20190361728 | Kumar et al. | Nov 2019 | A1 |
| 20190379610 | Srinivasan et al. | Dec 2019 | A1 |
| 20200036644 | Belogolovy et al. | Jan 2020 | A1 |
| 20200084150 | Burstein et al. | Mar 2020 | A1 |
| 20200145725 | Eberle et al. | May 2020 | A1 |
| 20200177505 | Li | Jun 2020 | A1 |
| 20200177521 | Blumrich et al. | Jun 2020 | A1 |
| 20200259755 | Wang et al. | Aug 2020 | A1 |
| 20200272579 | Humphrey et al. | Aug 2020 | A1 |
| 20200274832 | Humphrey et al. | Aug 2020 | A1 |
| 20200334195 | Chen et al. | Oct 2020 | A1 |
| 20200349098 | Caulfield et al. | Nov 2020 | A1 |
| 20210081410 | Chavan et al. | Mar 2021 | A1 |
| 20210152494 | Johnsen et al. | May 2021 | A1 |
| 20210263779 | Haghighat et al. | Aug 2021 | A1 |
| 20210334206 | Colgrove | Oct 2021 | A1 |
| 20210377156 | Michael | Dec 2021 | A1 |
| 20210409351 | Das et al. | Dec 2021 | A1 |
| 20220131768 | Ganapathi et al. | Apr 2022 | A1 |
| 20220166705 | Froese | May 2022 | A1 |
| 20220200900 | Roweth | Jun 2022 | A1 |
| 20220210058 | Bataineh et al. | Jun 2022 | A1 |
| 20220217078 | Ford et al. | Jul 2022 | A1 |
| 20220217101 | Yefet et al. | Jul 2022 | A1 |
| 20220245072 | Roweth et al. | Aug 2022 | A1 |
| 20220278941 | Shalev et al. | Sep 2022 | A1 |
| 20220309025 | Chen et al. | Sep 2022 | A1 |
| 20230035420 | Sankaran et al. | Feb 2023 | A1 |
| 20230046221 | Pismenny et al. | Feb 2023 | A1 |
| Number | Date | Country |
|---|---|---|
| 101729609 | Jun 2010 | CN |
| 102932203 | Feb 2013 | CN |
| 110324249 | Oct 2019 | CN |
| 0275135 | Jul 1988 | EP |
| 2187576 | May 2010 | EP |
| 2219329 | Aug 2010 | EP |
| 2947832 | Nov 2015 | EP |
| 3445006 | Feb 2019 | EP |
| 110601888 | Dec 2019 | IN |
| 2003-244196 | Aug 2003 | JP |
| 3459653 | Oct 2003 | JP |
| 10-2012-0062864 | Jun 2012 | KR |
| 10-2012-0082739 | Jul 2012 | KR |
| 10-2014-0100529 | Aug 2014 | KR |
| 10-2015-0026939 | Mar 2015 | KR |
| 10-2015-0104056 | Sep 2015 | KR |
| 10-2017-0110106 | Oct 2017 | KR |
| 10-1850749 | Apr 2018 | KR |
| 2001069851 | Sep 2001 | WO |
| 0247329 | Jun 2002 | WO |
| 2003019861 | Mar 2003 | WO |
| 2004001615 | Dec 2003 | WO |
| 2005094487 | Oct 2005 | WO |
| 2007034184 | Mar 2007 | WO |
| 2009010461 | Jan 2009 | WO |
| 2009018232 | Feb 2009 | WO |
| 2014092780 | Jun 2014 | WO |
| 2014137382 | Sep 2014 | WO |
| 2014141005 | Sep 2014 | WO |
| 2018004977 | Jan 2018 | WO |
| 2018046703 | Mar 2018 | WO |
| 2019072072 | Apr 2019 | WO |
| Entry |
|---|
| Awerbuch, B., et al.; “An On-Demand Secure Routing Protocol Resilient to Byzantine Failures”; Sep. 2002; 10 pages. |
| Belayneh L.W., et al.; “Method and Apparatus for Routing Data in an Inter-Nodal Communications Lattice of a Massively Parallel Computer System by Semi-Randomly Varying Routing Policies for Different Packets”; 2019; 3 pages. |
| Bhatele, A., et al.; “Analyzing Network Health and Congestion in Dragonfly-based Supercomputers”; May 23-27, 2016; 10 pages. |
| Blumrich, M.A., et al.; “Exploiting Idle Resources in a High-Radix Switch for Supplemental Storage”; Nov. 2018; 13 pages. |
| Chang, F., et al.; “PVW: Designing Vir PVW: Designing Virtual World Ser orld Server Infr er Infrastructur astructure”; 2010; 8 pages. |
| Chang, F., et al.; “PVW: Designing Virtual World Server Infrastructure”; 2010; 8 pages. |
| Chen, F., et al.; “Requirements for RoCEv3 Congestion Management”; Mar. 21, 2019; 8 pages. |
| Cisco Packet Tracer; “packet-tracer;—ping”; https://www.cisco.com/c/en/us/td/docs/security/asa/asa-command-reference/l-R/cmdref2/p1.html; 2017. |
| Cisco; “Understanding Rapid Spanning Tree Protocol (802.1w)”; Aug. 1, 2017; 13 pages. |
| Eardley, ED, P; “Pre-Congestion Notification (PCN) Architecture”; Jun. 2009; 54 pages. |
| Escudero-Sahuquillo, J., et al.; “Combining Congested-Flow Isolation and Injection Throttling in HPC Interconnection Networks”; Sep. 13-16, 2011; 3 pages. |
| Hong, Y.; “Mitigating the Cost, Performance, and Power Overheads Induced by Load Variations in Multicore Cloud Servers”; Fall 2013; 132 pages. |
| Huawei; “The Lossless Network For Data Centers”; Nov. 7, 2017; 15 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024248, dated Jul. 8, 2020, 11 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US20/024332, dated Jul. 8, 2020, 13 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US20/24243, dated Jul. 9, 2020, 10 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US20/24253, dated Jul. 6, 2020, 12 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US20/24256, dated Jul. 7, 2020, 11 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US20/24257, dated Jul. 7, 2020, 10 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US20/24258, dated Jul. 7, 2020, 9 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US20/24259, dated Jul. 9, 2020, 13 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US20/24260, dated Jul. 7, 2020, 11 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US20/24268, dated Jul. 9, 2020, 11 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US20/24269, dated Jul. 9, 2020, 11 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US20/24339, dated Jul. 8, 2020, 11 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024125, dated Jul. 10, 2020, 5 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024129, dated Jul. 10, 2020, 11 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024237, dated Jul. 14, 2020, 5 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024239, dated Jul. 14, 2020, 11 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024241, dated Jul. 14, 2020, 13 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024242, dated Jul. 6, 2020, 11 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024244, dated Jul. 13, 2020, 10 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024245, dated Jul. 14, 2020, 11 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024246, dated Jul. 14, 2020, 10 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024250, dated Jul. 14, 2020, 12 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024254, dated Jul. 13, 2020, 10 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024262, dated Jul. 13, 2020, 10 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024266, dated Jul. 9, 2020, 10 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024270, dated Jul. 10, 2020, 13 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024271, dated Jul. 9, 2020, 10 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024272, dated Jul. 9, 2020, 10 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024276, dated Jul. 13, 2020, 9 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024304, dated Jul. 15, 2020, 11 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024311, dated Jul. 17, 2020, 8 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024321, dated Jul. 9, 2020, 9 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024324, dated Jul. 14, 2020, 10 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/024327, dated Jul. 10, 2020, 15 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/24158, dated Jul. 6, 2020, 18 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/24251, dated Jul. 6, 2020, 11 pages. |
| International Search Report and Written Opinion received for PCT Application No. PCT/US2020/24267, dated Jul. 6, 2020, 9 pages. |
| International Search Report and Written Opinion received for PCT Patent Application No. PCT/US20/24303, dated Oct. 21, 2020, 9 pages. |
| International Search Report and Written Opinion received for PCT Patent Application No. PCT/US20/24340, dated Oct. 26, 2020, 9 pages. |
| International Search Report and Written Opinion received for PCT Patent Application No. PCT/US20/24342, dated Oct. 27, 2020, 10 pages. |
| International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2020/024192, dated Oct. 23, 2020, 9 pages. |
| International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2020/024221, dated Oct. 26, 2020, 9 pages. |
| International Search Report cited in PCT/US2020/024170 dated Dec. 16, 2020; 3 pages. |
| Maabi, S., et al.; “ERFAN: Efficient reconfigurable fault-tolerant deflection routing algorithm for 3-D Network-on-Chip”; Sep. 6-9, 2016. |
| Maglione-Mathey, G., et al.; “Scalable Deadlock-Free Deterministic Minimal-Path Routing Engine for InfiniBand-Based Dragonfly Networks”; Aug. 21, 2017; 15 pages. |
| Mamidala, A. R., et al.; “Efficient Barrier and Allreduce on Infiniband clusters using multicast and adaptive algorithms”; Sep. 20-23, 2004; 10 pages. |
| Mammeri, Z; “Reinforcement Learning Based Routing in Networks: Review and Classification of Approaches”; Apr. 29, 2019; 35 pages. |
| Mollah; M. A., et al.; “High Performance Computing Systems. Performance Modeling, Benchmarking, and Simulation: 8th International Workshop”; Nov. 13, 2017. |
| Open Networking Foundation; “OpenFlow Switch Specification”; Mar. 26, 2015; 283 pages. |
| Prakash, P., et al.; “The TCP Outcast Problem: Exposing Unfairness in Data Center Networks”; 2011; 15 pages. |
| Ramakrishnan, K., et al.; “The Addition of Explicit Congestion Notification (ECN) to IP”; Sep. 2001; 63 pages. |
| Roth, P. C., et al; “MRNet: A Software-Based Multicast/Reduction Network for Scalable Tools1”; Nov. 15-21, 2003; 16 pages. |
| Silveira, J., et al.; “Preprocessing of Scenarios for Fast and Efficient Routing Reconfiguration in Fault-Tolerant NoCs”; Mar. 4-6, 2015. |
| Tsunekawa, K.; “Fair bandwidth allocation among LSPs for AF class accommodating TCP and UDP traffic in a Diffserv-capable MPLS network”; Nov. 17, 2005; 9 pages. |
| Underwood, K.D., et al.; “A hardware acceleration unit for MPI queue processing”; Apr. 18, 2005; 10 pages. |
| Wu, J.; “Fault-tolerant adaptive and minimal routing in mesh-connected multicomputers using extended safety levels”; Feb. 2000; 11 pages. |
| Xiang, D., et al.; “Fault-Tolerant Adaptive Routing in Dragonfly Networks”; Apr. 12, 2017; 15 pages. |
| Xiang, D., et al; “Deadlock-Free Broadcast Routing in Dragonfly Networks without Virtual Channels”, submission to IEEE transactions on Parallel and Distributed Systems, 2015, 15 pages. |
| Extended European Search Report and Search Opinion received for EP Application No. 20809930.9, dated Mar. 2, 2023, 9 pages. |
| Extended European Search Report and Search Opinion received for EP Application No. 20810784.7, dated Mar. 9, 2023, 7 pages. |
| Ramakrishnan et al, RFC 3168, “The addition of Explicit Congestion Notification (ECN) to IP”, Sep. 2001 (Year: 2001). |
| Number | Date | Country | |
|---|---|---|---|
| 20220232111 A1 | Jul 2022 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62852289 | May 2019 | US | |
| 62852273 | May 2019 | US | |
| 62852203 | May 2019 | US |
