Patent Grant
9973435
The present invention relates generally to communication networks, and particularly to methods and systems for adaptive routing of packets.
Various techniques for adaptive routing of packets in communication networks are known in the art. Some adaptive routing techniques are used in Dragonfly-topology networks. The Dragonfly topology is described, for example, by Kim et al., in “Technology-Driven, Highly-Scalable Dragonfly Topology,” Proceedings of the 2008 International Symposium on Computer Architecture, Jun. 21-25, 2008, pages 77-88, which is incorporated herein by reference. U.S. Patent Application Publication 2010/0049942 to Kim et al., whose disclosure is incorporated herein by reference, describes a Dragonfly processor interconnect network that comprises a plurality of processor nodes, a plurality of routers, each router directly coupled to a plurality of terminal nodes, the routers coupled to one another and arranged into a group, and a plurality of groups of routers, such that each group is connected to each other group via at least one direct connection.
Jiang et al. describe indirect global adaptive routing (IAR) schemes in Dragonfly networks, in which the adaptive routing decision uses information that is not directly available at the source router, in “Indirect Adaptive Routing on Large Scale Interconnection Networks,” Proceedings of the 2009 International Symposium on Computer Architecture, Jun. 20-24, 2009, pages 220-231, which is incorporated herein by reference.
Garcia et al. describe a routing/flow-control scheme for Dragonfly networks, which decouples the routing and the deadlock avoidance mechanisms, in “On-the-Fly Adaptive Routing in High-Radix Hierarchical Networks,” Proceedings of the 2012 International Conference on Parallel Processing (ICPP), Sep. 10-13, 2012, which is incorporated herein by reference.
Prisacari et al. investigate indirect routing over Dragonfly networks, in in “Performance implications of remote-only load balancing under adversarial traffic in Dragonflies,” Proceedings of the 8th International Workshop on Interconnection Network Architecture: On-Chip, Multi-Chip, Jan. 22, 2014, which is incorporated herein by reference.
U.S. Patent Application Publication 2012/0144064, whose disclosure is incorporated herein by reference, describes a dragonfly processor interconnect network that comprises a plurality of processor nodes and a plurality of routers. The routers are operable to adaptively route data by selecting from among a plurality of network paths from a target node to a destination node in the dragonfly network, based on network congestion information from neighboring routers and failed network link information from neighboring routers.
As another example, U.S. Patent Application Publication 2012/0144065, whose disclosure is incorporated herein by reference describes a dragonfly processor interconnect network that comprises a plurality of processor nodes and a plurality of routers. The routers are operable to route data by selecting from among a plurality of network paths from a target node to a destination node in the dragonfly network based on one or more routing tables.
An embodiment that is described herein provides a method for communication including, in a network node that includes a plurality of ports, specifying for a given destination address at least one Adaptive Routing (AR) group. The at least one AR group includes two or more ports over which packets destined to the given destination address are to be adaptively routed. A packet destined to the given destination address is received at the network node, via one of the ports serving as an ingress port. An egress port is adaptively selected for the packet, from the ports in the at least one AR group but excluding the ingress port over which the packet was received. The packet is routed to the selected egress port.
In some embodiments, specifying the at least one AR group includes specifying for the given destination address multiple AR groups, such that (i) each AR group is specified for routing the packets arriving via one or more specified ingress ports and (ii) each AR group includes only egress ports that are different from the ingress ports for which that AR group is specified. In an embodiment, a given egress port is common to two or more of the AR groups.
In another embodiment, adaptively selecting the egress port includes identifying, from among the multiple AR groups, an AR group specified for the ingress port via which the packet was received, and adaptively selecting the egress port from the identified AR group. In yet another embodiment, adaptively selecting the egress port includes identifying, from among the multiple AR groups, an AR group specified for (i) the ingress port via which the packet was received and (ii) a Virtual Lane (VL) of the packet, and adaptively selecting the egress port from the identified AR group.
In alternative embodiments, specifying the at least one AR group includes adaptively specifying within an AR group a primary egress port and a secondary egress port, and adaptively selecting the egress port includes: selecting the primary egress port if the ingress port is different from the primary egress port; and selecting the secondary egress port if the ingress port is equal to the primary egress port.
In an example embodiment, selecting the secondary egress port includes retrieving an identity of the secondary egress port from a memory of the network node. In another embodiment, selecting the secondary egress port includes applying a hash function that selects the secondary egress port from among the ports in the AR group other than the primary egress port. In yet another embodiment, selecting the secondary egress port includes selecting a port specified for static routing.
There is additionally provided, in accordance with an embodiment of the present invention, a network node including a plurality of ports and circuitry. The circuitry, is configured to specify for a given destination address at least one Adaptive Routing (AR) group, which includes two or more ports over which packets destined to the given destination address are to be adaptively routed, to receive, via one of the ports serving as an ingress port, a packet destined to the given destination address, to adaptively select an egress port for the packet, from the ports in the at least one AR group but excluding the ingress port over which the packet was received, and to route the packet to the selected egress port.
The present invention will be more fully understood from the following detailed description of the embodiments thereof, taken together with the drawings in which:
In some network configurations it is important to prevent loopback paths. In a network switch or other network node, for example, preventing loopback means that a packet should not be forwarded over the same port via which it was received. When the network node uses Adaptive Routing (AR), however, loopback prevention may conflict with AR operation and cause performance degradation. Embodiments of the present invention that are described herein provide improved methods and systems for Adaptive Routing (AR). The disclosed techniques are loopback free, and at the same time cause minimal degradation in AR performance.
In some embodiments, a network node comprises multiple ports, which can serve as ingress ports for receiving packets and/or as egress ports for sending packets. In some embodiments, the node applies adaptive routing per destination address. The node specifies, per destination address, a group of ports from which to adaptively select an egress port for packets destined to that destination address. This group of ports is referred to herein as an AR group.
Typically, the node aims to re-select the egress port as rarely as possible, e.g., in order to reduce the likelihood that packets will reach the destination out-of-order. The node will typically retain the same egress port selection for a given destination address, unless performance over this port becomes unacceptable.
From time to time, however, packets destined to the given destination address may arrive over the same port that currently serves as the selected egress port. Forwarding such a packet over the currently-selected egress port (thereby preventing packet re-ordering) may create loopback. Forwarding such a packet over a different egress port (thereby preventing loopback) may create packet re-ordering. In some embodiments that are described herein, the node uses improved AR schemes that avoid the above-described dilemma.
In one embodiment, the node specifies multiple AR groups for a given destination address. Each AR group is specified for routing the packets arriving via one or more specified ingress ports. In addition, each AR group comprises only egress ports that are different from the ingress ports for which that AR group is specified. For example, for the case of four ports denoted A, B, C and D, the node may specify the AR group {B,C,D} for packets arriving over port A, the AR group {A,C,D} for packets arriving over port B, the AR group {A,B,D} for packets arriving over port C, and the AR group {A,B,C} for packets arriving over port D.
When a packet destined to the given destination address arrives over a certain ingress port, the node identifies the AR group that is specified for that ingress port, and routes the packet to one of the egress ports in this AR group. Within each AR group, the node can retain the egress-port selection for long periods of time. At the same time, because of the way the AR groups are specified, the selected egress port is guaranteed to be different from the ingress port.
In another embodiment, the node specifies a single AR group per destination address. Within the AR group, in addition to the currently-selected egress port (which is referred to in this embodiment as a primary egress port), the node also specifies a secondary egress port. When a packet destined to the given destination address arrives over a certain ingress port, the node checks whether the ingress port is different from the currently-selected egress port (the primary egress port). If so, the node forwards the packet over the primary egress port. Otherwise, the node forwards the packet over the secondary egress port. The latter technique has a slightly higher likelihood of packet re-ordering relative to the former technique. On the other hand, the latter technique requires considerably less memory space for storing the AR groups.
Several example implementations of the disclosed techniques are described in detail below. The disclosed AR schemes are demonstrated below in a network topology referred to as “Dragonfly plus,” but they are applicable in various other network topologies.
Network 20 comprises multiple network nodes 24, also referred to herein simply as nodes for brevity. Nodes 24 typically comprise packet switches or routers. Nodes 24 are arranged in multiple groups 28. Groups 28 are connected to one another using network links 32, e.g., optical fibers, each connected between a port of a node in one group and a port in a node of another group. Links 32 are referred to herein as inter-group links or global links.
The set of links 32 is referred to herein collectively as an inter-group network or global network. In the disclosed embodiments, the inter-group network has a mesh topology, i.e., every group 28 is connected to every other group 28 using at least one direct inter-group link 32. Put in another way, any pair of groups 28 comprise at least one respective pair of nodes 24 (one node in each group) that are connected to one another using a direct inter-group link 32.
The nodes within each group 28 are interconnected by network links 36, e.g., optical fibers. Each link 36 is connected between respective ports of two nodes within a given group 28. Links 36 are referred to herein as intra-group links or local links, and the set of links 36 in a given group 28 is referred to herein collectively as an intra-group network or local network. Within each group 28, nodes 24 are connected in a bipartite graph topology. In such a topology, the nodes are divided into two subsets, such that all links 36 connect a node of one subset and a node of the other subset. In other words, no direct links connect between nodes of the same subset.
In an example embodiment, the nodes in each group are connected in a Fat Tree (FT) topology. Alternatively, however, the intra-group network may be implemented using any other suitable bipartite topology. The bipartite topology may be full (i.e., every node in one subset is directly connected to every node in the other subset) or partial. The two subsets may be of the same size or of different sizes. In a partial bipartite topology, not every node of one subset is connected to every node of the other subset. A bipartite topology may be partial by design, e.g., in order to save cost, or as a result of link failure. In some bipartite topologies, a certain pair of nodes may be connected by two or more local links 36 in parallel. In the context of the present patent application and in the claims, all such variations are regarded as bipartite topologies.
Within a given group 28, one subset of nodes 24 (referred to as spine nodes) connect the group to the mesh network using links 32. The other subset of nodes 24 (referred to as leaf nodes) are connected to hosts 38, also referred to as endpoints or clients. Hosts 38 may comprise any suitable type of computing devices, such as servers. In the example of
Additional aspects of routing packets over network topologies such as the topology of network 20 are addressed in U.S. patent application Ser. No. 14/337,334, entitled “Dragonfly Plus: Communication over Bipartite Node Groups Connected by a Mesh Network,” filed Jul. 22, 2014, which is assigned to the assignee of the present patent application and whose disclosure is incorporated herein by reference.
An inset at the bottom-left of the figure shows a simplified view of the internal configuration of node 24, in an example embodiment. The other nodes typically have a similar structure. In this example, node 24 comprises multiple ports 40 for connecting to links 32 and/or 36 and/or endpoints 38, a switch fabric 44 that is configured to forward packets between ports 40, and a processor 48 that carries out the methods described herein. In the context of the present patent application and in the claims, fabric 44 and processor 48 are referred to collectively as circuitry that carries out the disclosed techniques.
In the embodiments described herein, network 20 operates in accordance with the InfiniBand™ standard. Infiniband communication is specified, for example, in “InfiniBand™ Architecture Specification,” Volume 1, Release 1.2.1, November, 2007, which is incorporated herein by reference. In particular, section 7.6 of this specification addresses Virtual Lanes (VL) mechanisms, section 7.9 addresses flow control, and chapter 14 addresses subnet management (SM) issues. In alternative embodiments, however, network 20 may operate in accordance with any other suitable communication protocol or standard, such as IPv4, IPv6 (which both support ECMP) and “controlled Ethernet.”
In some embodiments, network 20 is associated with a certain Infiniband subnet, and is managed by a processor referred to as a subnet manager (SM). The SM tasks may be carried out, for example, by software running on one or more of processors 48 of nodes 24, and/or on a separate processor. Typically, the SM configures switch fabrics 44 and/or processors 48 in the various nodes 24 to carry out the methods described herein.
The configurations of network 20 and node 24 shown in
The different elements of nodes 24 may be implemented using any suitable hardware, such as in an Application-Specific Integrated Circuit (ASIC) or Field-Programmable Gate Array (FPGA). In some embodiments, some elements of nodes 24 can be implemented using software, or using a combination of hardware and software elements. In some embodiments, processors 48 comprise general-purpose processors, which are programmed in software to carry out the functions described herein. The software may be downloaded to the processors in electronic form, over a network, for example, or it may, alternatively or additionally, be provided and/or stored on non-transitory tangible media, such as magnetic, optical, or electronic memory.
As can be seen in
Path 50A is the shortest path, traversing a local link, then a global link, and finally another local link. Such a path is most likely to have the smallest latency, especially since it the traffic is routed directly from group 28A to group 28D without traversing any intermediate group. Path 50B traverses a local link, then two global links (via group 28E—serving in this case as an intermediate group), and finally another local link. In this path, in intermediate group 28E the traffic traverses only one of the spine nodes without reaching the leaf nodes. Path 50C traverses a local link, then a global link to intermediate group 28E, then two local links inside group 28E, and finally another local link. In this path, in intermediate group 28E the traffic traverses two spine nodes and one leaf node.
In some embodiments, one or more network nodes 24 in network 20 route packets using routing schemes that implement Adaptive Routing (AR), and at the same time do not cause loopback paths. The description that follows focuses on a specific leaf node marked X, having four ports 54A . . . 54D, shown at the bottom of
In the present example, node X is connected to four spine nodes Y1, Y2, Y3 and Y4, via ports 54A, 54B, 54C and 54D, respectively. Consider, for example, packets that originate in source endpoint S1 in group 28A and are destined to destination endpoint D in group 28D. As explained above, such packets may traverse various possible paths in network 20. In particular, packets whose destination address is endpoint D may arrive at node X via any of nodes Y1 . . . Y4, i.e., via any of ports 54A . . . 54D.
In some embodiments, node X routes these packets using AR methods that are guaranteed not to create loopback, i.e., not to forward a packet over the same port via which the packet was received. The description that follows refers to an Infiniband-based implementation in which destination addresses are referred to as Destination Local Identifier (DLID). Generally, however, the disclosed techniques can be implemented in other suitable network types and with other suitable types of destination addresses.
For a given DLID, each AR group is specified for routing the packets arriving via one or more specified ingress ports. In addition, each AR group comprises only egress ports that are different from the ingress ports for which that AR group is specified. Two non-limiting examples of such AR groups are depicted in
At a packet reception step 64, the circuitry of node X receives a packet destined to the DLID of endpoint D. The packet is received via one of ports 54A . . . 54D that, for that packet, serves as ingress port. At a routing step 68, the circuitry of node X identifies the AR group that is specified for the ingress port via which the packet arrived, and then routes the packet to one of the egress ports in this AR group. The method then loops back to step 64 above for processing the next packet.
As can be seen in
The AR groups of
Within the AR group, the circuitry specifies a primary egress port and a secondary egress port for AR, for the DLID of endpoint D. The primary egress port is the port that was most recently chosen by AR for routing packets to destination endpoint D. The secondary egress port is reverted to when necessary to avoid loopback, i.e., for routing a packet that arrived via the primary egress port.
At a packet reception step 74, the circuitry of node X receives a packet destined to the DLID of endpoint D. The packet is received via one of ports 54A . . . 54D that, for that packet, serves as ingress port. At a checking step 78, the circuitry checks whether the ingress port is equal to the primary egress port. If not, the circuitry forwards the packet over the primary egress port, at a primary routing step 82. If the ingress port is equal to the primary egress port, the circuitry forwards the packet over the secondary egress port to avoid loopback, at a secondary routing step 86.
At a change checking step 90, the circuitry checks whether it is necessary to change the current AR decision, i.e., to re-specify the primary egress port. Such a change may be triggered, for example, by congestion on the primary egress port, by a failure of the primary egress port, or by some congestion or failure along the path from the primary egress port to the destination endpoint.
If no change is needed, the method loops back to step 74 above for processing the next packet. Otherwise, the circuitry re-selects the primary egress port, and possibly also the secondary egress port, at a AR decision change step 94. The method then loops back to step 74 above. In some embodiments, both the primary egress port and the secondary egress port are selected using AR. the tendency is typically to change the assignment of a primary or secondary egress port only when absolutely necessary, e.g., when the congestion on the currently-assigned egress port is unacceptable.
When implementing the method of
In an alternative embodiment, the circuitry holds in memory only the identity of the primary egress port, and not of the secondary egress port. When routing reverts to the secondary egress port, the circuitry calculates the identity of the secondary egress port by applying a hash function that maps some packet attributes (e.g., the DLID) to one of the egress ports in the AR group other than the primary egress port.
In yet another embodiment, the circuitry may already hold a definition of a port that is to be used for static routing, e.g., for packets that are not permitted to undergo AR. In an embodiment, the circuitry defines this static port as the secondary egress port. This implementation, too, is efficient in terms of memory space. Further alternatively, the circuitry may select the secondary egress port, and/or store the selection of the secondary egress port, in any other way.
The loopback-free AR schemes of
It will thus be appreciated that the embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and sub-combinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art. Documents incorporated by reference in the present patent application are to be considered an integral part of the application except that to the extent any terms are defined in these incorporated documents in a manner that conflicts with the definitions made explicitly or implicitly in the present specification, only the definitions in the present specification should be considered.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4312064 | Bench et al. | Jan 1982 | A |
| 6115385 | Vig | Sep 2000 | A |
| 6169741 | LeMaire et al. | Jan 2001 | B1 |
| 6480500 | Erimli et al. | Nov 2002 | B1 |
| 6532211 | Rathonyi et al. | Mar 2003 | B1 |
| 6553028 | Tang et al. | Apr 2003 | B1 |
| 6614758 | Wong | Sep 2003 | B2 |
| 6665297 | Harigochi et al. | Dec 2003 | B1 |
| 6775268 | Wang et al. | Aug 2004 | B1 |
| 6795886 | Nguyen | Sep 2004 | B1 |
| 6804532 | Moon et al. | Oct 2004 | B1 |
| 6831918 | Kavak | Dec 2004 | B1 |
| 6912604 | Tzeng et al. | Jun 2005 | B1 |
| 6950428 | Horst et al. | Sep 2005 | B1 |
| 7010607 | Bunton | Mar 2006 | B1 |
| 7076569 | Bailey et al. | Jul 2006 | B1 |
| 7234001 | Simpson et al. | Jun 2007 | B2 |
| 7286535 | Ishikawa et al. | Oct 2007 | B2 |
| 7676597 | Kagan et al. | Mar 2010 | B2 |
| 7746854 | Ambe et al. | Jun 2010 | B2 |
| 7936770 | Frattura et al. | May 2011 | B1 |
| 7969980 | Florit et al. | Jun 2011 | B1 |
| 8094569 | Gunukula et al. | Jan 2012 | B2 |
| 8175094 | Bauchot et al. | May 2012 | B2 |
| 8195989 | Lu et al. | Jun 2012 | B1 |
| 8213315 | Crupnicoff et al. | Jul 2012 | B2 |
| 8401012 | Underwood et al. | Mar 2013 | B2 |
| 8489718 | Brar et al. | Jul 2013 | B1 |
| 8495194 | Brar et al. | Jul 2013 | B1 |
| 8576715 | Bloch et al. | Nov 2013 | B2 |
| 8605575 | Gunukula et al. | Dec 2013 | B2 |
| 8621111 | Marr et al. | Dec 2013 | B2 |
| 8625427 | Terry et al. | Jan 2014 | B1 |
| 8755389 | Poutievski et al. | Jun 2014 | B1 |
| 8774063 | Beecroft | Jul 2014 | B2 |
| 8873567 | Mandal | Oct 2014 | B1 |
| 8908704 | Koren et al. | Dec 2014 | B2 |
| 9014006 | Haramaty et al. | Apr 2015 | B2 |
| 9042234 | Liljenstolpe et al. | May 2015 | B1 |
| 9571400 | Mandal | Feb 2017 | B1 |
| 20020013844 | Garrett et al. | Jan 2002 | A1 |
| 20020026525 | Armitage | Feb 2002 | A1 |
| 20020039357 | Lipasti et al. | Apr 2002 | A1 |
| 20020071439 | Reeves | Jun 2002 | A1 |
| 20020136163 | Kawakami et al. | Sep 2002 | A1 |
| 20020138645 | Shinomiya et al. | Sep 2002 | A1 |
| 20020141412 | Wong | Oct 2002 | A1 |
| 20020165897 | Kagan et al. | Nov 2002 | A1 |
| 20020176363 | Durinovic-Johri et al. | Nov 2002 | A1 |
| 20030016624 | Bare | Jan 2003 | A1 |
| 20030039260 | Fujisawa | Feb 2003 | A1 |
| 20030065856 | Kagan et al. | Apr 2003 | A1 |
| 20030079005 | Myers et al. | Apr 2003 | A1 |
| 20030223453 | Stoler et al. | Dec 2003 | A1 |
| 20040111651 | Mukherjee et al. | Jun 2004 | A1 |
| 20040202473 | Nakamura et al. | Oct 2004 | A1 |
| 20050013245 | Sreemanthula et al. | Jan 2005 | A1 |
| 20050157641 | Roy | Jul 2005 | A1 |
| 20050259588 | Preguica | Nov 2005 | A1 |
| 20060126627 | Diouf | Jun 2006 | A1 |
| 20060143300 | See | Jun 2006 | A1 |
| 20060182034 | Klinker et al. | Aug 2006 | A1 |
| 20060291480 | Cho et al. | Dec 2006 | A1 |
| 20070058536 | Vaananen et al. | Mar 2007 | A1 |
| 20070058646 | Hermoni | Mar 2007 | A1 |
| 20070070998 | Sethuram et al. | Mar 2007 | A1 |
| 20070091911 | Watanabe et al. | Apr 2007 | A1 |
| 20070223470 | Stahl | Sep 2007 | A1 |
| 20070237083 | Oh et al. | Oct 2007 | A9 |
| 20080002690 | Ver Steeg et al. | Jan 2008 | A1 |
| 20080112413 | Pong | May 2008 | A1 |
| 20080165797 | Aceves | Jul 2008 | A1 |
| 20080186981 | Seto | Aug 2008 | A1 |
| 20080189432 | Abali et al. | Aug 2008 | A1 |
| 20080267078 | Farinacci et al. | Oct 2008 | A1 |
| 20080298248 | Roeck et al. | Dec 2008 | A1 |
| 20090010159 | Brownell et al. | Jan 2009 | A1 |
| 20090103534 | Malledant et al. | Apr 2009 | A1 |
| 20090119565 | Park et al. | May 2009 | A1 |
| 20100039959 | Gilmartin | Feb 2010 | A1 |
| 20100049942 | Kim et al. | Feb 2010 | A1 |
| 20100111529 | Zeng et al. | May 2010 | A1 |
| 20100141428 | Mildenberger et al. | Jun 2010 | A1 |
| 20100216444 | Mariniello et al. | Aug 2010 | A1 |
| 20100284404 | Gopinath et al. | Nov 2010 | A1 |
| 20100290385 | Ankaiah | Nov 2010 | A1 |
| 20100290458 | Assarpour et al. | Nov 2010 | A1 |
| 20100315958 | Luo et al. | Dec 2010 | A1 |
| 20110019673 | Fernandez | Jan 2011 | A1 |
| 20110085440 | Owens et al. | Apr 2011 | A1 |
| 20110085449 | Jeyachandran et al. | Apr 2011 | A1 |
| 20110090784 | Gan | Apr 2011 | A1 |
| 20110164496 | Loh et al. | Jul 2011 | A1 |
| 20110225391 | Burroughs et al. | Sep 2011 | A1 |
| 20110249679 | Lin et al. | Oct 2011 | A1 |
| 20110255410 | Yamen et al. | Oct 2011 | A1 |
| 20110265006 | Morimura et al. | Oct 2011 | A1 |
| 20110299529 | Olsson et al. | Dec 2011 | A1 |
| 20120020207 | Corti et al. | Jan 2012 | A1 |
| 20120075999 | Ko et al. | Mar 2012 | A1 |
| 20120082057 | Welin et al. | Apr 2012 | A1 |
| 20120144064 | Parker et al. | Jun 2012 | A1 |
| 20120144065 | Parker et al. | Jun 2012 | A1 |
| 20120147752 | Ashwood-Smith et al. | Jun 2012 | A1 |
| 20120163797 | Wang | Jun 2012 | A1 |
| 20120170582 | Abts | Jul 2012 | A1 |
| 20120207175 | Raman | Aug 2012 | A1 |
| 20120287791 | Xi et al. | Nov 2012 | A1 |
| 20120300669 | Zahavi | Nov 2012 | A1 |
| 20120314706 | Liss | Dec 2012 | A1 |
| 20130044636 | Koponen et al. | Feb 2013 | A1 |
| 20130071116 | Ong | Mar 2013 | A1 |
| 20130083701 | Tomic et al. | Apr 2013 | A1 |
| 20130114599 | Arad | May 2013 | A1 |
| 20130114619 | Wakumoto | May 2013 | A1 |
| 20130170451 | Krause et al. | Jul 2013 | A1 |
| 20130208720 | Ellis | Aug 2013 | A1 |
| 20130242745 | Umezuki | Sep 2013 | A1 |
| 20130301646 | Bogdanski et al. | Nov 2013 | A1 |
| 20130315237 | Kagan et al. | Nov 2013 | A1 |
| 20130322256 | Bader et al. | Dec 2013 | A1 |
| 20130336116 | Vasseur et al. | Dec 2013 | A1 |
| 20140043959 | Owens et al. | Feb 2014 | A1 |
| 20140140341 | Bataineh et al. | May 2014 | A1 |
| 20140192646 | Mir et al. | Jul 2014 | A1 |
| 20140198636 | Thayalan | Jul 2014 | A1 |
| 20140313880 | Lu et al. | Oct 2014 | A1 |
| 20140328180 | Kim et al. | Nov 2014 | A1 |
| 20140343967 | Baker | Nov 2014 | A1 |
| 20150030033 | Vasseur et al. | Jan 2015 | A1 |
| 20150052252 | Gilde et al. | Feb 2015 | A1 |
| 20150092539 | Sivabalan | Apr 2015 | A1 |
| 20150098466 | Haramaty et al. | Apr 2015 | A1 |
| 20150124815 | Beliveau et al. | May 2015 | A1 |
| 20150163144 | Koponen et al. | Jun 2015 | A1 |
| 20150172070 | Csaszar | Jun 2015 | A1 |
| 20150194215 | Douglas et al. | Jul 2015 | A1 |
| 20150195204 | Haramaty et al. | Jul 2015 | A1 |
| 20150372898 | Haramaty et al. | Dec 2015 | A1 |
| 20150372916 | Haramaty et al. | Dec 2015 | A1 |
| 20160014636 | Bahr et al. | Jan 2016 | A1 |
| 20160080120 | Unger | Mar 2016 | A1 |
| 20160080321 | Pan | Mar 2016 | A1 |
| 20160182378 | Basavaraja et al. | Jun 2016 | A1 |
| 20170054591 | Hyoudou et al. | Feb 2017 | A1 |
| Number | Date | Country |
|---|---|---|
| WO 2016105446 | Jun 2016 | WO |
| Entry |
|---|
| Zahavi et al., “Distributed Adaptive Routing for Big-Data Applications Running on Data Center Networks,” Proceedings of the Eighth ACM/IEEE Symposium on Architectures for Networking and Communication Systems, New York, USA, pp. 99-110, Oct. 29-30, 2012. |
| U.S. Appl. No. 14/732,853 Office Action dated Jan. 26, 2017. |
| U.S. Appl. No. 14/662,259 Office Action dated Sep. 22, 2016. |
| Afek et al., “Sampling and Large Flow Detection in SDN”, SIGCOMM '15, pp. 345-346, Aug. 17-21, 2015, London, UK. |
| U.S. Appl. No. 14/745,488 Office Action dated Dec. 6, 2016. |
| U.S. Appl. No. 14/337,334 Office Action dated Oct. 20, 2016. |
| Dally et al., “Deadlock-Free Message Routing in Multiprocessor Interconnection Networks”, IEEE Transactions on Computers, vol. C-36, No. 5, May 1987, pp. 547-553. |
| Prisacari et al., “Performance implications of remote-only load balancing under adversarial traffic in Dragonflies”, Proceedings of the 8th International Workshop on Interconnection Network Architecture: On-Chip, Multi-Chip, 4 pages, Jan. 22, 2014. |
| Garcia et al., “On-the-Fly 10 Adaptive Routing in High-Radix Hierarchical Networks,” Proceedings of the 2012 International Conference on Parallel Processing (ICPP), pp. 279-288, Sep. 10-13, 2012. |
| Leiserson, C E., “Fat-Trees: Universal Networks for Hardware Efficient Supercomputing”, IEEE Transactions on Computers, vol. C-34, No. 10, pp. 892-901, Oct. 1985. |
| Ohring et al., “On Generalized Fat Trees”, Proceedings of the 9th International Symposium on Parallel Processing, pp. 37-44, Santa Barbara, USA, Apr. 25-28, 1995. |
| Zahavi, E., “D-Mod-K Routing Providing Non-Blocking Traffic for Shift Permutations on Real Life Fat Trees”, CCIT Technical Report #776, Technion—Israel Institute of Technology, Haifa, Israel, Aug. 2010. |
| Yuan et al., “Oblivious Routing for Fat-Tree Based System Area Networks with Uncertain Traffic Demands”, Proceedings of ACM SIGMETRICS—the International Conference on Measurement and Modeling of Computer Systems, pp. 337-348, San Diego, USA, Jun. 12-16, 2007. |
| Matsuoka S., “You Don't Really Need Big Fat Switches Anymore—Almost”, IPSJ SIG Technical Reports, vol. 2003, No. 83, pp. 157-162, year 2003. |
| Kim et al., “Technology-Driven, Highly-Scalable Dragonfly Topology”, 35th International Symposium on Computer Architecture, pp. 77-78, Beijing, China, Jun. 21-25, 2008. |
| Jiang et al., “Indirect Adaptive Routing on Large Scale Interconnection Networks”, 36th International Symposium on Computer Architecture, pp. 220-231, Austin, USA, Jun. 20-24, 2009. |
| Minkenberg et al., “Adaptive Routing in Data Center Bridges”, Proceedings of 17th IEEE Symposium on High Performance Interconnects, New York, USA, pp. 33-41, Aug. 25-27, 2009. |
| Kim et al., “Adaptive Routing in High-Radix Clos Network”, Proceedings of the 2006 ACM/IEEE Conference on Supercomputing (SC2006), Tampa, USA, Nov. 2006. |
| InfiniBand Trade Association, “InfiniBandTM Architecture Specification vol. 1”, Release 1.2.1, Nov. 2007. |
| Culley et al., “Marker PDU Aligned Framing for TCP Specification”, IETF Network Working Group, RFC 5044, Oct. 2007. |
| Shah et al., “Direct Data Placement over Reliable Transports”, IETF Network Working Group, RFC 5041, Oct. 2007. |
| Martinez et al., “Supporting fully adaptive routing in Infiniband networks”, Proceedings of the International Parallel and Distributed Processing Symposium (IPDPS'03),Apr. 22-26, 2003. |
| Joseph, S., “Adaptive routing in distributed decentralized systems: NeuroGrid, Gnutella & Freenet”, Proceedings of Workshop on Infrastructure for Agents, MAS and Scalable MAS, Montreal, Canada, 11 pages, year 2001. |
| Gusat et al., “R3C2: Reactive Route & Rate Control for CEE”, Proceedings of 18th IEEE Symposium on High Performance Interconnects, New York, USA, pp. 50-57, Aug. 10-27, 2010. |
| Wu et al., “DARD: Distributed adaptive routing datacenter networks”, Proceedings of IEEE 32nd International Conference Distributed Computing Systems, pp. 32-41, Jun. 18-21, 2012. |
| Ding et al., “Level-wise scheduling algorithm for fat tree interconnection networks”, Proceedings of the 2006 ACM/IEEE Conference on Supercomputing (SC 2006), 9 pages, Nov. 2006. |
| U.S. Appl. No. 14/046,976 Office Action dated Jun. 2, 2015. |
| Li et al., “Multicast Replication Using Dual Lookups in Large Packet-Based Switches”, 2006 IET International Conference on Wireless, Mobile and Multimedia Networks, , pp. 1-3, Nov. 6-9, 2006. |
| Nichols et al., “Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers”, Network Working Group, RFC 2474, 20 pages, Dec. 1998. |
| Microsoft., “How IPv4 Multicasting Works”, 22 pages, Mar. 28, 2003. |
| Suchara et al., “Network Architecture for Joint Failure Recovery and Traffic Engineering”, Proceedings of the ACM SIGMETRICS joint international conference on Measurement and modeling of computer systems, pp. 97-108, Jun. 7-11, 2011. |
| IEEE 802.1Q, “IEEE Standard for Local and metropolitan area networks Virtual Bridged Local Area Networks”, IEEE Computer Society, 303 pages, May 19, 2006. |
| Plummer, D., “An Ethernet Address Resolution Protocol,” Network Working Group ,Request for Comments (RFC) 826, 10 pages, Nov. 1982. |
| Hinden et al., “IP Version 6 Addressing Architecture,” Network Working Group ,Request for Comments (RFC) 2373, 26 pages, Jul. 1998. |
| U.S. Appl. No. 12/910,900 Office Action dated Apr. 9, 2013. |
| U.S. Appl. No. 14/046,976 Office Action dated Jan. 14, 2016. |
| Raindel et al., U.S. Appl. No. 14/673,892, filed Mar. 31, 2015. |
| “Equal-cost multi-path routing”, Wikipedia, 2 pages, Oct. 13, 2014. |
| Thaler et al., “Multipath Issues in Unicast and Multicast Next-Hop Selection”, Network Working Group, RFC 2991, 9 pages, Nov. 2000. |
| Nkposong et al., “Experiences with BGP in Large Scale Data Centers:Teaching an old protocol new tricks”, 44 pages, Jan. 31, 3014. |
| Mahalingam et al., “VXLAN: A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks”, Internet Draft, 20 pages, Aug. 22, 2012. |
| Sinha et al., “Harnessing TCP's Burstiness with Flowlet Switching”, 3rd ACM SIGCOMM Workshop on Hot Topics in Networks (HotNets), 6 pages, Nov. 11, 2004. |
| Vishnu et al., “Hot-Spot Avoidance With Multi-Pathing Over InfiniBand: An MPI Perspective”, Seventh IEEE International Symposium on Cluster Computing and the Grid (CCGrid'07), 8 pages, year 2007. |
| NOWLAB—Network Based Computing Lab, 2 pages, years 2002-2015 http://nowlab.cse.ohio-state.edu/publications/conf-presentations/2007/vishnu-ccgrid07.pdf. |
| Alizadeh et al.,“CONGA: Distributed Congestion-Aware Load Balancing for Datacenters”, Cisco Systems, 12 pages, Aug. 9, 2014. |
| Geoffray et al., “Adaptive Routing Strategies for Modern High Performance Networks”, 16th IEEE Symposium on High Performance Interconnects (HOTI '08), pp. 165-172, Aug. 26-28, 2008. |
| Anderson et al., “On the Stability of Adaptive Routing in the Presence of Congestion Control”, IEEE INFOCOM, 11 pages, 2003. |
| Perry et al., “Fastpass: A Centralized “Zero-Queue” Datacenter Network”, M.I.T. Computer Science & Artificial Intelligence Lab, 12 pages, year 2014. |
| Glass et al., “The turn model for adaptive routing”, Journal of the ACM, vol. 41, No. 5, pp. 874-903, Sep. 1994. |
| U.S. Appl. No. 14/673,892 Office Action dated Jun. 1, 2017. |
| U.S. Appl. No. 15/152,077 office action dated Dec. 1, 2017. |
| U.S. Appl. No. 15/050,480 office action dated Jan. 22, 2018. |
| U.S. Appl. No. 15/387,718 office action dated Mar. 9, 2018. |
| Number | Date | Country | |
|---|---|---|---|
| 20170180243 A1 | Jun 2017 | US |
