Patent Application
20150372898
The present invention relates generally to communication networks, and particularly to methods and systems for providing network high availability using temporary re-routing.
Various communication networks manage traffic routing, including recovery from route failure. For example, a network may comprise a centralized element or run a routing protocol that re-routes the traffic of a failing route via an alternative path. Methods for re-routing upon failure are known in the art.
For example, U.S. Patent Application Publication 2011/0249679, whose disclosure is incorporated herein by reference, describes a method for implementing fast re-route (FRR) by starting up an upper layer software protocol to manage and configure a FRR route. The upper layer software protocol sends down an active next hop of the FRR and a driver writes an IP address of the FRR into an ECMP table and creating a software table to record correspondence between a FRR group and an ECMP group. The upper layer software protocol informs the driver of a prefix address of a subnet route and the index of the FRR group. The driver finds the index of the ECMP group, and writes information of the subnet route and the index of the ECMP group into hardware. The upper layer software protocol informs the driver of the index of the FRR and an IP address of a new standby next hop. The driver looks up for the index of the ECMP group, and updates the next hop address of the ECMP group.
As another example, U.S. Pat. No. 7,234,001, whose disclosure is incorporated herein by reference, describes a method and apparatus that provide protection against resource failures in Open Shortest Path First (OSPF) networks. An internal router (IR) automatically maintains a back-up link in a dormant state until a network failure affecting communications with a primary adjacent area border router (ABR) is detected. Upon detection of the network failure, the IR activates the back-up link in order to enable traffic flow to an alternate adjacent ABR, which by-passes the network failure. Upon recovery of communications with the primary adjacent ABR, the IR automatically deactivates the back-up link and resumes traffic flow through the primary adjacent ABR.
As yet another example, in “Network Architecture for Joint Failure Recovery and Traffic Engineering,” whose disclosure is incorporated herein by reference, Suchara et al. describe a unified way to balance traffic load efficiently under a wide range of failure scenarios. The proposed architecture supports flexible splitting of traffic over multiple precomputed paths, with efficient path-level failure detection and automatic load balancing over the remaining paths. The authors propose two candidate solutions that differ in how the routers re-balance the load after a failure, leading to a trade-off between router complexity and load-balancing performance.
An embodiment of the present invention that is described herein provides a method in a network element that includes multiple interfaces for connecting to a communication network, including receiving via an ingress interface packets that are not allowed to undergo re-routing and that are addressed to a destination via a first egress interface. The packets are forwarded via the first egress interface when there is a valid path from the first egress interface to the destination. When there is no valid path from the first egress interface to the destination, a second egress interface is selected from a group of multiple egress interfaces that have respective paths to the destination and are assigned to packets for which re-routing is allowed, and the packets are forwarded via the second egress interface until recovering a path to the destination.
In some embodiments, the first egress interface is additionally assigned within the group of multiple egress interfaces only for packets for which re-routing is allowed. In other embodiments, the first egress interface is not assigned within the group of multiple egress interfaces for which re-routing is allowed. In yet other embodiments, selecting the second egress interface includes switching to the second egress interface in response to detecting, by the network element, that there is no valid path from the first egress interface to the destination.
In an embodiment, selecting the second egress interface includes switching to the second egress interface in response to receiving from the communication network a notification indicating that a route to the destination that traverses the first egress interface has failed. In another embodiment, recovering the path includes re-configuring the network element based on a global, network-wide routing decision. In yet another embodiment, forwarding the packets via the second interface includes forwarding all the packets destined to the first egress interface via the second egress interface.
In some embodiments, selecting the second egress interface includes choosing the second egress interface from among the multiple egress interfaces in the group, based on one or more fields in headers of the packets. In other embodiments, choosing the second egress interface includes holding a table that maps hash results calculated over the fields to respective egress interfaces in the group. In yet other embodiments, choosing the second egress interface includes calculating a hash function having a number of hash results equal to a number of entries of the table that are mapped to egress interfaces that have a valid path to the destination.
In an embodiment, choosing the second egress interface includes removing from the table one or more entries that are mapped to one or more egress interfaces that have no valid paths to the destination, and calculating a hash function having a number of hash results equal to a remaining number of entries corresponding to egress interfaces that have valid paths to the destination. In another embodiment, selecting the second egress interface includes switching to the second egress interface once, and not re-switching again until recovering the path to the destination or until forwarding the packets via the second egress interface to the destination fails. In yet another embodiment, the method includes receiving via the ingress interface packets to be delivered to the destination via a third egress interface that belongs to the group, forwarding the packets via the third egress interface when there is a valid path from the third egress interface to the destination, and, when there is no valid path from the third egress interface to the destination, selecting a fourth egress interface within the group that has a valid path to the destination, and forwarding the packets via the fourth egress interface until recovering a path to the destination.
There is additionally provided, in accordance with an embodiment of the present invention, a network element that includes multiple interfaces and switching circuitry. The interfaces are configured to connect to a communication network. The switching circuitry is configured to receive via an ingress interface packets that are not allowed to undergo re-routing and that are addressed to a destination via a first egress interface, to forward the packets via the first egress interface when there is a valid path from the first egress interface to the destination, and, when there is no valid path from the first egress interface to the destination, to select a second egress interface from a group of multiple egress interfaces that have respective paths to the destination and are assigned to packets for which re-routing is allowed, and to forward the packets via the second egress interface until recovering a path to the destination.
There is additionally provided, in accordance with an embodiment of the present invention, a method in a network element that comprises multiple interfaces for connecting to a communication network. A packet for which re-routing is allowed and that is addressed to a destination via a group of multiple first egress interfaces that have respective paths to the destination is received via an ingress interface. The packet is forwarded to the destination, via the group, when there is a valid path from at least one of the first egress interfaces to the destination. When there is no valid path from any of the first egress interfaces to the destination. A second egress interface outside the group that has a valid path to the destination is selected, and the packet is forwarded via the second egress interface until recovering a path from the first egress interfaces to the destination.
There is additionally provided, in accordance with an embodiment of the present invention, a network element that comprises multiple interfaces and switching circuitry. The multiple interfaces are configured to connect to a communication network. The switching circuitry is configured to receive via an ingress interface a packet for which re-routing is allowed and that is addressed to a destination via a group of multiple first egress interfaces that have respective paths to the destination, to forward the packet to the destination, via the group, when there is a valid path from at least one of the first egress interfaces to the destination, and, when there is no valid path from any of the first egress interfaces to the destination, to select a second egress interface outside the group that has a valid path to the destination, and to forward the packet via the second egress interface until recovering a path from the first egress interfaces to the destination.
The present invention will be more fully understood from the following detailed description of the embodiments thereof, taken together with the drawings in which:
Embodiments of the present disclosure that are described herein provide improved methods and systems for fast re-routing of a flow (or part thereof) to an alternative path in case of failure. The flow is typically re-routed until the failing route recovers or until the network re-configures the paths in the network using a centralized network element or a routing protocol. In the disclosed embodiments, the decisions related to fast re-routing are made locally by the network elements.
The description that follows refers mainly to network elements that comprise network switches. Alternatively, the network elements may comprise routers, bridges, gateways, or any other suitable type of network elements.
A network element such as a switch typically comprises multiple ports connecting to the network. Ports that are configured to receive or transmit packets are also referred to as ingress interfaces and egress interfaces, respectively.
In the description that follows, the term “flow” refers to a sequence of packets that are delivered from a source to a destination over the network. A given flow may comprise packets that should be delivered in order (e.g., TCP packets) as well as packets that are allowed to be delivered out of order (e.g., UDP packets).
In some embodiments, the switch assigns an egress interface to packets that require static routing, i.e., a route that does not change over time. For example, packets that should arrive at the destination in the same order they were sent may be assigned static routing. For packets that require static routing, this egress interface is also referred to as a static egress interface. The switch additionally configures a group of multiple egress interfaces in which re-routing within the group is allowed. Such a group may be used, for example, for adaptive routing (AR) and is referred to herein as an AR group. Note that the AR group can be used for other purposes, e.g., in load balancing such as in equal-cost multi-path routing (ECMP). In general, routing using an AR group is also referred to as dynamic routing.
A given switch may allow adaptive routing for some packets, e.g., when this interface belongs to an AR group, and force static routing for other packets.
In some embodiments, for a given flow, the egress interface that is assigned to route statically packets that require static routing also belongs to the AR group, i.e., with respect to the flow packets that are allowed to be routed adaptively. The disclosed techniques apply for both cases in which the static egress interface belongs, or does not belong, to the AR group. Alternatively, an egress interface may be assigned for static routing with respect to one flow and the AR group with respect to other one or more flows.
In some of the disclosed embodiments, when the currently used static route that passes via the static egress interface fails, the switch selects a fallback interface within the AR group and forwards the packets via the selected interface rather than via the static egress interface. The switch may use any suitable method to detect a failing route. In an example embodiment, the switch detects that the link connecting to the static egress interface is disconnected or has failed for any other reason. Alternatively, the switch receives from other elements in the network a notification that indicates the failing route.
The switch may select a fallback interface in various ways. In some embodiments, the fallback interface is predefined during initial configuration of the switch. Alternatively, the switch selects the fallback interface when actually re-routing the packets.
In some embodiments, selecting the fallback interface is based on certain fields in the packet headers, such as, for example, one or more of the source/destination address, source/destination port and the underlying protocol. In an embodiment, the switch calculates a hash function over one or more packet fields, and selects the fallback interface based on the hash result. For example, the switch may hold one or more tables that map hash values to respective egress interfaces in the AR group, and select one of the tables based, for example, on the destination address.
In some embodiments, when the hash result corresponds to a failing route, the switch attempts another selection, for example, using a random selection, by calculating a hash function with a smaller number of possible hash results, or by excluding from the table entries corresponding to one or more failing routes.
The methods described above are also applicable to situations in which a route from an egress interface that belongs to the AR group fails. In such situations, the switch may use methods similar to the methods described above for selecting a fallback interface within the AR group.
In the disclosed techniques, the switch re-routes packets that are to be forwarded statically via a failing route to a fallback interface that belongs to a group of interfaces that normally use for adaptive routing. For a given flow, the switch selects a single fallback interface, and therefore the flow is re-routed once, which may result with only temporary out-of-order packet delivery.
Although re-configuring while having a global network view such as when using a routing protocol may result in better utilization of network resources, the local re-routing is typically much faster than re-configuring the paths by a centralized element or by a routing protocol and therefore significantly improves high availability capabilities in the network.
Network 20 provides packet communication for multiple nodes 24. A node may comprise, for example, a Host Channel Adapter (HCA) or other Network Interface Card (NIC) of a computer. Each flow of packets originates from a certain source node and is addressed to a certain destination node. The packets of the flow are forwarded over a route through the network that traverses various switches 28 and network links 32 and 34. The links are assumed bidirectional. In the disclosed embodiments, network 20 supports multi-path communication, i.e., a source node may communicate with a destination node over multiple routes or paths simultaneously.
Since different routes may have different latencies, re-routing a flow may result in out-of-order packet delivery to the destination. In the description that follows the term “static routing” means that the route does not change over time. In other words, packets of a flow that require static routing are delivered via the same path of switches and network links. A static route can be assigned to one or more flows simultaneously. For example, in
Re-routing may be applied to flows that permit out of order packet delivery. For example, in
Note that a route or part thereof can deliver both packets that require static routing and packets that are allowed adaptive routing. For example, in
In an Infiniband application, the switches and links belong to a certain Infiniband subnet that is managed by a Subnet Manager (SM) 26. Without loss of generality the SM may be embodied in one of the switches, e.g., in switch 28A. Other networks, such as IP networks, are typically managed using a routing protocol such as, for example, the Open Shortest Path First (OSPF) protocol, the Intermediate System to Intermediate System (IS-IS) protocol or the Border Gateway Protocol (BGP). In order to demonstrate the disclosed techniques, the figure shows a simplified example of a source node, a destination node, switches 28A . . . 28F, links 32A . . . 32D that are part of a static route for a given flow, and links 34A . . . 34G that are part of routes that participate in adaptive routing for the same or other flows. In a real-life network, the switches may be interconnected in various topologies. An example Fat-Tree (FT) topology is addressed in greater detail further below.
Each switch 28 comprises multiple ports 36 that are configured to receive and transmit packets, switch fabric 40 that is configured to forward the packets between the ports, and a control unit 44 that controls and configures the various switch elements. Among other tasks, control units 44 of the switches in network 20 carry out re-routing techniques to provide high availability in network 20.
Switch ports 36 that are configured to receive packets are also referred to as ingress interfaces and ports that are configured to transmit packets are also referred to as egress interfaces. For a given flow, an egress interface can be configured to deliver all the flow packets statically or adaptively. Alternatively, the egress interface may be configured to deliver certain flow packets statically and other packets adaptively.
A currently used route may fail at some point in time. For example, a link along the route path may be disconnected or otherwise fail. Alternatively or additionally, a switch in the route path may malfunction. In some embodiments, a centralized element in the network, such as SM 26, is notified about a failing route and re-configures the switches to create an alternative route. In other embodiments, a switch communicates with other switches in the network using a routing protocol to re-configure the routing paths. As noted, recovering the path is not limited to local switch decisions, and therefore re-configuring the switch is rather based on a global, network-wide routing decision.
The description that follows refers to a certain partitioning of functions between the switch fabric and the control unit. In alternative embodiments, however, any other suitable functional partitioning can be used. In the context of the present patent application and in the claims, switch fabric 40 and control unit 44 are referred to jointly as switching circuitry that carries out the methods described herein.
The network and switch configurations shown in
In some embodiments, certain switch functions, such as certain functions of control unit 44, may be implemented using a general-purpose processor, which is programmed in software to carry out the functions described herein. The software may be downloaded to the processor in electronic form, over a network, for example, or it may, alternatively or additionally, be provided and/or stored on non-transitory tangible media, such as magnetic, optical, or electronic memory.
In the present example, EP1 is configured as a static egress interface for packets of a given flow that require in-order delivery. Thus, the switch forwards packets of the given flow that should be delivered to the destination in the same order they were sent via EP1. Additionally, EP2 . . . EP5 are configured as a group 50A of egress interfaces for other packets of the given flow that are allowed out-of-order delivery, wherein re-routing within the group is allowed. For example, EP2 . . . EP5 can serve as an AR group for adaptive routing. Alternatively, EP2 . . . EP5 serve in applying equal-cost multi-path routing (ECMP) or in other suitable load balancing methods. Switch 28 is configured to forward packets that are allowed to undergo out-of-order delivery to one of egress interfaces EP2 . . . EP5.
In alternative embodiments, EP1 . . . EP5 are all assigned as an AR group 50B, and EP1 additionally serves as a static egress interface for the packets of the given flow that require in-order delivery. In this case, the switch forwards incoming packets statically via EP1 or adaptively via one of EP1 . . . EP5 based on certain fields in the packets. Although the techniques described below refer mainly to the case in which EP1 is assigned only for static routing, these techniques are similarly applicable to the case in which EP1 belongs to the AR group.
Consider now an event in which the route via EP1 to the destination fails. As a result, the packets that the switch normally forwards to EP1 may be lost. In some embodiments, when the route via EP1 fails, the switch re-routes packets that should be forwarded via EP1 to a fallback egress interface selected from EP2 . . . EP5. This re-routing may persist until a suitable condition is met.
For example, the switch re-routes the packets until the failing route recovers, e.g., a failing link or switch along the failing route recovers, or until the SM or a routing protocol re-configures the switch. Note that the switch selects only a single egress interface out of EP2 . . . EP5 to replace EP1 upon failure, and therefore, although re-routing may cause some packets to arrive at the destination out-of-order, delivery order is typically maintained shortly after the re-routing starts.
In some embodiments, when one of the routes via EP2 . . . EP5 fails, switch 28 re-routes the packets destined to the failing interface to another egress interface within the group. In
The method begins by control unit 44 of switch 28 configuring forwarding rules to a given ingress interface such as IP1 of
The forwarding rules configured at step 100 are typically based on certain fields in the packet headers. Such fields may include, for example, one or more of the source/destination address, source/destination port number and the underlying protocol type. In some embodiments, the switch calculates a hash function over one or more of the fields and uses the hash result for selecting a respective egress interface, as will be described in greater detail below.
In this example, control unit 44 assigns EP1 as a static egress interface to packets that require static routing, and an AR group EP2 . . . EP5 to packets that may be routed within the AR group using adaptive routing techniques. At a reception step 104, switch 28 receives via IP1 a packet that should be forwarded statically, and control unit 44 identifies that the packet should be forwarded via EP1. As noted above, in alternative embodiments, EP1 . . . EP5 are assigned for adaptive routing, wherein EP1 is additionally assigned for static routing for certain packets for which re-routing is not allowed.
At a validation step 108, the control unit checks whether a valid route exists from EP1 to the destination. When such a valid route exists, the switch forwards the packet via EP1, at a static forwarding step 112. Otherwise, the switch selects a fallback egress interface that belongs to the AR group, at a fallback forwarding step 116, and forwards the packet accordingly. Following steps 112 and 116, the method loops back to step 104 to receive subsequent packets via the ingress interface.
Control unit 44 may check the route validity at step 108 in various ways. In some embodiments, the switch senses the state of the link connecting to the selected egress interface to verify whether the link is up. In other embodiments, the switch receives from the network a notification that indicates that the route corresponding to the static routing has failed. In yet other embodiments, the control unit considers that a valid route exists when the network re-configures the switch using, for example, a SM or a routing protocol.
In the network, any suitable method can be used for remote signaling of a failing route. For example, U.S. patent application Ser. No. 13/754,921, filed Jan. 31, 2013, which is incorporated herein by reference, describes methods of this type that are collectively referred to as Adaptive Routing Notification (ARN). As another example, notifications of a failing route may be sent via Explicit Congestion Notification (ECN) packets that are conventionally used in congestion control.
The methods of
In one embodiment, the AR group (e.g., AR 50A in
In the alternative embodiment in which EP1 is assigned for static routing of certain packets and also belongs to the AR group, when one or more of EP2 . . . EP5 fails the relevant packets may be forwarded via EP1.
At step 116 of the method of
In some embodiments, the fallback egress interface is pre-configured, e.g., at step 100. Assigning fallback routes before they are actually required can be implemented efficiently in hardware, resulting in fast re-routing upon failure. The hardware can be configured to return to the static routing when the static route recovers or when the switch is re-configured. Although the description below refers mainly to an AR group that includes EP2 . . . EP5, the disclosed embodiments are applicable mutatis mutandis also to the case in which the AR group includes EP1 . . . EP5.
In other embodiments, selecting the fallback egress interface is based on flow parameters carried in certain fields of the packet headers. For example, the selection can be based on hashing the destination address of the packet. Alternatively, the selection is based on hashing various packet fields. Further alternatively, the selection may be based on the identity of the failing egress interface, i.e., re-routing all the traffic from the failing egress interface to another egress interface. This technique, however, may result in overloading the fallback egress interface. Further alternatively, when the routing via one of EP2 . . . EP5 fails, the control unit may select the first next egress interface within EP2 . . . EP5 (using cyclic counting) that has a valid route to the destination. For example, re-route to EP5 upon failure in forwarding via EP4. An egress interface that has a valid route to the destination is also referred to herein as a “valid egress interface”.
As mentioned above, selecting a fallback egress interface may be based on calculating a hash function over one or more fields in the packet headers. We now describe several hash-based selection methods that can be used, for example, when EP2 . . . EP5 are configured as an ECMP group.
In an embodiment, the switch holds a mapping table that maps hash results to respective egress interfaces. In the present example, the mapping table additionally includes marking the validity status of the egress interfaces, i.e., an indication of whether there is a valid route from the egress interface to the destination. Table 1 depicts an example mapping table that maps six hash results to respective egress interfaces. Note that multiple hash results can be mapped to the same egress interface. For example, in Table 1 hash results 2 and 3 are mapped to egress interface 13, and hash results 4 and 5 are mapped to egress interface 14.
In some embodiments, the switch comprises multiple mapping tables similar in structure to Table 1, and the control unit selects one of these mapping tables based, for example, on the destination address.
When the hash result maps to an egress interface that is marked as valid, the switch forwards the packet accordingly. Otherwise, the control unit selects a fallback egress interface out of the remaining valid egress interfaces as described herein. In one embodiment, the control unit calculates a hash function that has a number of hash results equal to the number of remaining egress interfaces. For example, when routing via EP13 fails, calculate a hash function that has four outcomes: 0,1,2,3 corresponding to the table entries 4,5,0,1, respectively. In other embodiments, the control unit removes from the mapping table entries of invalid egress interfaces and hashes over the packet fields using a hash function having a number of hash results equal to the remaining entries.
In an embodiment, selecting an egress interface involves calculating two or more different hash functions and using one of the hash results as required. In the example of Table 1 above, assume that the hash result generated by a default hash function equals three, which corresponds to the invalid egress interface EP13. Further assume that the switch calculates an additional hash function whose outcome equals four. In this case EP14 is selected since hash result four corresponds to the valid egress interface EP14. Re-selection of an egress interface using an alternative hash function is also referred to herein as “rehashing.”
In some embodiments, the control unit applies rehashing one or more times, or selects a fallback egress interface at random, in attempt to select a valid egress interface, and after a limit number of attempts the control unit selects a predefined egress interface. In other embodiments, the control unit selects the fallback egress interface by cyclically advancing a predefined number of entries in the mapping table. In this case when the hash result equals 2 or 3 (both mapped to invalid EP13) the control unit advances three entries to entry 5 or 0, respectively.
The disclosed techniques are applicable in various network topologies. One popular network topology is Fat-Tree (FT), which is addressed, for example, by Ding et al., in “Level-wise Scheduling Algorithm for Fat Tree Interconnection Networks,” Proceedings of the 2006 ACM/IEEE Conference on Supercomputing (SC 2006), November, 2006, which is incorporated herein by reference.
The network of
The network is arranged in three levels: Level 0 (L0) comprises switches that are numbered (0,xx), Level 1 (L1) comprises switches that are numbered (1,xx), and Level 2 (L2) comprises switches that are numbered (2,xx). Clients (e.g., HCAs) are connected only to the down ports of the switches in L0. Each switch is similar to switches 28 of
Consider now a source node (0,011) connecting to switch (0,01) that communicates with a destination node (0,121) connected to switch (0,12). To send packets over the static or adaptive routes, switch (0,01) should forward the packets that require static routing upwards to switch (1,01) and packets that require adaptive routing to switches (1,11) or (1,21). In one embodiment, for providing high availability, switch (0,01) may use the static route as long as the static route is functional, and re-route to using one of the adaptive routes as a fallback route when the static route fails. In other embodiments, upon failure of the static route, the source node communicates with the destination node over both the fallback routes, and when one of them fails, the traffic is routed via the other fallback route.
The FT topology configuration in
It will be appreciated that the embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and sub-combinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art. Documents incorporated by reference in the present patent application are to be considered an integral part of the application except that to the extent any terms are defined in these incorporated documents in a manner that conflicts with the definitions made explicitly or implicitly in the present specification, only the definitions in the present specification should be considered.
This application claims the benefit of U.S. Provisional Patent Application 62/015,550, filed Jun. 23, 2014, whose disclosure is incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 62015550 | Jun 2014 | US |
