Patent Application
20140293786
This disclosure relates to networking. This disclosure also relates to path resolution in network devices such as switches and routers.
High speed data networks form part of the backbone of what has become indispensable worldwide data connectivity. Within the data networks, network devices such as switching devices direct data packets from source ports to destination ports, helping to eventually guide the data packets from a source to a destination. Improvements in packet handling, including improvements in path resolution, will further enhance performance of data networks.
The innovation may be better understood with reference to the following drawings and description.
Example Architecture
The switch architecture 100 includes several tiles, such as the tiles specifically labeled as tile A 102 and the tile D 104. In this example, each tile has processing logic for handling packet ingress and processing logic for handling packet egress. A switch fabric 106 connects the tiles. Packets, sent for example by source network devices such as application servers, arrive at the network interfaces 116. The network interfaces 116 may include any number of physical ports 118. The ingress logic 108 buffers the packets in memory buffers. Under control of the switch architecture 100, the packets flow from an ingress tile, through the fabric interface 120 through the switching fabric 106, to an egress tile, and into egress buffers in the receiving tile. The egress logic sends the packets out of specific ports toward their ultimate destination network device, such as a destination application server.
Each ingress tile and egress tile may be implemented as a unit (e.g., on a single die or system on a chip), as opposed to physically separate units. Each tile may handle multiple ports, any of which may be configured to be input only, output only, or bi-directional. Thus, each tile may be locally responsible for the reception, queueing, processing, and transmission of packets received and sent over the ports associated with that tile.
As an example, in
The techniques described below are not limited to any particular configuration of line rate, number of ports, or number of tiles, nor to any particular network device architecture. Instead, the techniques described below are applicable to any network device that incorporates the path resolution analysis logic described below. The network devices may be switches, routers, bridges, blades, hubs, or any other network device that handles routing packets from sources to destinations through a network. The network devices are part of one or more networks that connect, for example, application servers together across the networks. The network devices may be present in one or more data centers that are responsible for routing packets from a source to a destination.
The tiles include packet processing logic, which may include ingress logic 108, egress logic 110, analysis logic, and any other logic in support of the functions of the network device. The ingress logic 108 processes incoming packets, including buffering the incoming packets by storing the packets in memory. The ingress logic 108 may define, for example, virtual output queues 112 (VoQs), by which the ingress logic 108 maintains one or more queues linking packets in memory for the egress ports. The ingress logic 108 maps incoming packets from input ports to output ports, and determines the VoQ to be used for linking the incoming packet in memory. The mapping may include, as examples, analyzing addressee information in the packet headers, and performing a lookup in a mapping table that matches addressee information to output port(s).
The egress logic 110 may maintain one or more output buffers 114 for one or more of the ports in its tile. The egress logic 110 in any tile may monitor the output buffers 114 for congestion. When the egress logic 110 senses congestion (e.g., when any particular output buffer for any particular port is within a threshold of reaching capacity), the egress logic 110 may throttle back its rate of granting bandwidth credit to the ingress logic 108 in any tile for bandwidth of the congested output port. The ingress logic 108 responds by reducing the rate at which packets are sent to the egress logic 110, and therefore to the output ports associated with the congested output buffers.
The ingress logic 108 receives packets arriving at the tiles through the network interface 116. In the ingress logic 108, a packet processor may perform link-layer processing, tunnel termination, forwarding, filtering, and other packet processing functions on the received packets. The packets may then flow to an ingress traffic manager (ITM). The ITM writes the packet data to a buffer, from which the ITM may decide whether to accept or reject the packet. The ITM associates accepted packets to a specific VoQ, e.g., for a particular output port. The ingress logic 108 may manage one or more VoQs that are linked to or associated with any particular output port. Each VoQ may hold packets of any particular characteristic, such as output port, class of service (COS), priority, packet type, or other characteristic.
The ITM, upon linking the packet to a VoQ, generates an enqueue report. The ITM may also send the enqueue report to an ingress packet scheduler. The enqueue report may include the VoQ number, queue size, and other information. The ITM may further determine whether a received packet should be placed on a cut-through path or on a store and forward path. If the receive packet should be on a cut-through path, then the ITM may send the packet directly to an output port with as low latency as possible as unscheduled traffic, and without waiting for or checking for any available bandwidth credit for the output port. The ITM may also perform packet dequeueing functions, such as retrieving packets from memory, forwarding the packets to the destination egress tiles, and issuing dequeue reports. The ITM may also perform buffer management, such as admission control, maintaining queue and device statistics, triggering flow control, and other management functions.
In the egress logic 110, packets arrive via the fabric interface 120. A packet processor may write the received packets into an output buffer 114 (e.g., a queue for an output port through which the packet will exit) in the egress traffic manager (ETM). Packets are scheduled for transmission and pass through an egress transmit packet processor (ETPP) and ultimately out of the output ports.
The ETM may perform, as examples: egress packet reassembly, through which incoming cells that arrive interleaved from multiple source tiles are reassembled according to source tile contexts that are maintained for reassembly purposes; egress multicast replication, through which the egress tile supports packet replication to physical and logical ports at the egress tile; and buffer management, through which, prior to enqueueing the packet, admission control tests are performed based on resource utilization (i.e., buffer and packet descriptors). The ETM may also perform packet enqueue/dequeue, by processing enqueue requests coming from the ERPP to store incoming frames into per egress port class of service (CoS) queues prior to transmission (there may be any number of such CoS queues, such as 2, 4, or 8) per output port.
The ETM may also include an egress packet scheduler to determine packet dequeue events, resulting in packets flowing from the ETM to the ETPP. The ETM may also perform egress packet scheduling by arbitrating across the outgoing ports and COS queues handled by the tile, to select packets for transmission; flow control of egress credit scheduler (ECS), by which, based on total egress tile, per egress port, and per egress port and queue buffer utilization, flow control is sent to the ECS to adjust the rate of transmission of credit grants (e.g., by implementing an ON/OFF type of control over credit grants); flow control of tile fabric data receive, through which, based on total ETM buffer utilization, link level flow control is sent to the fabric interface 120 to cease sending any traffic to the ETM.
In the example of
The resolution configuration information 212 may guide the operation of the path resolution instructions 210. For example, the resolution configuration information 212 may specify the number of size of ECMP groups, ECMP member tables, may specify hash functions, the number of stages in the path resolution, or other parameters employed by the multiple stage resolution techniques described below.
Path Resolution
In a network of interconnected nodes, there may be multiple paths from a source A to reach a destination B. The nodes may be routers or switches, as examples, or may be other types of network devices. Each node may make an independent decision of which path to take to reach the destination B and each node may determine a next hop node, e.g., the next node along a particular path (the “next hop”) to which to forward the packet. For each packet a node may perform ECMP resolution and may determine the next hop node on one of the equal cost paths to the destination B. One goal of ECMP resolution is to increase bandwidth available between A and B by distributing traffic among the equal cost paths.
In weighted ECMP, the paths between A and B forming an ECMP group may be weighted differently. The Weighted (W) ECMP (W-ECMP) resolution may then select a path from an ECMP group based on the weights of each path, typically given by the weights on the next hop nodes.
In
In order to select among potentially multiple next hops 306 in the ECMP member table 304 for the ECMP group, the system may determine a hash value 308. The hash value 308 may be a function of the data in selected packet fields. Given the hash value 308, the next hop may be selected from the ECMP member table 304. In particular, the system may determine the member index 310 into the ECMP member table 304, at which the identifier of the next hop is stored, according to:
member_index=(hash_value %(member_count+1))+base—ptr
Where the ‘%’ operator is the modulo operator: remainder after division.
To accommodate next hops within a group has with different weights, the next hop may appear multiple times in the ECMP member table 304 for the group, in proportion to its weighting. The multiple appearances in the ECMP member table 304 implements the weighting for the next hop by providing additional or fewer entries for the next hop, leading to additional or fewer selections of the next hop.
In this example, the overlay network 400 includes a layer N and a layer M. Within layer M is a first ECMP group 402. Within layer N is a second ECMP group 404 and a third ECMP group 406.
{R2, R6}, {R2, R7}, {R3, R8}, {R3, R9}, {R3, R10}
The nodes R6, R7 and R8, R9, R10 are assumed, in this example, to forward only in Layer N. Any node, including the nodes R6-R10, may also perform ECMP resolution to select the next hops in Layer N to reach node R2, or R3 respectively. The example below is given from the perspective of the node R1 making a decision on which node is the next hop for a particular packet it has received.
Note that nodes, e.g., R1, in an overlay network may need to resolve ECMP paths in multiple layers. The ECMP paths in one or more layers may be weighted.
Table 1, below, summarizes the weights shown in
To implement the weighting, the number of entries in the ECMP member table may grow as a multiplicative function of the weights. For this example:
[(W8*Wb)+(W9*Wb)+(W10*Wb)]*2+[(W6*Wa)+(W7*Wa)]*3=24+36=60 entries. In other words, there will be 24 entries of next hops from tunnel B and 36 entries of next hops from tunnel A, so that 1.5 times the traffic is routed through tunnel A as is routed through tunnel B. Within the 24 entries for tunnel B, there will be 4 node R8 entries, 8 node R9 entries, and 12 node R10 entries. Within the 36 entries for tunnel A, there will be 9 node R6 entries and 27 node R7 entries.
In other words, the number of entries per node in the ECMP member table reflects the desired percentage of traffic sent through that node. In the example above,
R6 handles (3/5)*(1/4) of all traffic=(3/20)=15% percent of all traffic
R7 handles (3/5)*(3/4)=(9/20)=45%
R8 handles (2/5)*(1/6)=(2/30)=6.66%
R9 handles (2/5)*(2/6)=(4/30)=13.33%
R10 handles (2/5)*(3/6)=(6/30)=20%
Sixty (60) is the least number, n, for which n* percentage of traffic is an integer, for all path probabilities, because 60 includes 20 and 30 as a factor:
60*15%=9 entries for R6
60*45%=27 entries for R7
60*6.66%=4 entries for R8
60*13.33%=8 entries for R9
60*20%=12 entries for R10
When the relative probabilities change, the minimum number of entries will also change, and the minimum number of entries is very often a multiplicative function of the weights. This causes the ECMP member table 304 to grow quickly, consuming valuable resources in the system.
However, with the path resolution techniques described below, the number of ECMP member table entries may be reduced. For the example above, using the techniques described below, the number of ECMP member table entries may be reduced to:
Wa+Wb+W8+W9+W10+W6+W7=15 entries.
In other words, the path resolution techniques described below avoid growth in the number of entries as a function of the multiplication of the weights in the multiple layers. The reduction in the number of entries may translate into, as examples, a lower memory requirement for routing, freeing existing memory for other uses, or permitting less memory to be installed in the node, or other benefits.
A network device (e.g., as implemented by the architectures 100 and 200, or by other architectures) may perform ECMP resolution in multiple stages. The multiple stage resolution may occur in hardware, or for example by executing the path resolution instructions 210 with the processor 204, or in a combination of hardware and software. Examples of multiple stage ECMP resolution are shown in
Continuing the example of
The second stage 604 resolves in Layer N (e.g., proceeding to the next lower network layer). The output 622 of the second stage 604 is next hop R6 or R7 to reach R2 (when stage 1 determined that R2 was the next hop), or next hop R8 or R9 or R10 to reach R3 (when stage 1 determined that R2 was the next hop). The stage 2 ECMP member table 624 (e.g., 8K entries in size) implements the relative weighting of R6, R7, R8, R9, and R10 (e.g., 3 entries for R7 and 1 entry for R6 as noted in Table 1).
Note that the ECMP member table 608 may specify a next hop when, e.g., a single level of resolution is performed, when the current stage resolves down to an actual next hop, or may specify a next ECMP group, e.g., that identifies a group in the next network layer down. A type entry (e.g., a bit field) in the ECMP member table 608 may specify which type of result (e.g., a next hop or a group) is found in any entry in the table. Further, different types of packets may be subject to different numbers of levels of resolution. If in this example, the network device is only forwarding in layer N for a particular packet, then there may be only one ECMP group to check.
Further, the output selection logic 610 may be responsive to the output selection signal 632. The output selection signal 632 may determine whether the path resolution is finished at a particular stage (e.g., finished at stage 1602). In other words, the output selection signal 632 may force the resolution to end at any given stage, and, as a specific example, to be a single level resolution. The output selection signal 632 may be provided for backwards compatibility and for low latency operation by avoiding multiple sequential table lookups. In that case, the first stage 602 may be configured to operate as previously described, to resolve one or more stages of path selection using many more entries in the ECMP member table 608, for example. In other words, the output selection signal 632 may facilitate operation in a reduced number of levels mode (e.g., a single level mode), in which there may be, in the final stage, a relatively larger ECMP member table as described above that holds a number of entries that may be a multiplicative function of the weights to implementing path weighting.
As a specific example,
In
Traffic Redistribution
Described below are techniques to redistribute traffic to a downed next hop quickly and without reassigning traffic that was going to other unaffected next hops, using multi-stage ECMP resolution. For the purposes of illustration, assume that node A may forward packets to node B via 3 next hop routers R1, R2, R3, forming an ECMP Group. Assume also that ECMP Group member count is programmed to 3 and ECMP member table has 3 entries, R1, R2, R3. When R3 goes down, the network device updates member count to 2. The update, however, may cause traffic that was not flowing to R3 to be potentially reassigned to a different next hop, and this may result in temporary re-ordering of packets within a flow received at node B.
It may be desirable that only traffic that was previously assigned to R3 should be affected by R3 going down, and that only the R3 traffic should be redirected to either R1 or R2. In other words, traffic previously assigned to R1 should not change assignment to R2 and traffic previously assigned to R2 should not change to R1. It may also take a certain amount of time for the network device to reprogram the ECMP group table and each ECMP member table entry that included an R3 next hop entry (e.g., to remove the entry).
Continuing the example with respect to
Explained more generally, the architecture 1000 may establish fallback ECMP groups that selectively omit specific next hops for which protection is desired. For example, to protect against next hop 1 failure, an ECMP group is defined to include next hop 2 and next hop 3. Similarly, to protect against next hop 2 failure, an ECMP group is defined to include next hop 1 and next hop 3. And, to protect against next hop 3 failure, an ECMP group is defined to include next hop 2 and next hop 1. Accordingly regardless of which next hop fails, there is another ECMP group that omits the failed next hop and that can resolve the next hop in the path by specifying the allowable routing options other than the failed next hop. Note that a processing stage subsequent to the stage that detects the failure may resolve the fallback group.
As shown in
Recall that ECMP member table A 1002 may include member table entries (e.g., the member table entry 1020) that include: next hop ID 1014, which identifies a selected next hop, and the following redistribution protection entries: fallback group 1016 (set to 101 in this example), which identifies the ECMP group to use if a protection status is set; and protection group pointer 1018 (set to 10 in this example), which points to a protection group table from which to obtain status information.
When ECMP Group 100 resolves to next hop 1, the network device retrieves the protection group pointer 1018 from the member table entry 1020, and reads the protection group 10 in the protection group table 1102. The status information for protection group 10 indicates that next hop 1 is down. As a result, the network device selects the fallback ECMP group specified by fallback group 1016: ECMP group 101. Recall that ECMP group 101 includes next hop 2 and next hop 3 as members, and thus will not route any packets through next hop 1.
The network device passes the ECMP group selection (101) to the resolution stage 21006. The network device may also set the stage 2 ECMP flag 1104 to indicate that the second stage 1006 should act on the output of the first stage 1004. The second stage 1006 thus resolves ECMP group 101, and obtains either next hop 2 or next hop 3 as a next hop. The second stage 1006 may also check whether the selected next hop is down, using the protection group table, and member table entries described above. Thus, referring back to
In the approach described in
In each stage, the ECMP group table is established to include a group entry for each group that the stage will handle (1204). In each stage also, an ECMP member table is established to include group member entries for each group that reflect the weighting of the group members in each group (1206).
When the network device receives a packet (1208), the network device may perform multi-stage ECMP resolution. The network device need not use multi-stage ECMP resolution for every packet, however. Instead the network device may decide for which packets to perform ECMP resolution based on packet characteristics and packet criteria that may be present, for example, in the resolution configuration information 212.
When the network device will perform multi-stage ECMP resolution, the network device starts the next stage of resolution (1212). The result of the stage may be a next hop, for example (1214). In that case, the network device may send the packet to the next hop determined by the resolution stage (1216). Note that the network device may stop resolution at any stage (1218). If resolution will continue, then the network device may pass the current resolution stage result on to the next stage (1220). The current resolution stage result may be an identifier of a next group (e.g., for routing in the next network layer), for example. Resolution may continue through as many stages as desired, until a next hop is identified, or until the network device decides to stop the resolution. When multi-stage resolution is not performed, then the logic 1200 may perform single stage resolution and forward the packet to the next hop (1222).
During operation, the network device, receives a packet (1308), and also monitors for next hop failure, and sets status bits accordingly, e.g., in the appropriate protection group tables. When the packet is subject to multi-stage path resolution, the logic 1300 submits the packet to the next stage of path resolution (1310). In that respect, the logic 1300 may, for example, retrieve the protection group pointer from the member entry, and read the protection group in the protection group table for the next hop selected by the resolution stage (1312). The protection group table, as noted above, includes status information that indicates whether the next hop is down, and the member group entry for a next hop includes identifies a fallback group to use in the next stage when the next hop is down.
When the next hop determined by the current stage is down, then the logic 1300 may select the fallback ECMP group specified by fallback group identifier in the next hop member entry (1314). The logic 1300 provides the fallback group identifier to the next resolution stage (1316). For example, the logic 1300 may provide a pointer into the ECMP group table in the next stage that points to the fallback group. ECMP resolution may then continue in the subsequent stage, e.g., to select from among the next hops in the fallback group as the next hop for the packet.
The methods, devices, techniques, and logic described above may be implemented in many different ways in many different combinations of hardware, software or both hardware and software. For example, all or parts of the system may include circuitry in a controller, a microprocessor, or an application specific integrated circuit (ASIC), or may be implemented with discrete logic or components, or a combination of other types of analog or digital circuitry, combined on a single integrated circuit or distributed among multiple integrated circuits. All or part of the logic described above may be implemented as instructions for execution by a processor, controller, or other processing device and may be stored in a tangible or non-transitory machine-readable or computer-readable medium such as flash memory, random access memory (RAM) or read only memory (ROM), erasable programmable read only memory (EPROM) or other machine-readable medium such as a compact disc read only memory (CDROM), or magnetic or optical disk. Thus, a product, such as a computer program product, may include a storage medium and computer readable instructions stored on the medium, which when executed in an endpoint, computer system, or other device, cause the device to perform operations according to any of the description above.
The processing capability described above may be distributed among multiple system components, such as among multiple processors and memories, optionally including multiple distributed processing systems. Parameters, databases, and other data structures may be separately stored and managed, may be incorporated into a single memory or database, may be logically and physically organized in many different ways, and may implemented in many ways, including data structures such as linked lists, hash tables, or implicit storage mechanisms. Programs may be parts (e.g., subroutines) of a single program, separate programs, distributed across several memories and processors, or implemented in many different ways, such as in a library, such as a shared library (e.g., a dynamic link library (DLL)). The DLL, for example, may store code that performs any of the system processing described above. While various embodiments of the invention have been described, it will be apparent to those of ordinary skill in the art that many more embodiments and implementations are possible within the scope of the invention. Accordingly, the invention is not to be restricted except in light of the attached claims and their equivalents.
This application claims priority to, and incorporates by reference, U.S. Provisional Patent Application Ser. No. 61/807,181, filed 1 Apr. 2013.
| Number | Date | Country | |
|---|---|---|---|
| 61807181 | Apr 2013 | US |
