The present invention relates to a method for routing packets in a direct interconnect network. More particularly, the present invention relates to a method for distributing multipath flows of packets in a direct interconnect network.
One method of distributing packets from a source node S to a destination node D involves the use of source routing, wherein the source node determines the entire path that a packet must follow to reach the destination node. In this respect, a head flit header in a packet may be populated with a series of node ports to use, which defines the path through the network. In the case where a single flow is distributed over multiple paths, as shown in
Of course, in some implementations the destination nodes may be capable of reordering packets to the original sequence to prevent the foregoing issue. Packet reordering can be achieved using, for instance, a well-known technique of adding PSNs (Packet Sequence Numbers) to the packets in a flow, storing the received packets, and using a bit-map reorder window and pointer to track the PSNs and read the packets as stored in the correct order.
One issue with using this technique, however, is that the destination node must be capable of absorbing sufficient out-of-order packets to prevent mis-ordering. This requires careful design considerations involving multiple resources, including: a packet memory pool capable of holding all received packets for both ordered and misordered flows; a sufficient number of reorder windows (i.e. the number of incast flow sources that can be processed at once); and a sufficiently sized reorder window (which limits the out-of-order degree for a single reorderable flow). Exceeding the limits of any of these resources will cause delays or packet mis-ordering.
U.S. Pat. Nos. 10,142,219 and 10,693,767 to Rockport Networks Inc., the disclosures of which are incorporated herein by reference, disclose methods of sending packets in a direct interconnect network from a source node S to a destination node D over multiple diverse paths. The packets are divided into flits, which may be sent over the network links using wormhole switching techniques, and may require re-ordering at the destination node D. More particularly, one of the disclosed methods comprises discovering all nodes and all output ports on each node in a network topology; including the discovered nodes and output ports in the network topology in a topology database in order to allow the nodes and ports to be included in shortest or disjoint path routing computations; calculating the shortest or disjoint paths from every output port on each node to every other node in the network topology based on those nodes and output ports contained in the topology database; generating a source routing database on each node containing the shortest or disjoint paths from every output port on each node to all other nodes in the network topology; receiving packets at the source node; sending the received packets to the output ports of the source node in a round robin, weighted round robin, random distribution or other calculated path selection process, whereby each of the received packets is thereafter segmented into flits at the output port of the source node and distributed along the shortest or disjoint path from the output port on the source node to the destination node using worm-hole switching, such that the packets are thereby distributed along alternate routes in the network topology; and re-assembling and re-ordering the packets at the destination node so that the packets accord with their original form and order.
The present invention seeks to improve upon the various techniques disclosed in U.S. Pat. Nos. 10,142,219 and 10,693,767 by providing methods of routing packets in a direct interconnect network that seek to provide one or more of the following advantages, namely: preventing packet mis-ordering when multipath flows to a single destination would exceed that destination’s reordering window resources; detecting lost packets without waiting for long timeouts; and dynamically avoiding excessive skew between paths to control and reduce reorder window sizes and packet storage requirements.
The techniques disclosed herein are intended to minimize the total amount of additional metadata required to be passed with the packets and do not require the use of timestamps or network synchronization techniques to be employed, while achieving packet loss detection and dynamic path distribution functionality.
In one embodiment, the present invention provides a method of routing a flow of packets from a source node to a destination node comprising the steps of: at the source node, determining if the flow of packets is eligible for distribution along multiple pathways between the source node and the destination node; if the flow of packets is not eligible for such distribution, then routing the entire flow of packets over only one pathway between the source node and the destination node, but if the flow of packets is eligible for such distribution, then commencing routing the flow of packets over only one pathway between the source node and the destination node, and including a request for multipath operation in metadata contained in a packet of the flow of packets; at the destination node, upon detection of a request for multipath operation in metadata contained in the packet of the flow of packets, determining if a reordering resource is available for use with multipath operation; if no reordering resource is available, then receiving the flow of packets over only one pathway between the source node and the destination node, but if a reordering resource is available, then allocating the reordering resource for the flow of packets, and sending a grant code in a control flit to the source node; at the source node, upon detection of the grant code in the control flit, distributing the flow of packets along multiple pathways between the source node and the destination node, whereby the destination node uses the available reordering resource to reorder packets from the flow of packets.
In another embodiment, the present invention provides a method of detecting packet loss without incurring timeout delays when routing packets in a flow of packets over multiple pathways from a source node to a destination node comprising the steps of: routing the packets in the flow of packets along multiple pathways from the source node to the destination node, wherein each packet comprises a packet sequence number denoting a sequential location of said packet within the flow of packets, and wherein the source node records the packet sequence number for each packet sent on each pathway within the multiple pathways in order to track the sequence of packets sent on each pathway within the multiple pathways, and wherein each packet further comprises a previous packet sequence number denoting the packet sequence number of an immediately prior packet sent on a same pathway within the multiple pathways; for each packet that arrives at the destination node, setting a bitmap bit within a window bitmap that corresponds to the packet sequence number of said packet, and determining if a bitmap bit within the window bitmap that corresponds to the previous packet sequence number has been set; and if the bitmap bit corresponding to the previous packet sequence number has been set, then normal packet processing proceeds, but if the bitmap bit corresponding to the previous packet sequence number has not been set, then the packet that corresponds to the previous packet sequence number was lost and cannot be retrieved, and packet processing proceeds without incurring a timeout.
In yet another embodiment, the present invention provides a method of dynamically avoiding slower paths when routing packets between a source node and a destination node along multiple pathways, said method comprising: commencing routing the packets from the source node to the destination node along multiple pathways, and including metadata with the packets describing the routing distribution of the packets along the multiple pathways, said metadata comprising a number of packets sent on each pathway within the multiple pathways; monitoring relative packet skew between the multiple pathways at the destination node, wherein the destination node counts the number of packets that arrive on each pathway within the multiple pathways, compares the number of packets that arrive on each pathway to the number of packets sent on each pathway as per the metadata, and determining path skew status based on this comparison; using a backwards multipath control flit mechanism to send the path skew status from the destination node to the source node; and implementing a weighted path distribution mechanism at the source node to dynamically reduce the use of or avoid any skewed pathways in the multiple pathways when routing packets from the source node to the destination node along multiple pathways.
In yet a further embodiment, the present invention provides a method of avoiding overflow of a destination node reorder window when routing packets between a source node and a destination node along multiple pathways in a network, said method comprising: attaching a packet sequence number (PSN) to each packet at the source node as metadata and maintaining said metadata when routing each such packet through the network until each such packet is read from a receive packet buffer memory at the destination node; using the PSN of each such packet read from the receive packet buffer memory at the destination node as a credit PSN (CPSN) and sending said CPSN from the destination node to the source node as control flit metadata; at the source node, comparing the CPSN to a latest PSN for packets sent from the source node to the destination node, and using the difference between the CPSN and such latest PSN as a measure of a number of packets queued in the source node, in-flight in the network, or queued in the destination node reorder window; and, if the difference between the CPSN and such latest PSN exceeds a programmable limit, halting the routing of packets along multiple pathways from the source node to the destination node to prevent overflow of the destination node reorder window.
The embodiments of the invention will now be described, by way of example, with reference to the accompanying drawings in which:
The present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which preferred embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Those of ordinary skill in the art realize that the following descriptions of the embodiments of the present invention are illustrative and are not intended to be limiting in any way. Other embodiments of the present invention will readily suggest themselves to such skilled persons having the benefit of this disclosure.
Although the following detailed description contains many specifics for the purposes of illustration, anyone of ordinary skill in the art will appreciate that many variations and alterations to the following details are within the scope of the invention. Accordingly, the following embodiments of the invention are set forth without any loss of generality to, and without imposing limitations upon, the claimed invention.
The present invention seeks to address one or more issues that arise with the use of multipath flows between source and destination nodes, particularly when packet reordering is necessitated by the use of multipath flows.
In one embodiment, the present invention provides methods to improve the implementation of multipaths flows. One such method can be described with reference to
Further details of this functionality are provided below. Firstly, however, it must be noted that the required multipath metadata transferred with a packet between source and destination nodes includes, but may not be limited to, the following:
A first issue that often arises in multipath systems relates to re-ordering window allocation. In particular, there are generally a finite number of reordering window data structures available at any destination node. As such, if the number of multipath flows requiring packet reordering exceeds the available resources, then packets may be dropped or mis-ordered.
In order to prevent this from happening, in one embodiment the present invention may provide the functionality to dynamically connect source nodes to destination reorder windows. To invoke this method, a source node must firstly be capable of identifying which flows will be treated as multipath-capable, based on selectable criteria such as Class of Service, priority, protocol type, or other applicable criteria. With such capability, the source node may choose to identify any or all flows as being multipath-capable. For example, all RoCEv2 flows might be designated as multipath, while other TCP control flows might be designated to use a single path.
In operation, with this functionality, the skilled person can seek to ensure that adequate reorder window data structures are available for all multipath flows. In particular, as a general overview, when a source node S receives packets for a flow that it selects as being eligible to distribute over multiple paths, it will at first send the packets over only one selected path, while requesting multipath operation via optional metadata contained in the Head Flit header, for instance. When the destination node D receives the first packet of the flow, and sees a metadata request for multipath operation, the destination node D will allocate a reordering window (only if such resource is available) and send back a window grant code in a control flit to the source node S. This will assist in ensuring that adequate reordering window data structures are available to reorder the packet flow if necessary. Only when the source node S receives the window grant code in a control flit will it actually commence multipath distribution.
The source node must be capable of maintaining a state for each active flow to each destination. In this respect, for each flow, the source node’s path distribution function may employ a Finite State Machine using the states IDLE, REQ, and GRANTED, as shown in
The destination node’s Reorder Manager (ROM) function must also be capable of maintaining a limited number of reorder windows which can be dynamically assigned to specific source node multipath flows. For each reorder window various data structures and state information will be maintained. The reorder windows may be assigned based on active requests or based on other configured criteria. Windows may be released due to inactivity using a timeout mechanism, or any other criteria such as volume of traffic, or under network management control.
In this respect, for each flow, the destination node’s Reorder Manager (ROM) function may employ a Finite State Machine using the states IDLE, ACTIVE, RELEASING, and RESTARTING as shown in
Since reorder windows are a finite resource, the ROM may release the reorder window so that it may be reused by other requesting resources, based on any criteria such as traffic volume or timeouts. On entering the RELEASING state, a release message will be sent, and after a window free timeout the window will be freed. This two-stage timeout prevents mis-ordering in the case where the source node starts to send packets as the release message is being sent.
Some network errors such as the failure of intermediate links or nodes used for multipath paths may cause excessive packet loss and abnormal reordering conditions. To recover from any out of bounds events the ROM can move to the RESTARTING state. This will signal a restart message which will force the MPP back to its IDLE state to restart the process. While in the RESTARTING state the ROM will send all received packets in order but will not wait for any missing packets within the reorder window. When the ROM receives the first packet for the flow marked again as REQ, and this reaches the head of the window, the ROM FSM will return to the ACTIVE state.
A preferred implementation of the ROM packet reordering method for a reorder window is explained with reference to
A second issue that arises when packet reordering is necessitated by the use of multipath flows relates to detection of packet loss (i.e. packets failing to reach the destination node D), causing timeouts. In any optical network there is the possibility of packet loss due to optical bit error rates or network congestion, for instance. The problem is exacerbated when re-ordering is needed for multipath flows, because if a packet never arrives, at some point the packet must be declared lost, the reordering function may have to be terminated depending on the application (particularly if multiple packets have been lost), and data transmission must otherwise continue.
A well-known technique to assist in overcoming problems associated with lost packets is to use a timeout mechanism. However, the disadvantage of timeouts is that the reorder process may be stalled for an unacceptable period of time, thereby requiring a large reorder window and packet memory.
Therefore, in another embodiment, the present invention provides a method of detecting packet loss on multipath flows that avoids using timeouts in cases of isolated lost packets. The method assumes that on a given path, the packets from a flow will be in order and cannot pass each other (i.e. there would only be mis-ordered packets between flows sent over different pathways). Thus, if a loss of sequence can be detected on a particular path, it is clear that a packet has been lost and there is no need to use a timeout to wait for the lost packet. With this assumption, the method, in general, is implemented as follows:
To implement the above described PPSN mechanism the source node S will need to keep track of the PSN values sent to each path so that it can populate the PPSN field the next time the same path is selected. Since the PSN space will be of finite size, and a given path may not be selected for a period of time in the path selection function, due to path weighting, path recalculations, or for other reasons, then the PSN space may wrap around, thus making the stored PSN-per-path state invalid. This case may be detected by comparing the stored PSN-per-path values with the current PSN and invalidating the entries. The invalid PPSN is indicated in the metadata PPSN valid flag sent with the PPSN to the destination node D.
A further issue that arises when packet reordering is necessitated by the use of multipath flows relates to congestion. Congestion at intermediate nodes may cause rapid backup of source and destination packet buffering due to the nature of worm-hole routing in a direct interconnect network. If a packet flow is distributed over multiple paths and some paths are more congested than others resulting in differences in throughput, then the packets on the faster paths will accumulate in the destination buffers and the packets assigned to the slower paths may back up into the source buffers. It is thus desirable to dynamically avoid slower paths when distributing packets over multiple paths in order to avoid congestion. To make best use of the available bandwidth the paths should be used in accordance with their effective throughput.
The destination node is also preferably capable of detecting any skew between paths and signaling this information back to the source node. In order to have such capability, the destination node is capable of counting how many packets have arrived on each path by using a Path Index field in the flit header, as well as knowing how many packets were sent on each path (in order to be able to detect path skew). Without this capability, if the source node is dynamically avoiding congested paths, then the destination node cannot know on which path a packet(s) it is waiting for will arrive. As such, the present invention may preferably involve a method that includes the following functionality: a method for the source node to send additional metadata describing the distribution of packets; a method for the destination node to monitor the relative packet skew on paths and to accordingly determine path skew status; a backwards Multipath Control Flit (MCF) mechanism capable of passing the path skew status to the source node; and a method for the source node to implement a weighted path distribution mechanism to dynamically reduce the use of, or avoid, any excessively skewed paths.
The required multipath metadata transferred between the destination node D and the source node S in the MCF includes, but is not limited to, the following:
To control the total number of packets in-flight or occupying a reorder window, the destination node will preferably send the CPSN values to the source node, where it can limit the number of packets sent based on the difference between its current PSN value and the CPSN received from the destination node.
In one embodiment, and with reference to steps (2) to (16) as shown in
The Multipath Control Flits are sent immediately when a new window is being allocated, and then periodically to return the current skew counter and CPSN values. One optimization is to perform a lookup for the shortest available path to the original source node, to speed up the signaling process. A caching mechanism may also be employed to reduce the number of path lookups required when the path status is unchanged.
Also shown in
The following Table shows some example calculations using 8-bit fractional math, where the target weight in each row is recalculated in the next row using the equations above, depending on whether the received skew value is below or above the MIN_SKEW_THR. For clarity, the values are shown in both binary and as the number of 1/32 fractions. Using the constant values:
The source node packet distribution function may use any well-known technique such as a weighted round-robin to distribute the packets over the paths using the per-path target weights.
All the data structures such as reorder window size, number of paths in use, MCF transmission frequency, etc., and the configuration parameters described above must be adjusted appropriately based on the system parameters such as the number of hops in each path, the line rate on each link, and the ranges and distribution of packet sizes.
In terms of deployment, in one embodiment the methods described herein may be used in association with a direct interconnect network, such as, for example, those implemented in accordance with U.S. Pat. Nos. 9,965,429 and 10,303,640 to Rockport Networks Inc., the disclosures of which are incorporated in their entirety herein by reference. U.S. Pat. Nos. 9,965,429 and 10,303,640 describe systems that provide for the easy deployment of direct interconnect network topologies and disclose a novel method for managing the wiring and growth of direct interconnect networks implemented on torus or higher radix interconnect structures.
The systems of U.S. Pat. Nos. 9,965,429 and 10,303,640 involve the use of a passive patch panel having connectors that are internally interconnected (e.g. in a mesh) within the passive patch panel. In order to provide the ability to easily grow the network structure, the connectors are initially populated by interconnect plugs to initially close the ring connections. By simply removing and replacing an interconnect plug with a connection to a node, the node is discovered and added to the network structure. If a person skilled in the art of network architecture desired to interconnect all the nodes in such a passive patch panel at once, there are no restrictions - the nodes can be added in random fashion. This approach greatly simplifies deployment, as nodes are added/connected to connectors without any special connectivity rules, and the integrity of the torus structure is maintained.
In another preferred embodiment, the methods disclosed herein may be used in association with devices that interconnect nodes in a direct interconnect network (i.e. shuffles) as described in PCT Publication No. WO 2022/096927 A1 to Rockport Networks Inc., the disclosure of which is incorporated in its entirety herein by reference. The shuffles described therein are novel optical interconnect devices capable of providing the direct interconnection of nodes in various topologies as desired (including torus, dragonfly, slim fly, and other higher radix topologies for instance) by connecting fiber paths from a node(s) to fiber paths of other node(s) within an enclosure to create optical channels between the nodes. This assists in optimizing networks by moving the switching function to the endpoints. The optical paths in the shuffles of PCT Publication No. WO 2022/096927 A1 are pre-determined to create the direct interconnect structure of choice, and the internal connections are preferably optimized such that when nodes are connected to a shuffle in a predetermined manner an optimal direct interconnect network is created during build-out.
The nodes themselves may potentially be any number of different devices, including but not limited to processing units, memory modules, I/O modules, PCIe cards, network interface cards (NICs), PCs, laptops, mobile phones, servers (e.g. application servers, database servers, file servers, game servers, web servers, etc.), or any other device that is capable of creating, receiving, or transmitting information over a network. As an example, in one preferred embodiment, the node may be a network card, such as a Rockport RO6100 Network Card, as described in PCT Publication No. WO 2022/096927 A1. Such network cards are installed in servers, but use no server resources (CPU, memory, and storage) other than power, and appear to be an industry-standard Ethernet NIC to the Linux operating system. Each Rockport RO6100 Network Card supports an embedded 400 Gbps switch (twelve 25 Gbps network links; 100 Gbps host bandwidth) and contains software that implements the switchless network over the shuffle topology (see e.g. the methods of routing packets in U.S. Patent Nos. 10,142,219 and 10,693,767 to Rockport Networks Inc., the disclosures of which are incorporated in their entirety herein by reference).
Although specific embodiments of the invention have been described, it will be apparent to one skilled in the art that variations and modifications to the embodiments may be made within the scope of the following claims.
Some of the illustrative aspects of the present invention may be advantageous in solving the problems herein described and other problems not discussed which are discoverable by a skilled artisan.
While the above description contains much specificity, these should not be construed as limitations on the scope of any embodiment, but as exemplifications of the presented embodiments thereof. Many other ramifications and variations are possible within the teachings of the various embodiments. While the invention has been described with reference to exemplary embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted for elements thereof without departing from the scope of the invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the invention without departing from the essential scope thereof. Therefore, it is intended that the invention not be limited to the particular embodiment disclosed as the best or only mode contemplated for carrying out this invention, but that the invention will include all embodiments falling within the scope of the appended claims. Also, in the drawings and the description, there have been disclosed exemplary embodiments of the invention and, although specific terms may have been employed, they are unless otherwise stated used in a generic and descriptive sense only and not for purposes of limitation, the scope of the invention therefore not being so limited. Moreover, the use of the terms first, second, etc. do not denote any order or importance, but rather the terms first, second, etc. are used to distinguish one element from another. Furthermore, the use of the terms a, an, etc. do not denote a limitation of quantity, but rather denote the presence of at least one of the referenced item.
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/IB2022/000317 | 6/8/2022 | WO |
Number | Date | Country | |
---|---|---|---|
63208774 | Jun 2021 | US |