Processors, and specifically network processors, route packets to and from destinations on a network. In doing so, the processors can perform direct memory access of packets. Certain processors can route the packets to various internal, and in some cases external, functions.
In an embodiment, a method includes, in response to detecting available memory of a destination node of a packet flow of one or more nodes to the destination node being below a particular threshold, marking the destination node as being in a backpressure state. The destination node in the backpressure state, sends a signal indicating a condition of packet backpressure to the one or more nodes of the packet flow, and initiates a timer for a particular time period. The method further marks, at the end of the particular time period, the destination node as being in a bad actor state if the available memory is below the particular threshold, and as being in a good actor state if the memory is above the particular threshold. The method further, in response to marking the destination node as being in a bad actor state, sends a signal to the one or more nodes of the packet flow causing the one or more nodes to drop packets directed to the destination node.
In an embodiment, packet flow is the distribution of packets from a first node to a destination node, optionally via intermediary nodes. Nodes along the packet flow from the first node to a node before the destination node can be considered upstream from the destination node in the packet flow. Backpressure is applied from the destination node upstream in the packet flow.
In an embodiment, in response to marking the destination node as being in a good actor state, the method sends a signal to the one or more nodes of the packet flow causing the one or more nodes to continue sending packets to the destination node.
In an embodiment, a method marking the destination node as being in the bad actor state further includes throwing an interrupt to other entities related to the packet flow.
In an embodiment, the destination node is a virtual function (VF) ring.
In an embodiment, the threshold is a watermark.
In an embodiment, the method further comprises, at startup, initiating the destination node as being in the bad actor state.
In an embodiment, sending the signal (e.g., applying backpressure) includes sending a signal to the one or more nodes of the packet flow indicating the destination node is not accepting new packets.
In an embodiment, a system includes a processor configured to, in response to detecting available memory of a destination node of a packet flow of one or more nodes to the destination node being below a particular threshold, mark the destination node as being in a backpressure state. The destination node, in the backpressure state sends a signal indicating a condition of packet backpressure to the one or more nodes of the packet flow, and initiating a timer for a particular time period. The processor is further configured to mark, at the end of the particular time period, the destination node as being in a bad actor state if the available memory is below the particular threshold, and as being in a good actor state if the memory is above the particular threshold. In response to marking the destination node as being in a bad actor state, the processor sends a signal to the one or more nodes of the packet flow causing the one or more nodes to drop packets directed to the destination node.
The foregoing will be apparent from the following more particular description of example embodiments, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating embodiments.
A description of example embodiments follows.
The SDP 102 has 512 physical SDP input rings that are paired with SDP output rings. A person having ordinary skill in the art can recognize that the exact number of rings can vary, however. A PCIe host or other external device may submit instructions/packets to the SDP 102, which is a way to feed packets or commands to the chip on which the SDP resides.
The SDP 102 can further stop instruction fetches for a physical ring when buffer counts are low. The SDP 102 does not fetch new instructions for a physical ring when the network interface controller (NIC) is exerting backpressure for its respective pair.
In processors that handle packets and packet flows, packet flow management to a particular destination node can be a problem. For example, the destination node can run out of buffer space or memory, which can cause packets to be dropped. To prevent packet dropping, backpressure can be applied to upstream flows when a given function (e.g., virtual function (VF) or physical function (PF)) cannot process packets to prevent overflowing a destination node function with packets. Applying backpressure sends a signal to all upstream devices to stop sending packets to the destination node, however, already sent packets to continue along the packet flow to the destination node. However, if the given function cannot process packets for long periods of time, applying back pressure can congest upstream systems and slow other flows unrelated to the given function. In other words, applying backpressure to systems when resources are limited allows more control in software but can cause congestion to upstream systems.
In an embodiment of the present disclosure, a multi-state framework can solve this congestion problem. In an embodiment, all destination node functions (e.g., VFs and PFs) are assigned a good actor state, bad actor state, and a backpressure state. Other network elements change their behavior with respect to the destination node based on the assigned state.
Limiting backpressure to only destinations node that are behaving in a desirable manner prevents such congestion. Embodiments of the present disclosure identify functions that are not processing packets in a timely manner and prevents them from applying backpressure. Packets sent to bad actor state destinations are dropped, which prevents such packets from filling up local memory.
For example, for destinations in the bad actor state, instead of backpressure, hardware drops all packets destined for the destination node. For destinations in the good actor state, network elements send packets to the destination node as in normal operation. For destinations in the backpressure state, network elements act as if backpressure is applied by not sending any new packets to the destination node.
At startup, all destinations initialize in bad actor states. After sufficient memory buffers are allocated to send packets, the destination node changes to the good actor state. If a destination node does not have sufficient buffers to send packets, it is placed in the backpressure state and a timer is started for that function. When a destination node starts to allow packet traffic to flow again, the timer stops and resets to zero. If a destination node continues not accepting packets and its timer has reached a programmable threshold, the destination node is moved to the bad actor state. Destination nodes can also move from good actor states to bad actor states if a function is disabled or reset, indicating it cannot receive packets. A destination node can also move to bad actor state if a packet is received and there are no buffers to send it to prevent head of line blocking and allow the packet to be dropped.
A configurable watermark level 206 indicates number of buffers that should be available for ideal performance.
In one embodiment, the configurable watermark level 206 can indicate a number of buffers to remain empty. A doorbell or other process can determine the number of filled buffers and compare the number of filled buffers to the watermark. In another embodiment, the doorbell or other process can determine the number of empty buffers and compare the number of empty buffers to the watermark. Such a comparison can be performed either in hardware or by a processor.
In
The node, periodically (e.g., after a set number of clock cycles), checks its buffer levels. If the buffer levels are below the watermark level, indicating there is enough memory, the node transitions to a good actor state 406. In the good actor state, the node can receive packets normally with no backpressure or dropped packets.
From the good actor state 406, the used memory of the buffer can rise above the watermark as packets are received. In response, the node can transition to a backpressure state 408. In the backpressure state 408, a signal is sent to all upstream nodes to stop sending new packets. In addition, upon entering the backpressure state, a timer begins to count time or cycles for a configurable amount of time. If, at the end of the period, the used memory remains above the watermark, the node transitions to the bad actor state 404, where backpressure is no longer applied, hardware can drop packets, and no more packets are sent to the destination node. However, if, after the period of time has elapsed, the used memory amount falls below the watermark, the node returns to the good actor state 406.
A person having ordinary skill in the art can recognize that, in other embodiments, events can trigger the good actor state 406 transitioning to the bad actor state 404, such as disabling the node/ring/function, user shutdown of the node/ring/function, an error condition, or no buffers being available.
Upon detecting that the used memory falls below the watermark, the node transitions to a good actor state 506. When used memory rises above the watermark, the node remains in the good actor state 506 but applies backpressure, as described above, and starts a timer. If, after the timer expires, the used memory remains above the watermark, the node enters the bad actor state 504.
The teachings of all patents, published applications and references cited herein are incorporated by reference in their entirety.
While example embodiments have been particularly shown and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the embodiments encompassed by the appended claims.
This application is a continuation of U.S. application Ser. No. 16/029,031, filed Jul. 6, 2018, now U.S. Pat. No. 10,721,172. The entire teachings of the above application are incorporated herein by reference.
Number | Name | Date | Kind |
---|---|---|---|
5745778 | Alfieri | Apr 1998 | A |
6189074 | Pedneau | Feb 2001 | B1 |
6253262 | Rozario | Jun 2001 | B1 |
6289369 | Sundaresan | Sep 2001 | B1 |
6356989 | Hays | Mar 2002 | B1 |
6496847 | Bugnion | Dec 2002 | B1 |
6861865 | Carlson | Mar 2005 | B1 |
6862694 | Tormey | Mar 2005 | B1 |
6954770 | Carlson | Oct 2005 | B1 |
7035889 | Carlson | Apr 2006 | B1 |
7076059 | Kiszeley | Jul 2006 | B1 |
7120117 | Liu | Oct 2006 | B1 |
7205785 | Carlson | Apr 2007 | B1 |
7209531 | Katz | Apr 2007 | B1 |
7240203 | Kessler | Jul 2007 | B2 |
7260217 | Carlson | Aug 2007 | B1 |
7275249 | Miller | Sep 2007 | B1 |
7305567 | Hussain | Dec 2007 | B1 |
7310722 | Moy | Dec 2007 | B2 |
7337314 | Hussain | Feb 2008 | B2 |
7372857 | Kappler | May 2008 | B1 |
7398386 | Kessler | Jul 2008 | B2 |
7421533 | Zimmer | Sep 2008 | B2 |
7657933 | Hussain | Feb 2010 | B2 |
7661130 | Hussain | Feb 2010 | B2 |
7814310 | Bouchard | Oct 2010 | B2 |
7911960 | Aydemir | Mar 2011 | B1 |
8156495 | Chew | Apr 2012 | B2 |
8424014 | Auernhammer | Apr 2013 | B2 |
8504750 | Sonksen | Aug 2013 | B1 |
8881150 | Sawa | Nov 2014 | B2 |
8892962 | Iwanaga | Nov 2014 | B2 |
9444751 | Mresaini | Sep 2016 | B1 |
10721172 | Kessler | Jul 2020 | B2 |
20040216101 | Burky | Oct 2004 | A1 |
20040268105 | Michaelis | Dec 2004 | A1 |
20050235123 | Zimmer | Oct 2005 | A1 |
20060176810 | Kekki | Aug 2006 | A1 |
20060288189 | Seth | Dec 2006 | A1 |
20070220203 | Murase | Sep 2007 | A1 |
20080013715 | Feghali | Jan 2008 | A1 |
20080074433 | Jiao | Mar 2008 | A1 |
20080077909 | Collins | Mar 2008 | A1 |
20080133709 | Aloni | Jun 2008 | A1 |
20080320016 | Singh | Dec 2008 | A1 |
20090024804 | Wheeler | Jan 2009 | A1 |
20090070768 | Choudhury | Mar 2009 | A1 |
20090096525 | Staszewski | Apr 2009 | A1 |
20090119684 | Mahalingam | May 2009 | A1 |
20090249094 | Marshall | Oct 2009 | A1 |
20090300606 | Miller | Dec 2009 | A1 |
20100082603 | Krompass | Apr 2010 | A1 |
20100138829 | Hanquez | Jun 2010 | A1 |
20100205603 | Marten | Aug 2010 | A1 |
20100275199 | Smith | Oct 2010 | A1 |
20100332212 | Finkelman | Dec 2010 | A1 |
20110161943 | Bellows | Jun 2011 | A1 |
20110314478 | Louise | Dec 2011 | A1 |
20120039169 | Susitaival | Feb 2012 | A1 |
20120052866 | Froehlich | Mar 2012 | A1 |
20120096192 | Tanaka | Apr 2012 | A1 |
20120179844 | Brownlow | Jul 2012 | A1 |
20120260257 | Accapadi | Oct 2012 | A1 |
20120327770 | Vijayasankar | Dec 2012 | A1 |
20130055254 | Avasthi | Feb 2013 | A1 |
20130097350 | Ansari | Apr 2013 | A1 |
20160261512 | Lautenschlaeger | Sep 2016 | A1 |
20160344636 | Elias | Nov 2016 | A1 |
20180278536 | Haramaty | Sep 2018 | A1 |
20200014629 | Kessler et al. | Jan 2020 | A1 |
Entry |
---|
“Single Root I/O Virtualization and Sharing Specification Revision 1.01,” PCI-SIG®, pp. 1-100 (Jan. 20, 2010). |
Number | Date | Country | |
---|---|---|---|
20200336433 A1 | Oct 2020 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 16029031 | Jul 2018 | US |
Child | 16921884 | US |