Patent Application
20060067228
Networks enable computers and other devices to communicate. For example, networks can carry data representing video, audio, e-mail, and so forth. Typically, data sent across a network is divided into smaller messages carried in packets. By analogy, a packet is much like an envelope you drop in a mailbox. A packet typically includes “payload” and a “header”. The packet's “payload” is analogous to the letter inside the envelope. The packet's “header” is much like the information written on the envelope itself. The header can include information to help network devices handle the packet appropriately.
A number of network protocols cooperate to handle the complexity of network communication. For example, a protocol known as Transmission Control Protocol (TCP) provides “connection” services that enable remote applications to communicate. That is, TCP provides applications with simple mechanisms for establishing a connection and transferring data across a network. Behind the scenes, TCP handles a variety of communication issues such as data retransmission, adapting to network traffic congestion, and so forth.
To provide these services, TCP operates on packets known as segments. Generally, a TCP segment travels across a network within (“encapsulated” by) a larger packet such as an Internet Protocol (IP) datagram. Frequently, an IP datagram is further encapsulated by an even larger packet such as an Ethernet frame. The payload of a TCP segment carries a portion of a stream of data sent across a network by an application. A receiver can restore the original stream of data by reassembling the received segments. To permit reassembly and acknowledgment (ACK) of received data back to the sender, TCP associates a sequence number with each payload byte.
Many computer systems and other devices feature host processors (e.g., general purpose Central Processing Units (CPUs)) that handle a wide variety of computing tasks. Often these tasks include handling network traffic such as TCP/IP connections.
The increases in network traffic and connection speeds have increased the burden of packet processing on host systems. In short, more packets need to be processed in less time. Fortunately, processor speeds have continued to increase, partially absorbing these increased demands. Improvements in the speed of memory, however, have generally failed to keep pace. Each memory access that occurs during packet processing represents a potential delay as the processor awaits completion of the memory operation. Many network protocol implementations access memory a number of times for each packet. For example, a typical TCP/IP implementation performs a number of memory operations for each received packet including copying payload data to an application buffer, looking up connection related data, and so forth.
As described above, each memory operation that occurs during packet processing represents a potential delay. As an example, in many current systems, after receiving a packet, a network interface controller (a.k.a. a network adaptor) performs a Direct Memory Access (DMA) to place the packet in memory. Protocol software (e.g., a Transmission Control Protocol/Internet Protocol (TCP/IP) stack) then copies the data from the place where the network interface controller deposited it to where an application requested placement. This copy operation can potentially involve thousands of bytes and may represent a significant use of computing resources and consume considerable packet processing time.
Page-flipping can be used in packet processing by having a network interface controller deposit packet data for packets of a given flow into the same page and then, instead of copying this data to an application specified destination virtual address, simply remapping the virtual page including the destination virtual address to the page storing the network interface controller deposited packet data. To illustrate,
The sample implementation shown in
As shown in
As shown in
As shown in
As shown, the network interface controller 100 can include circuitry 306 to handle packets based on their flow and circuitry 312 to handle packets on an individual basis. A given packet may be routed to circuitry 312 by the MAC circuitry 304, for example, if the packet is not a TCP/IP packet. The circuitry 312 can then access a packet descriptor provided by driver software executing on the processor that identifies where to store the packet data in memory. The circuitry 312 can then cause a Direct Memory Access (DMA) controller 308 to transfer the packet data to memory 102. The controller 100 can signal an interrupt to the processor 120 that initiates processing of the DMA-ed descriptor/packet data by the protocol stack. Typically, this will scatter the packets of a given flow across many different pages of memory.
In contrast to the operation of circuitry 312, flow-based packet processing circuitry 306 can perform operations described in conjunction with
Potentially, the network interface controller 100 may only perform page-flipping for a limited subset of on-going flows handled by the controller 100. Thus, circuitry 306 can store data 112 (e.g., in a Content-Addressable Memory (CAM)) that identifies flows to be handled using page-flipping. Absence (e.g., a CAM miss for the flow identifier) from the flow list may indicate that the packet should not be handled using page-flipping. The data 112 associated with a given flow may include the flow identifier (e.g., a hash of a TCP/IP tuple) and the next address within the flow's current page to append packet data. The flows listed may be established, for example, by a driver or dynamically managed by the controller 100, for example, based on currently on-going flows. As described below, flows may be ejected from the list based on a variety of criteria. For example, flows may be removed in favor of more recently active flows based on a Least Recently Used (LRU) scheme. Potentially, the circuitry 306 may maintain a list of flows (not shown) not to be processed using page-flipping instead of/in addition to data 112. This can prevent a flow from thrashing back and forth between page-flipping/no-page-flipping status.
For flows included in the page-flipping list, the controller 100 may attempt to determine 212 whether to remove the flow from the list. For example, if a packet is received out-of-order, the controller 100 may instead use circuitry 312 to deposit packet data in pages in memory and allow the protocol stack to perform reassembly. The test performed to determine whether a packet is out-of-order may be a determination of whether a previous sequence number received for a flow is greater than the sequence number of the received packet.
Other packet characteristics may cause a flow to be removed from the flow list. For example, if the received packet terminates a flow (e.g., a TCP FIN or RST packet), if the packet reduces the TCP window of the flow, or if the packet identifies itself as a fragmented IP packet. Additionally, the controller 100 may remove a flow from the list if the flow features too many smaller sized packets (e.g., a number of packets that fall beneath some threshold or an average packet size falls below a threshold).
The controller 100 may also periodically gauge flow activity and remove inactive flows from the flow list such as flows not having any received packets or not advancing a TCP receive window after some interval. Additionally, the controller 100 may remove flows, for example, if an out-bound packet being transmitted through the controller 100 closes a flow (e.g., a FIN or RST packet).
If a flow is removed the controller 100 can signal the driver to indicate 214 data previously placed in a page associated with the flow. Once removed from the flow list 216, subsequent packets belonging to the flow would be handled using descriptors identifying locations in memory instead of concentrating packet data from a flow into the flow's page(s).
As shown in
The implementations describe above are merely exemplary and a wide variety of variations are possible. For example, instead of being a separate component, the controller may be integrated into a chipset or a processor. The techniques may be implemented in a variety of architectures including processors and network devices having designs other than those shown. The term packet can apply to IP (Internet Protocol) datagrams, TCP (Transmission Control Protocol) segments, ATM (Asynchronous Transfer Mode) cells, Ethernet frames, among other protocol data units. Additionally, the above often referred to packet data instead of simply a packet. This reflects that a controller, or other component, may remove and/or add data to a packet as the packet data travels along a receive or transmit path.
The term circuitry as used herein includes hardwired circuitry, digital circuitry, analog circuitry, programmable circuitry, and so forth. The programmable circuitry may operate on executable instructions disposed on an article of manufacture. For example, the instructions may be disposed on a Read-Only-Memory (ROM) such as a Programmable Read-Only-Memory (PROM)) or other medium such as a Compact Disk (CD) and other volatile or non-volatile storage.
Other embodiments are within the scope of the following claims.
