Patent Application
20060050693
This patent application relates to building data packets for an Advanced Switching (AS) fabric.
PCI (Peripheral Component Interconnect) Express is a serialized I/O interconnect standard developed to meet the increasing bandwidth needs of the next generation of computer systems. PCI Express was designed to be fully compatible with the widely used PCI local bus standard. PCI is beginning to hit the limits of its capabilities, and while extensions to the PCI standard have been developed to support higher bandwidths and faster clock speeds, these extensions may be insufficient to meet the rapidly increasing bandwidth demands of PCs in the near future. With its high-speed and scalable serial architecture, PCI Express may be an attractive option for use with, or as a possible replacement for, PCI in computer systems. [The PCI Express architecture is described in the PCI Express Base Architecture Specification, Revision 1.0 (Initial release Jul. 22, 2002), which is available through the PCI-SIG (PCI-Special Interest Group) (http://www.pcisig.com)].
AS is an extension to the PCI Express architecture. AS utilizes a packet-based transaction layer protocol that operates over the PCI Express physical and data link layers. The AS architecture provides a number of features common to multi-host, peer-to-peer communication devices such as blade servers, clusters, storage arrays, telecom routers, and switches. These features include support for flexible topologies, packet routing, congestion management (e.g., credit-based flow control), fabric redundancy, and fail-over mechanisms. The AS architecture is described in the Advanced Switching Core Architecture Specification, Revision 1.0 (December 2003), which is available through the ASI-SIG (Advanced Switching Interconnect-SIG) (http//:www.asi-sig.org).
Like reference numerals in different figures indicate like elements.
Generally speaking, a switching fabric is a combination of hardware and software that moves data coming into a network node out the correct port to a next network node. A switching fabric includes switching elements, e.g., individual devices in a network node, integrated circuits contained therein, and software that controls switching paths through the switch fabric.
In
As shown in
Referring to
The PI field in the AS route header specifies the format of the encapsulated packet. The PI field is inserted by the end node that originates the AS packet and is used by the end node that terminates the packet to correctly interpret the packet contents. The separation of routing information from the remainder of the packet enables an AS fabric to tunnel packets of any protocol.
PIs represent fabric management and application-level interfaces to AS fabric 11. Table 1 provides a list of PIs currently supported by the AS Specification.
PIs 0-7 are reserved for various fabric management tasks, and PIs 8-254 are application-level interfaces. As shown in Table 1, PI8 is used to tunnel or encapsulate native PCI Express. Other PIs may be used to tunnel various other protocols, e.g., Ethernet, Fibre Channel, ATM (Asynchronous Transfer Mode), InfiniBand®, and SLS (Simple Load Store). An advantage of an AS switch fabric is that a mixture of protocols may be simultaneously tunneled through a single, universal switch fabric making it a powerful and desirable feature for next generation modular applications such as media gateways, broadband access routers, and blade servers.
The AS architecture supports the establishment of direct end node-to-end node logical paths known as Virtual Channels (VCs). This enables a single AS fabric network to service multiple, independent logical interconnects simultaneously. Each VC interconnecting AS end nodes for control, management and data. Each VC provides its own queue so that blocking in one VC does not cause blocking in another. Since each VC has independent packet ordering requirements, each VC can be scheduled without dependencies on the other VCs.
The AS architecture defines three VC types: Bypass Capable Unicast (BVC); Ordered-Only Unicast (OVC); and Multicast (MVC). BVCs have bypass capability, which may be necessary for deadlock free tunneling of some, typically load/store, protocols. OVCs are single queue unicast VCs, which are suitable for message oriented “push” traffic. MVCs are single queue VCs for multicast “push” traffic.
The AS architecture provides a number of congestion management techniques, one of which is a credit-based flow control technique that ensures that packets are not lost due to congestion. Link partners in the network (e.g., an end node 14a and a switch element 12a) exchange flow control credit information to guarantee that the receiving end of a link has the capacity to accept packets. Flow control credits are computed on a VC-basis by the receiving end of the link and communicated to the transmitting end of the link. Typically, packets are transmitted only when there are enough credits available for a particular VC to carry the packet. Upon sending a packet, the transmitting end of the link debits its available credit account by an amount of flow control credits that reflects the packet size. As the receiving end of the link processes (e.g., forwards to an end node 14a) the received packet, space is made available on the corresponding VC and flow control credits are returned to the transmission end of the link. The transmission end of the link then adds the flow control credits to its credit account.
The AS architecture supports an AS Configuration Space in each AS device in the network. The AS Configuration Space is a storage area that includes fields that specify device characteristics, as well as fields used to control the AS device. The information is presented in the form of capability structures and other storage structures, such as tables and a set of registers. The information stored in the AS-native capability structures can be accessed through PI-4 packets, which are used for device management. In one embodiment of an AS fabric network, AS end node devices are restricted to read-only access of another AS device's AS native capability structures, with the exception of one or more AS end nodes that have been elected as fabric managers.
A fabric manager election process may be initiated by a variety of hardware or software mechanisms. A fabric manager is an AS end node that “owns” all of the AS devices, including itself, in the network. If multiple fabric managers, e.g., a primary fabric manager and a secondary fabric manager, are elected, then each fabric manager may own a subset of the AS devices in the network. Alternatively, the secondary fabric manager may declare ownership of the AS devices in the network upon a failure of the primary fabric manager, e.g., resulting from a fabric redundancy and fail-over mechanism.
Once a fabric manager declares ownership, it has privileged access to its AS devices' AS native capability structures. In other words, the fabric manager has read and write access to the AS native capability structures of all of the AS devices in the network, while the other AS devices are restricted to read-only access, unless granted write permission by the fabric manager.
AS fabrice 11 supports the simple load store (SLS) protocol. SLS is a protocol that allows one end node device, such as the fabric manager, to store, and access, data in another end node device's memory, including, but not limited to, the device's configuration space. Memory accesses that are executed via SLS may be direct, meaning that an accessing device need not go through a local controller or processor on an accessed device in order to get to the memory of the accessed device. SLS data packets are recognized by specific packet headers that are familiar to AS end node devices, and are passed directly to hardware on the end node devices, which performs the requested memory access(s).
End node device 14a uses direct memory access (DMA) technology to build data packets for transmission to AS fabric 11. DMA is a technique for transferring data from memory without passing the data through a central controller (e.g., a processor) on the device. Device 14a may be a work station, a personal computer, a server, a portable computing device, or any other type of intelligent device capable of executing instructions and connecting to AS fabric 11.
Device 14a includes a central processing unit (CPU) 24. CPU 24 may be a microprocessor, microcontroller, programmable logic, or the like, which is capable of executing instructions (e.g., a computer program) to perform one or more operations. Such instructions may be stored in system memory 25, which may be one or more hard drives or other internal or external memory devices connected to CPU 24 via one or more communications media, such as a bus 26. System memory 25 may include designated ring buffers 27, which together make up a queue, for use in transmitting data packets to, and receiving data packets from, AS fabric 11.
Device 14a also includes protocol interface (PI) engine 29. PI engine 29 may include one or more separate hardware devices, or may be implemented in software running on CPU 24. In this embodiment, PI engine 29 is implemented on a separate chip which communicates with CPU 24 via bus 26, and which may communicate with one or more PCI express devices (not shown) via PCI express bus(es) (also not shown).
PI engine 29 functions as CPU 24's interface to AS fabric 11. In this embodiment, PI engine 29 contains a DMA engine 30, a work manager engine 31, one or more acceleration engines 32, and an arbiter 100. Registers 34 are included in PI engine 29 for use by its various components, and may include one or more first-in first-out (FIFO) registers. Transmit registers 28 provide a “transmit” interface to advanced switching (AS) fabric 11. PI engine 29 also contains a response engine 102 which receives data packets from a “receive” interface (not shown) to AS fabric 11.
DMA engine 30 is a direct memory access engine, which retrieves descriptors from ring buffers 27, and which stores the descriptors in registers 34. As described below, descriptors are data structures that contain information used to build data packets. Work manager 31 is an engine that controls work flow among entities used to build data packets, including DMA engine 30 and acceleration engines 32. Work manager 31 also builds data packets for non-native AS protocols. Acceleration engines 32 are protocol-specific engines, which build data packets for predefined native AS protocols. The operation of these components of PI engine 29 is described below with respect to
Different descriptor formats are supported by device 14a. Examples of such descriptor formats include the “immediate” descriptor format, the “indirect” descriptor format, and the “packet-type” descriptor format. An immediate descriptor contains all data needed to build a data packet for transmission over the AS fabric, including the payload of the packet. An indirect descriptor contains all data needed to build a data packet, except for the packet's payload. The indirect descriptor format instead contains one or more addresses identifying the location, in ring buffers 27, of data for the payload. A packet-type descriptor identifies a section of memory that is to be extracted and transmitted as a data packet. The packet-type descriptor is not used to format a packet, but instead is simply used to extract data specified at defined memory addresses, and to transmit that data as a packet. In this embodiment, each descriptor is 32 bits (one “D-word”) wide and sixteen D-words long.
An example of an immediate descriptor 35 is shown in
An example of an indirect descriptor 47 is shown in
An example of a packet-type descriptor 52 is shown in
DMA engine 30 retrieves descriptors from ring buffers 27 for storage in registers 34. In this embodiment, there are eight registers capable of holding two descriptors each, and DMA engine 30 retrieves the descriptors in the order in which they were stored in the buffers, i.e., first-in, first-out.
Each of registers 34 includes one or more associated status bits. These status bits indicate whether a register contains zero, one or two descriptors. The status bits are set, either by DMA engine 30 or work manager 31 (described below). DMA engine 30 determines whether to store the descriptors based on the status bits of registers 34. More specifically, as described below, work manager 31 processes (i.e., “consumes”) descriptors from registers 34. Once a descriptor has been consumed from a register, work manager 31 resets the status bits associated with that register to indicate that the register is no longer full. DMA engine 30 examines (62) the status bits periodically to determine whether a register has room for a descriptor. If so, DMA engine 30 retrieves (63) a descriptor (or two) from the ring buffers and consults arbiter 100 as to whether DMA engine 30 can store the descriptor(s) in registers 30. As described below, arbiter 100 is a part of PI engine 29 that arbitrates access to registers 34 by DMA engine 30 and response engine 102 (also described below). If storage is approved by arbiter 100, DMA engine 30 stores that descriptor in an appropriate register. DMA engine 30 stores the descriptor in a register that is dedicated to the same virtual channel as the ring buffer from which the descriptor was retrieved. DMA engine 30 may also store a tag associated with each descriptor in registers 34. Use of this tag is described below.
Work manager 31 examines (64) the status bits of each register to determine whether a descriptor is available for processing. If a descriptor is available, work manager 31 retrieves (65) that descriptor and process the descriptor in the manner described below.
A priority level associated with each register may affect how the work manager retrieves descriptors from the registers. More specifically, each register may be assigned a priority level. The priority level indicates, to work manager 31, a number of descriptors to retrieve from a target register before retrieving descriptors from other registers. Circuitry (not shown), such as a counter, associated with each register maintains the priority level of each register. The circuitry stores a value that corresponds to the priority level of an associated register, e.g., a higher value indicates a higher priority level. Each time work manager 31 retrieves a descriptor from the target register, the circuitry increments a count, and the current value of the count is compared to the priority level value. So long as the count is less than or equal to the priority level value of a target register, work manager 31 continues to retrieve descriptors only from the target register. If no descriptors are available from the target register, work manager 31 may move on to another register, and retrieve descriptors from that other register until descriptors from the target register become available.
Work manager 31 examines (66) retrieved descriptors in order to determine a type of the descriptor. In particular, work manager 31 examines the ID bytes of each descriptor to determine the type of the descriptor. Since packet-type descriptors simply define “chunks” of data as a packet, packet-type descriptors do not contain acceleration control information (see
For immediate descriptors and indirect descriptors, work manager 31 also examines the descriptor to determine whether the descriptor is for a data packet having a protocol that is native to AS, such as SLS, or for packets that have a protocol that is non-native to AS, such as ATM. In particular, work manager 31 examines the acceleration control information of immediate descriptors and indirect descriptors.
If the acceleration control information indicates (69) that the descriptor is for a data packet having a protocol that is non-native to AS fabric 11, work manager 31 builds (71) one or more data packets from the descriptor.
If the descriptor is an immediate descriptor, work manager 31 builds a data packet using the descriptor. In particular, work manager 31 builds a packet header from D-words 41 and 42 (
If the descriptor is an indirect descriptor, work manager 31 builds a header for the data packet in the same manner as for an immediate descriptor. Work manager 31 builds a packet payload by retrieving a payload for the packet from address(es) 50 (
Work manager 31 stores data packets in transmit registers 28, from which the data packets are output to AS fabric 11.
Referring back to
Work manager 31 uses a tag system to keep track of packet processing with acceleration engines 32. As noted above, work manager 31 retrieves all necessary information/data it needs from registers 34, along with an associated tag. For native AS packets, work manager 31 instructs an acceleration engine 32 to build the packet (e.g., if the packet is SLS).
After building the packet header, acceleration engine 32 sends a payload fetch request to work manager 31 to request payload for the packet. Along with the request, the acceleration engine sends a copy of the tag that work manager 31 forwarded to acceleration engine 32. The returned tag, however, has been altered to instruct work manager 31 to retrieve payload for the packet, and to provide the payload to the acceleration engine for packet building.
When building the data packet, acceleration engine 32a issues a write back command to work manager 31. If the payload of the data packet is too big to be accommodated in a single packet, the write back command identifies the data that has been packetized by acceleration engine 32a. Specifically, the write back command specifies the ending address of the packetized data. Work manager 31 receives the write back command and determines whether all of the data in the original descriptor has been packetized (e.g., work manager 31 determines if the ending address in the write back command corresponds to the ending address of the total amount of data to be packetized). If all of the data has been packetized, work manager 31 sets the status bits of a corresponding register 34 to indicate that there is room for another descriptor. If all of the data in the original descriptor has not been packetized, work manager 31 instructs acceleration engine 32a to build another data packet using substantially the same packet header information as the previous data packet. Work manager 31 instructs acceleration engine 32a that the payload for this next packet is to start at the address at which the previous packet ended. The back-and-forth process between the acceleration engine and the work manager continues until the entire descriptor has been consumed.
Acceleration engine 32a stores completed data packets in transmit registers 28 for transmission onto AS fabric 11.
Response engine 102 processes the received request packet. For example, response engine 102 parses the data packet to identify its type (e.g., SLS), data that is being requested via the receive packet, the destination to which that data should be sent, and any other relevant information contained in the request packet. When processing the request packet, response engine 102 determines which data to provide in a response based, e.g., on address information contained in the request packet. Address translations or conversions may be performed by response engine 102 in order to correlate address(es) in the request packet to system memory address(es) from which data is to be read.
Response engine 102 retrieves (106) data from appropriate addresses in system memory and builds one or more descriptors using that data. The descriptors are essentially the same as the descriptors described above, except that they are built by response engine 102 instead of by CPU 24. Response engine 102 may support any of the descriptor formats described herein. Different formats may be associated, e.g., with different AS end node devices or different types of request packets.
Response engine 102 notifies (107) arbiter 100 that response engine 102 has descriptors to write/store in registers 34. Arbiter 100 gives priority to response engine 102, meaning that arbiter 100 allows response engine 102 to write its descriptors to registers 34 first, ahead of DMA engine 30. This is because, generally speaking, PI engine 29 gives priority to external requests over data transmissions. Response engine 102 writes (108) its descriptors to registers 34. Response engine 102 may also store a tag associated with each descriptor in registers 34, as described. Thereafter processing proceeds as set forth in
The AS end node device described herein may be used in any context. For example, an AS end node device may be used in a storage system 80, as shown in
As shown in
PI engine 89 includes a DMA engine to retrieve descriptors from memory 87 and to store those descriptors in a register. PI engine 89 also includes a work manager that retrieves a descriptor, and that determines, based on the descriptor, whether the data packet has a format that is native to AS fabric 81. If the data packet has a format that is non-native, then the work manager builds one or more data packets from the descriptor, as described above. If the data packet has a format that is native, the work manager sends the descriptor to an acceleration engine, along with a command. The acceleration engine receives the descriptor from the work manager, and uses the information in the descriptor to build the data packet. As described above, the work manager and the acceleration engine may operate together to build multiple data packets from the same descriptor.
The data packets generated by PI engine 89 may be SLS data packets, which enable management server 82 to access and to store data in memories of the other data servers 84a, 84b, 84c without “going through” their CPUs.
One or more of the other data serves 84a, 84b, 84c may act as a local management server for a sub-set of data servers (or other data servers). Each server in this sub-set may include RAID or other storage media, which the local management server can access without going through a local CPU. The architecture of such a data server 84a is similar to that of management server 82. For example, the local management server may include a local processor that stores local descriptors in a local queue. The local descriptors contain information used to packetize local data for transmission across AS fabric 81 or another AS fabric. The data packets may have an SLS format or other format. A local PI engine retrieves local descriptors from local memory, and uses the local descriptors to generate data packets for transmission to one or more other data servers' memory.
The AS end node device described herein may also be used in connection with a network processor. For example, as shown in
The foregoing are only two examples of systems in which an AS end node device of
Furthermore, processes 60 and 104 are not limited to use with the hardware and software described herein; they may find applicability in any computing or processing environment Processes 60 and 104 can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. The processes can be implemented as a computer program product or other article of manufacture, e.g., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable storage device or in a propagated signal, for execution by, or to control the operation of, data processing apparatus, e.g., a programmable processor, a computer, or multiple computers. A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.
Processes 60 and 104 can be performed by one or more programmable processors executing a computer program to perform functions. Process 60 and 104 can also be performed by, and apparatus of the process 60 and 104 can be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).
Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. Elements of a computer include a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks.
Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, e.g., EPROM (electrically programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM (compact disc read-only memory) and DVD-ROM (digital video disc read-only memory). The processor and the memory can be supplemented by, or incorporated in special purpose logic circuitry.
Processes 60 and 104 can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer, or any combination of such back-end, middleware, or front-end components.
The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.
The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
Other embodiments not described herein are also within the scope of the following claims.
