Patent Grant
7111102
This application is related to prior co-pending commonly assigned application Ser. No. 09/790,970, filed Feb. 22, 2001, entitled “Apparatus and technique for conveying per-channel flow control information to a forwarding engine of an intermediate network node,” of Guy Fedorkow et al., the entire contents of which are hereby incorporated by reference as if fully set forth herein.
A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction of the patent disclosure, as it appears in the Patent & Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. Copyright © 2003 Cisco Systems, Inc.
This disclosure includes a CD-ROM Appendix consisting of 134 pages in one file on one CD-ROM, the entire contents of which are hereby incorporated by reference as if fully set forth herein.
This invention generally relates to digital computer systems, and relates more particularly to digital computers that include a SPI-4 bus.
The approaches described in this section could be pursued, but are not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.
Digital computers use input/output (I/O) buses for transferring information between peripheral devices and a computer central processing unit and computer memory. I/O functions are also required in systems with multiple distributed processors and multiple distributed memories.
A variety of I/O bus architectures are used in such computer systems, including Personal Computer Interface (PCI). The System Packet Interface-4 bus is a relatively new high-bandwidth bus that is generally used in data packet processing systems for computer networks, such as high-speed routers and switches. Characteristics of the SPI4.2 bus architecture are described in an Interface Specification that is available in the document www.oiforum.com/public/impagreements.html. In this document, the term “SPI-4” is equivalent to “SPI4.2,” and includes variants and equivalents of the SPI4.2 bus architecture.
Although the SPI-4 bus provides a high-speed communication path for packet data within a computer system, the SPI-4 bus is not suitable for direct communication to external networks or devices. Interfacing a host with a SPI-4 bus to a network normally requires providing logical or physical ports or interfaces that are coupled to other devices or networks. Some port adapters are architected as service adapters that have no ports or interfaces, but provide a particular kind of packet processing service for a host, such as compression or decompression, encryption or decryption, etc.
Users and manufacturers particularly desire to have host systems that can accommodate ports and interfaces that use different technologies, such as Ethernet, Fast Ethernet, Gigabit Ethernet, optical, serial or other interfaces. In one approach, a host router or switch is hard-wired with a variety of different ports. However, a user cannot re-configure such a host if the user's port requirements change. Such users and manufacturers want to have a host system that is adaptable to changing port and interface requirements.
Hot swapping may also damage some devices connected to the SPI-4 bus such as devices using Complimentary Metal Oxide Semiconductor (CMOS) technology. CMOS devices are exposed to large currents when inputs to CMOS receivers are within the CMOS switching region. Some CMOS receivers have two field effect transistors (FETs) connected in series with a first FET connected to a positive power supply rail and a second FET connected to a negative power supply rail. When the input to the two FETS is in the switching region, both FETs can be continuously turned on at the same time creating a DC current path directly through the CMOS device. The continuous on state of the two FETs can dissipate enough power to damage the CMOS device.
CMOS devices also experience latch-up conditions when an input is driven beyond one of the CMOS power supply rails. In the latch-up condition, parasitic transistors in the CMOS structure dissipate large amounts of power that can destroy the CMOS device. Both power dissipation conditions described above can result from hot swapping on the host interface bus.
U.S. Pat. No. 5,793,987 and U.S. Pat. No. 6,163,824 of Quackenbush et al. disclose a port adapter with separate PCI local bus and local bus, and associated processing methods. A port adapter is an electronic device that provides one or more ports and that plugs into a host system to provide additional features or functions for the host. The technology of Quackenbush et al. has been used in PCI bus-based port adapters in the Cisco 7200 Series Routers and Cisco 7500 Series Routers, from Cisco Systems, Inc., San Jose, Calif. However, the technology of Quackenbush et al. is not suitable for hosts having SPI-4 bus architectures because of vast technical differences between the PCI bus and the SPI-4 bus. For example, the PCI bus cannot process data that is arriving from interfaces at high rates such as 10 gigabits per second (Gbps).
Still another drawback of existing port adapters is that they do not interoperate seamlessly with heterogeneous network environments. For example, a host with a plurality of port adapters may communicate with external networks or devices using any of a large number of network technologies. As a result, data packets that are received at the port adapters may have any of a large number of different formats. Requiring the host system to understand and process a large number of different packet formats would be complicated and lack scalability to new technologies. Further, it would be impractical to have one generic packet format used between each type of port adapter and the host system, because of differences in the type and quantity of data carried in packets of different technologies.
Thus, there is a need for a port adapter that can process a particular packet format for a particular technology, and provides data to the host in a single consistent packet format for internal processing.
Based on the foregoing, there is a clear need in the relevant technical field for a port adapter that can interface a host system having a SPI-4 bus architecture to different network technologies. More broadly, there is a need for an apparatus that can provide a hot-pluggable adaptive interface from the SPI-4 bus of a host to external peripheral equipment.
The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:
A hot-pluggable port adapter for a high-speed bus is described. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention.
Embodiments are described herein according to the following outline:
The needs identified in the foregoing Background, and other needs and objects that will become apparent from the following description, are achieved in the present invention, which comprises, in one aspect, a hot-pluggable port adapter for connecting network interfaces to a host system through a SPI-4 bus. The port adapter communicates with the host system through a port adapter/host interface that includes the SPI-4 bus and a control bus; an extended flow control bus may be provided. Methods are provided for selecting and using one of a small plurality of different packet formats for various networking technologies, so that the port adapter can hide details of the technology that it handles from the host system, and for operating the host system's SPI-4 bus at one of several speeds based on bandwidth requirements of the port adapter.
According to one aspect, the invention provides a port adapter for coupling zero or more network interfaces to a host system having a SPI-4 bus, the port adapter comprising zero or more network interfaces; a SPI-4 bus coupled to a host system to provide a communication channel between the host and the network interfaces; a control bus coupled to the host system for controlling and monitoring the port adapter; and interface logic that interfaces the SPI-4 bus and the control bus to the network interfaces.
According to one feature, the interface logic comprises a field programmable gate array (FPGA), application-specific integrated circuit (ASIC), a combination of these and one or more other hardware elements, or a combination of one or more other hardware elements. According to another feature, an identity bus is coupled to the host system to allow identification of the port adapter by the host system. In yet another feature, an extended flow control bus is provided on which the port adapter may convey FIFO status to the host system on a plurality of separate logical channels. In still another feature, a clock bus communicates network timing information between a port and the host system, for providing synchronization of a port to another port, synchronization of a host system reference oscillator to a port, or synchronization of a port to a reference clock that is external to the host system.
The port adapter may further comprise a power control circuit that selectively generates power for the adapter during on line insertion and removal of the port adapter from the host system while the host system remains powered on. According to one feature, an identification repository stores a unique identifier of a type of the port adapter. In a related feature, the identification repository further stores one or more configuration parameter values associated with the port adapter. In yet another related feature, the identification repository comprises an electrically erasable programmable read only memory. The identification repository may store values that allow the host to determine whether the port adapter can be supported by the host system. The identification repository may store values that allow the host to determine one or more operating frequencies of the SPI-4 bus.
In another feature, an extended flow control bus is coupled to the host system that enables the adapter to communicate information on the SPI-4 bus using more than the 256 logical channels that are conventionally available using the SPI-4 bus. In a related feature, flow control interface logic for the flow control bus comprises a calendar-based mechanism that allows the port adapter to convey buffer fill status of thousands of logical channels to the host system. The extended flow control bus may comprise a TDM calendar frame sync signal, a flow control clock signal, a status signal, and a parity signal.
According to one feature, the adapter comprises zero interfaces, and wherein the interface logic is configured to receive one or more packets from the host system, transform the packets according to a specified function, and send the transformed packets to the host system. In a related feature, the specified function comprises encryption or decryption.
In another aspect, the invention provides a method of selectively determining an operating frequency for a SPI-4 bus of a host computer system that uses a port adapter, wherein the operating frequency may be different than a conventional SPI-4 bus operating frequency, comprising the steps of issuing a query from a host computer system having a SPI-4 bus to a port adapter, the port adapter comprising a SPI-4 bus that can be coupled to a host system for control and data between the host and the SPI-4 device, a control bus coupled in parallel with the SPI-4 bus between the host system and the port adapter for the port adapter independently from the SPI-4 bus, and interface logic that interfaces the SPI-4 bus and the control bus to one of a plurality of line interfaces, and an identification repository; receiving, from the identification repository, an identification of the port adapter; determining, based on the information received from the identification repository, whether the host system SPI-4 bus can operate at a frequency that is compatible with at least one of the SPI-4 bus operating frequencies that are supported by the port adapter; and setting an operating frequency of the host system SPI-4 bus equal to a selected one of the SPI-4 bus operating frequencies that are supported by the port adapter.
In one feature of this aspect, the setting step comprises setting the operating frequency of the host system SPI-4 bus equal to a fastest one of the SPI-4 bus operating frequencies that are supported by the port adapter. In another feature, the method includes powering-on the port adapter only when the host system SPI-4 bus can operate at a frequency that is compatible with at least one of the SPI-4 bus operating frequencies that are supported by the port adapter. In a related feature, the method comprises powering-on the port adapter only when one or more factors are satisfied, wherein the factors are selected from the set consisting of: the host system has software support for a packet format required by the port adapter; the port adapter dissipates less than a maximum amount of power dissipation allowed by the host system; the host system can match a bandwidth required by the port adapter; or a license authorization requirement associated with the port adapter allows the port adapter to run on the host system.
In another feature, the method further comprises receiving, from the identification repository, values that allow the host to determine whether the port adapter can be supported by the host system, and one or more operating frequencies of the SPI-4 bus. The method may further comprise receiving, from the identification repository, values that allow the host to determine a packet format of data that is sent across the SPI-4 bus by the port adapter. In a related feature, the method may further comprise receiving, from the identification repository, one or more values specifying a packet format of data that is sent across the SPI-4 bus by the port adapter.
In yet another aspect, the invention provides a port adapter for coupling zero or more network interfaces to a host system having a SPI-4 bus, the port adapter comprising: zero or more network interfaces; a SPI-4 bus coupled to a host system to provide a communication channel between the host and the network interfaces; a control bus coupled to the host system for controlling and monitoring the port adapter; interface logic that interfaces the SPI-4 bus and the control bus to the network interfaces; and packet processing logic for pre-processing packets received on the interfaces by performing the steps of: receiving a first packet on an ingress interface of the port adapter; creating a second packet that conforms to a selected one of the internal packet formats; transforming data from one or more fields of the first packet to one or more corresponding fields of the second packet; providing the second packet to a host system.
In one feature of this aspect, the packet processing logic further comprises the steps of moving a remainder of a packet header and packet body from the first packet into the second packet. The packet processing logic may be configured to perform the step of selecting one of a plurality of internal packet formats. The ingress interface may be, for example, an Ethernet interface, ATM interface, frame relay, serial interface, highly channelized interface, RPR interface, or POS interface, or any other interface now known or invented hereafter.
In other aspects, the invention encompasses a computer apparatus and a computer-readable medium configured to carry out the foregoing steps.
2.0 Structural and Functional Overview
2.1 Port Adapter Architecture
A pluggable port adapter is used to connect zero or more ports or interfaces to a host system through a SPI-4 bus to add functionality to the host system. Typically the ports or interfaces are high-bandwidth optical ports or interfaces. The port adapter communicates with the host system through a port adapter/host interface that includes the SPI-4 bus, a control bus, an extended flow control bus, and other signals and power lines. The ports or interfaces are configured and communicate in a normal manner over the SPI-4 bus while other functionality on the port adapter is controlled independently through the control bus.
In this document, the term “SPI-4” is equivalent to “SPI4.2” and includes variants of the SPI4.2 bus architecture. Thus, an embodiment may use a bus that strictly adheres to the SPI-4 specification, or may use variants, enhancements, modifications or improvements to the SPI-4 specification.
The host system uses a specialized identity bus to determine the identity of a port adapter, which the host system then uses to determine what programming and configuration is required. The control bus is used by the host system for configuring and controlling devices on the port adapter, and for updating programmable circuitry on the port adapter such as field programmable gate arrays (FPGAs). Alternatively, a JTAG bus can be used to update such programmable devices. FPGAs with volatile program memory must be programmed each time they are powered up and can be reprogrammed in the field by the host system to repair bugs and to enhance performance and/or functionality.
In one embodiment, the SPI-4 bus in the port adapter is coupled to SPI-4 termination logic, which is coupled to one or more network interfaces, such as a framer, ATM SAR, etc. In cooperation, the SPI-4 termination logic and network interfaces control zero or more ports, which are coupled to zero or more communication lines, devices, or networks such as local area networks (LAN) and wide area networks (WAN). The SPI-4 termination logic and network interfaces cooperate to receive data from the ports or interfaces and then retransmit the data on the SPI-4 bus in a reprocessed form, and to receive data from the host system and retransmit such data on a port.
The control bus can be used for programming logic on the port adapter such as interface logic, network interfaces and general-purpose registers or other programmable elements. The control bus also provides access to control interfaces of devices on the port adapter. The power bus is used, in conjunction with software that controls application of power, for conducting hot swap operations in which the port adapter is unplugged from or plugged into the host system while the host system either is powered and operating or is powered down. The port adapter may include a connector with one or more detect pins that are shorter than other power bus pins and other signal pins in the connector. In one embodiment, the shorter pins are at opposite ends of the connector. The shorter detect pins allow the port adapter and host system to anticipate and, in turn, respond to a hot swap condition by enabling power to the port adapter only after the port adapter is fully inserted. The shorter pins also enable the host system to determine that all pins are seated correctly. Upon removal of a port adapter, the short pins disconnect first and enable the port adapter to send signals to the host that are used to disable power to the port adapter.
When the port adapter is connected to the host system during a hot swap condition, control circuitry starts a controlled power-up sequence. When the port adapter is disconnected from the host system during a hot swap condition, the control circuitry starts a controlled power-down sequence.
A hot swap protocol between the port adapter and the host system discontinues data communications on the SPI-4 bus in the port adapter when the port adapter is not at an operational power level. When the port adapter is disconnected from the host system, signals coming from host circuitry are changed to known safe states to prevent potentially high currents from damaging devices during on-line insertion operations. The hot swap protocol also prevents corruption of data on the SPI-4 bus and corrupting logic states in the host.
A port adapter as disclosed herein can process data that is arriving from interfaces up to 10 gigabits per second (Gbps). In other embodiments, improvements to the SPI-4 bus architecture that allow faster data rates may be accommodated.
Host system 100 may have one or more hot-pluggable port adapters 104A, 104B, 104N. There may be any number of port adapters in a practical system. A port adapter is also referenced herein by the equivalent term “shared port adapter” or SPA, or “versatile port adapter” or VPA. Each of the port adapters 104A, 104B, 104N comprises zero or more ports 106A, 106B, 106N. Each port is communicatively coupled to one of the networks 110A, 110B, 110N or devices within such networks using any suitable network communication technology, such as Ethernet, Fast Ethernet, Gigabit Ethernet, optical, serial or other interfaces. There may be any number of ports on a port adapter in a practical system. Each port 106A, 106B, 106N may be coupled to a different network 110A, 110B, 110N.
An embodiment with zero ports may comprise a service adapter in which the port adapter provides a computational or packet processing service rather than an interface function. A port adapter as described herein may have zero ports but may provide, for example, an encryption or compression function for the host. Further, in another embodiment a combination service adapter and port adapter may be provided that has one or more ports and also provides a packet processing service.
In the configuration of
In the example of
In embodiments with zero ports, network interface 204 is omitted and other logic for performing packet processing services may be provided in its place. For example, an encryption engine or compression engine may occupy the same logical location as network interface 204.
Control bus 206 is connected to local control logic 208 in port adapter 104C. Identity bus 228 is connected to identity element 212, which can be queried by the host system 100 to determine the hardware arrangement and logical configuration of the port adapter 104C. The port adapter 104C further may include a power control element 214 and clock distribution circuit 216 that are respectively coupled to power control bus 226 and clock bus 224.
Extended flow control bus 222 is coupled to extended flow control logic 220. Details of the extended flow control bus are described further in a separate section below.
The port adapter 104C may be implemented as a plurality of integrated circuits that are mounted on one or more printed circuit cards that are enclosed in a protective housing. In one embodiment, each port adapter is mounted in a slot of a processing circuit card in the host system (“host card”). The port adapter housing may have any of several form factors, thereby providing a modular arrangement so that multiple different port adapters are interchangeable in the same host chassis. In one embodiment, a port adapter housing may have a half-height, full-height, double-wide, or high-power form factor based on the number and type of ports or interfaces provided in the port adapter, the amount of power dissipated by the port adapter, or the area required for the circuitry in the port adapter.
Local control logic 208, extended flow control logic 220, clock distribution circuit 216, power control circuit 214, and identity element 212 are represented in simplified, block form for clarity. In particular, connections to each such element are simplified, and each such element may have other connections in specific embodiments. Further, port adapter 104C may include circuit elements other than the specific elements that are shown in
A host connector 302 provides a physical connection to the host system 100 (not shown in
SPI-4 bus 332 is coupled to FPGA 330. The SPI-4 bus 332 typically is an 86-pin packet data transfer bus that carries data bus signals, flow control signals, clock signals, etc. In certain embodiments, the operating speed of all such signals may be changed for compatibility among different hosts and port adapters, as further described herein. A SPA bus 206A is coupled from the host system 100 to the FPGA 330. SPA control bus 206A enables the host system to control and communicate with internal elements of the port adapter 104D. A JTAG bus 307 may carry test signals that are used for testing or PLD programming through communication among the host system 100 and programmable components; such as an FPGA 330 that implements the functions of SPI-4 termination logic 202 and local control logic 208 of
FPGA 330 is coupled to framer 204A by a PL3 bus 309, microprocessor bus 310, and Transport Overhead (TOH) path 312. In one embodiment, framer 204A is the PM5360 S/UNI Multi-48 SONET/SDH framer, from PMC-Sierra, Inc., Santa Clara, Calif., which provides four (4) ports that are selectable between OC-12/STM-4 and OC-3/STM-1 bandwidth.
Framer 204A is coupled to one or more small form-factor pluggable (SFP) optics modules 314A, 314B, 314C, 314D that provide network ports and interfaces. The FPGA330 detects insertion or extraction of the modules 314A, 314B, 314C, 314D to or from port adapter 104D.
Generally, FPGA 330 functions to decode and interface signals of the SPA bus 206A to signals from microprocessor bus 310. The FPGA 330 also provides control and status information relating to SFPs 314A, 314B, 314C, 314D. Further, the FPGA 330 provides bridging, queuing, and scheduling for communications among the PL3 bus 309 and SPI-4 bus 332, including management of ingress and egress FIFO queues, and the FPGA 330 may be involved in online insertion and removal and power control functions. The FPGA 330 is also configured for inserting and extracting SONET overhead information from packets that are communicated to or from the framer 204A. In one embodiment, FPGA 330 is implemented using the Xilinx 2V1500 and the SPI4, PL3, and HDLC IP cores.
Port adapter 104D also may include various other functional elements including clock generation/recovery module 216A, identity electrically erasable programmable read-only memory (“ID EEPROM”) 212A, voltage supervisor 228A, temperature sensors 320, and voltage margining unit 322. The clock generation/recovery module 216A receives a 77.76 MHz SONET reference clock 224B from the host through host connector 302, provides a recovered 19.44 MHz clock 224A to the host via host connector 302, provides a 77.76 MHz reference clock 224C to the framer 204A, and receives a recovered 77.76 MHz clock signal 224D from the framer. The use of a recovered clock enables the clock generation/recovery module 216A to derive a system clock from any attached SONET port. The clock generation/recovery module 216A also generates a 100 MHz clock for operating FPGA 330.
The temperature sensors 320 may have a programmable temperature range for detecting and signaling over-temperature problems.
Optionally, an extended flow control bus coupled from host system 100 to the port adapter 104D functions to provide back pressure for port adapters with very high counts of physical or virtual ports. For example, in ATM, numerous virtual circuits may be present on one physical link. Thus, the extended flow control bus may be used for highly channelized port adapters, ATM port adapters, etc.
A detailed specification for an embodiment of a port adapter is provided in the Appendix to this document, the entire contents of which are hereby incorporated by reference as if fully set forth herein.
2.2 Adaptation To Different SPI-4 Operating Speed Rates
The host system 100 can query the ID EEPROM 212A using an identity bus 228A to determine the configuration of the port adapter 104D and to perform power control functions. In one embodiment, bus 228A conforms to the I2C signal format. The ID EEPROM 212A is an example of an identification repository. Based on the identifying information, software executed by the host system determines values specifying an operating frequency of the port adapter, and the format of data that is sent across the SPI-4 bus. For example, the host system software may include a lookup table that maps bus speed values, data packet formats, etc., to various port adapter identifiers. In an alternate embodiment, the identification repository stores bus speed values, data packet formats, and other configuration parameters in association with one or more port adapter identifiers for that port adapter or several different port adapters.
The information in the identification repository enables the host system to adapt its operational behavior to particular characteristics of the port adapter or its ports. For example, the standard operating frequency of the SPI-4 bus is 350 MHz (“full rate SPI-4”). However, not all port adapters require this frequency. For example, a port adapter that supports an aggregate data communication bandwidth of greater than 2.4 Gbps on its interfaces may require a full rate SPI-4 bus, but other port adapters that support only aggregate data communication bandwidth of less than or equal to 2.4 Gbps may operate adequately using SPI-4 bus signaling at less than 350 MHz.
Therefore, in one embodiment, the SPI-4 bus of port adapter 104D may be configured to operate at a quarter-rate speed of 87.5 MHz. In other embodiments, the SPI-4 bus of port adapter 104D may be configured to operate at any other speed, e.g., 700 MHz providing double-rate speed, etc. The identification repository of a port adapter contains a port adapter type identifier. Based on the port adapter type identifier, software executed by the host system can determine whether the port adapter supports a full rate SPI-4 bus speed, quarter rate, or both, or some other speed. Generally, in one embodiment,
Using this arrangement, the host system may query the identification repository and adapt its operational behavior based on information in the identification repository.
In block 404, a response is received from the port adapter that includes a unique identifier for the port adapter. For example, reading ID EEPROM 212A results in port adapter 104D providing its unique identifier value. In block 405, the host determines one or more SPI-4 bus operating rates that are supported by the port adapter. For example, the host uses a stored lookup table to associate the received unique identifier value with one or more operating frequency values for the port adapter. Additionally, the host system may determine whether the port adapter can be supported by the host system, and the format of data that is sent by the port adapter on the SPI-4 bus. Alternatively, such values and configuration parameters are provided from the identification repository of the port adapter.
In block 406, the host system determines whether it is compatible with one of the supported rates that the host determined based on the identifier received from the identification repository of the port adapter. Block 406 may involve applying rules 1–4 as denoted above to determine whether a port adapter and host are compatible. For example, if the port adapter supports only quarter rate SPI-4, and the host requires full rate, then the host is not compatible with the port adapter. If the host is not compatible, then in block 407, the host does not power-up the port adapter, which cannot be used by the host system. The rules 1–4 above may be implemented in software executed by the host system.
Optionally, the process involves powering-on the port adapter only when one or more factors are satisfied. For example, block 406 can involve evaluating factors such as: whether the host system has software support for a packet format required by the port adapter; whether the host system has software support for the port adapter; whether the port adapter dissipates less than a maximum amount of power dissipation allowed by the host system; whether the host system can match a bandwidth required by the port adapter; whether a license authorization requirement associated with the port adapter allows the port adapter to run on the host system; etc.
If the host is compatible with the port adapter, then in block 408 the host changes the operating rate of its own SPI-4 bus to the fastest compatible supported rate. For example, if the port adapter identification repository indicates that the port adapter supports both quarter rate and full rate SPI-4, then the host changes its SPI-4 operating rate to full rate. In an alternative embodiment, the host changes the operating rate of its own SPI-4 bus to any one of the compatible supported rates.
In block 410, the host powers-up the port adapter by sending appropriate control signals; in the example of
Optionally, in other embodiments, the order of performing steps 408, 410, 412 may be changed, and the order of performing such steps is not critical.
Thus, using the approach of
Further, based on the received information, the host system may determine an operating frequency at which to run the SPI-4 bus. For example, full rate SPI-4 may be used, quarter rate may be used, etc.
The host system may also determine a particular format for data communication on the SPI-4 bus, as described further in section 2.4 below, for example.
2.3 Extended Flow Control Bus
A conventional SPI-4 bus addresses a maximum of 256 channels, and provides support for FIFO queue status indications for 256 channels in a normal addressing mode. However, port adapters that have a large number of channels (“highly channelized” or ATM SPAs, for example) may need 1,000 or more channels. Therefore, it is desirable to have a port adapter flow control bus that can support more than 256 channels per port adapter.
Accordingly, an extended flow control bus and associated method is provided to extend a port adapter to enable use with more than 256 channels. In this arrangement, a port adapter requiring less than or equal to 256 channels may use a conventional SPI-4 control bus for flow control, and optionally may use an extended flow control bus as defined herein. If a port adapter uses the extended flow control bus as defined herein, the port adapter also still uses the conventional SPI-4 flow control bus for gross (rather than subchannel or virtual channel) flow control of traffic aggregates such as port adapter-level or physical port-level flow control.
In one embodiment, Extended Flow Control Bus 222 carries a time domain multiplexed (TDM) calendar frame sync signal, a flow control clock signal, a status signal, and a parity signal. The flow control clock signal provides a source clock that is used by the host to clock in the data value on the status signal, and is sourced by the sender of flow control data, which is normally the port adapter. An example clock frequency is 50 MHz, but any other suitable clock frequency may be used.
In one embodiment, the status signal is a one-bit signal, but other forms of status signaling may be used. The status signal provides an indication whether channel FIFO status is above or below a threshold value, corresponding to the channel programmed for the TDM timeslot. The parity value provides even or odd parity, in various embodiments, across the status signal and frame sync signal for a particular clock cycle. Use of a separate parity signal allows flexibility in changing the frame size to any length, in various embodiments. Optionally, a port adapter may not support the extended flow control bus, in which case the foregoing signals are not connected.
Thus, in an embodiment, the extended flow control bus uses a TDM calendar-based mechanism that carries per-channel FIFO status information over a single data bit. The calendar is programmed by the host system 100 when channels are configured and set up at the port adapter and host. In one embodiment, time slots are allocated in proportion to the bandwidth of the channel. Embodiments may approximate channel bandwidths to the closest power of 2 and may allocate time slots in a way that reduces the total number of flow control time slots. In one embodiment, the calendar comprises a table in which rows correspond to timeslots and columns carry channel numbers and FIFO status information. In one particular embodiment, there are 16584 rows each comprising a channel number in 12 bits and one status bit.
The port adapter uses the calendar to determine which channel is polled for FIFO status and which channel is sent in a particular timeslot or clock period. The host uses a similarly configured calendar to determine which channel's FIFO status flow control information is carried in a particular timeslot.
In one embodiment, the number of supported channels is configurable so that it can adjust to the capabilities of a particular host. For example, a host card may support only 1K flow-controllable entities, and therefore certain port adapters may need to support fewer than the maximum number of channels.
A detailed description of the extended flow control bus is provided in section 2.3 of the Appendix.
2.4 Pre-Processing Packets With Port Adapter
In one embodiment, each port adapter 104A, 104B, 104N may communicate with external networks or devices using any of a large number of network technologies. As a result, data packets that are received at the port adapters may have any of a large number of different formats. In an embodiment, each port adapter provides data to the host in one of a small number of basic packet formats, all of which are understood by the host. For example, in one specific embodiment, four (4) packet formats are used, and a port adapter supports one or more of the four formats to communicate with a host. In this approach, since port adapters are targeted at many different host systems, the formats hide the detail and processing burden associated with a specific media type as much as possible within the port adapter to assist the host to operate at high speed or with less complex packet processing. In addition, the packet formats provide header fields that are as small as possible, to reduce the bandwidth utilized on the SPI-4 bus.
In block 422, a packet is received on an ingress interface of a port adapter. In one embodiment, the process of
In block 424, one of a plurality of different packet formats is selected. Block 424 typically involves selecting one of several packet formats, e.g., a format other than the native format in which the packet was received. In one specific embodiment described further below, a packet format is selected from among Ethernet SPA 8-byte shim format, ATM SPA 4-byte shim format, Highly Channelized SPA 4-byte shim format, and a no shim format. The selected format may include more or fewer data fields than the fields that are in the received packet. Performing block 424 may comprise simply selecting one specified packet format associated with the then-current port adapter. Further, in the case of a port adapter that supports Ethernet packets, a particular packet format may be selected based on a VLAN identifier carried in a packet.
In block 426, a new packet conforming to the selected format is created.
In block 428, data from the fields of the received packet is transformed into one or more corresponding fields of the new packet. The data transformation may be performed according to a data-driven mapping or programmatic rules that specify which fields of a particular ingress packet format are transformed to which other fields of the target packet format. Further, the mapping or rules may specify transformations of data or values obtained from sources other than the packet, such as interface identifier, packet length, congestion status, packet validity checks, etc.
In block 430, the new packet is provided to the host system. For example, in
The four packet formats used in an embodiment may be designated, for example, as:
The generalized format of
Stripping the Layer 2 header is optional on a per-packet basis, allowing support for Layer 2 tunnels such as Ethernet over MPLS. If the Layer 2 header is left on the packet, then it can also optionally be padded with two or three bytes to bring the Layer 3 header to 4-byte alignment, as an optional optimization for some hosts. The first byte of the padding indicates the number of padding bytes present, for example.
In this case, because the format of the packet leaving the port adapter can include optional stripping of the variable-length Layer 2 encapsulation, and the addition of a shim header, the Length Indicator value 504 indicates the number of bytes by which the packet is shorter as compared to when the packet was first received. The Layer 3 engine of host system 100 can determine the original Layer 2 length by adding the value of Length Indicator value 504 to the total number of bytes received from the port adapter.
Header fields 508 may include the Protocol ID (“PID”) of the Layer 2 header of the packet, and the port adapter may have translated the value. Certain special values of the PID field indicate that the host must apply special treatment to the particular packet; the special values are software configurable. For example, special PIDs may be used to indicate a tunneled packet, exception packet, or other special characteristics. For a tunneled packet, when the VLANID and port number of the arriving packet are configured to enter an Layer 2 tunnel, then the entire packet with its original Layer 2 encapsulation is brought into the host system. An exception packet indicates that the port adapter has detected something about the packet that requires the host to perform special treatment on the packet. More than one exception packet special PID may be defined. This may allow classification of the packets into different priority CPU queues, for example.
Optionally, as part of transforming a first packet into a particular selected packet format, the header 510 and/or body 512 of the original packet may be placed in the transformed packet 500. Thus, the packet format used within the host and port adapter may include the original packet header 510 and/or original packet body 512. The original header and body may be omitted depending on the nature of the traffic that is processed or the context in which it is processed.
Similar transformation techniques may be applied to other different packet formats of inbound packets.
Additionally or alternatively, rather than transforming packets, packets may be dropped. For example, if a port adapter receives a packet from a Layer 2 address or VLAN that is of no interest to the port adapter or host, then that packet may be dropped.
3.0 Extensions and Alternatives
In the foregoing specification, the invention has been described with reference to specific embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.
For example, a port adapter may classify packets as high priority or low priority and provide priority information in the transformed packet format to enable the host to determine which packets to process first. As an alternative to carrying packet priority information in the transformed packet format, two or more logical SPI-4 channels may be associated with one physical port, in which a first logical channel carries port traffic associated with a first priority level and the second or additional channel(s) carry traffic associated with a second or other priority level. In this approach, the host adapter may be configured through software to process all packets arriving on the higher priority channel first without having to consult priority information within the packets.
In another variation of the architecture described above, one of the logical SPI-4 channels may be used as a control path as an alternative to providing some or all control signals on control bus 206 (
In still another variation, a specified SPI-4 logical channel can be used to carry flow control information, e.g., in the form of events.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5793987 | Quackenbush et al. | Aug 1998 | A |
| 5884043 | Teplitsky | Mar 1999 | A |
| 5991817 | Rowett et al. | Nov 1999 | A |
| 6054942 | Stemmler | Apr 2000 | A |
| 6119215 | Key et al. | Sep 2000 | A |
| 6163543 | Chin et al. | Dec 2000 | A |
| 6163824 | Quackenbush et al. | Dec 2000 | A |
| 6513109 | Kerr et al. | Jan 2003 | B1 |
| 6792561 | Mamata | Sep 2004 | B1 |
| 6912602 | Sano et al. | Jun 2005 | B1 |
| 20040028063 | Roy et al. | Feb 2004 | A1 |
| 20040153891 | Slutz et al. | Aug 2004 | A1 |
| 20050025055 | Jain et al. | Feb 2005 | A1 |
| 20050078601 | Moll et al. | Apr 2005 | A1 |
| 20050108518 | Pandya | May 2005 | A1 |
| Number | Date | Country |
|---|---|---|
| WO 0209551 | Nov 2002 | WO |
| WO 02095511 | Nov 2002 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20050149651 A1 | Jul 2005 | US |
