Patent Grant
7978702
1. Field of the Invention
The invention relates generally to network switches.
2. Background Art
A network switch is a device that provides a switching function (i.e., determines a physical path) in a data communications network. Switching involves transferring information, such as digital data packets or frames, among entities of the network. Typically, a switch is a computer having a plurality of circuit cards coupled to a backplane. In the switching art, the circuit cards are typically called “blades,” The blades are interconnected by a “switch fabric.” Each blade includes a number of physical ports that couple the switch to the other network entities over various types of media, such as Ethernet, FDDI (Fiber Distributed Data Interface), or token ring connections. A network entity includes any device that transmits and/or receives data packets over such media.
The switching function provided by the switch typically includes receiving data at a source port from a network entity and transferring the data to a destination port. The source and destination ports may be located on the same or different blades. In the case of “local” switching, the source and destination ports are on the same blade. Otherwise, the source and destination ports are on different blades and switching requires that the data be transferred through the switch fabric from the source blade to the destination blade. In some case, the data may be provided to a plurality of destination ports of the switch. This is known as a multicast data transfer.
Switches operate by examining the header information that accompanies data in the data frame. The header information includes the international standards organization (ISO) 7-layer OSI (open-systems interconnection model). In the OSI model, switches generally route data frames based on the lower level protocols such as Layer 2 or Layer 3. In contrast, routers generally route based on the higher level protocols and by determining the physical path of a data frame based on table look-ups or other configured forwarding or management routines to determine the physical path (i.e., route).
Ethernet is a widely used lower-layer network protocol that uses broadcast technology. The Ethernet frame has six fields. These fields include a preamble, a destination address, source address, type, data and a frame check sequence. In the case of an ethernet frame, the digital switch will determine the physical path of the frame based on the source and destination addresses. Standard Ethernet operates at a ten Mbit/s data rate. Another implementation of Ethernet known as “Fast Ethernet” (FE) has a data rate of 100 Megabits/s. Yet another implementation of FE operates at 10 Gigabits/sec.
A digital switch will typically have physical ports that are configured to communicate using different protocols at different data rates. For example, a blade within a switch may have certain ports that are 10 Mbit/s, or 100 Mbit/s ports. It may have other ports that conform to optical standards such as SONET and are capable of such data rates as 10 gigabits per second.
A performance of a digital switch is often assessed based on metrics such as the number of physical ports that are present, and the total bandwidth or number of bits per second that can be switched without blocking or slowing the data traffic. A limiting factor in the bit carrying capacity of many switches is the switch fabric. For example, one conventional switch fabric was limited to 8 gigabits per second per blade. In an eight blade example, this equates to 64 gigabits per second of traffic. It is possible to increase the data rate of a particular blade to greater than 8 gigabits per second. However, the switch fabric would be unable to handle the increased traffic.
It is desired to take advantage of new optical technologies and increase port densities and data rates on blades. However, what is needed is a switch and a switch fabric capable of handling higher bit rates and providing a maximum aggregate bit carrying capacity well in excess of conventional switches.
The present invention provides a high-performance network switch. Serial link technology is used in a switching fabric. Serial data streams, rather than parallel data streams, are switched in a switching fabric. Blades output serial data streams in serial pipes. A serial pipe can be a number of serial links coupling a blade to the switching fabric. The serial data streams represent an aggregation of input serial data streams provided through physical ports to a respective blade. Each blade outputs serial data streams with in-band control information in multiple stripes to the switching fabric.
In one embodiment, the serial data streams carry packets of data in wide striped cells across multiple stripes. Wide striped cells are encoded. In-band control information is carried in one or more blocks of a wide cell. For example, the initial block of a wide cell includes control information and state information. Further, the control information and state information is carried in each stripe. In particular, the control information and state information is carried in each subblock of the initial block of a wide cell. In this way, the control information and state information is available in-band in the serial data streams (also called stripes). Control information is provided in-band to indicate traffic flow conditions, such as, a start of cell, an end of packet, abort, or other error conditions.
A wide cell has one or more blocks. Each block extends across five stripes. Each block has a size of twenty bytes made up of five subblocks each having a size of four bytes. In one example, a wide cell has a maximum size of eight blocks (160 bytes) which can carry a 148 bytes of payload data and 12 bytes of in-band control information. Packets of data for full-duplex traffic can be carried in the wide cells at a 50 Gb/sec rate in each direction through one slot of the digital switch. According to one feature, the choice of maximum wide cell block size of 160 bytes as determined by the inventors allows a 4×10 Gigabit/sec Ethernet (also called 4×10 GE) line rate to be maintained through the backplane interface adapter. This line rate is maintained for Ethernet packets having a range of sizes accepted in the Ethernet standard including, but not limited to, packet sizes between 84 and 254 bytes.
In one embodiment, a digital switch has a plurality of blades coupled to a switching fabric via serial pipes. The switching fabric can be provided on a backplane and/or one or more blades. Each blade outputs serial data streams with in-band control information in multiple stripes to the switching fabric. The switching fabric includes a plurality of cross points corresponding to the multiple stripes. Each cross point has a plurality of port slices coupled to the plurality of blades. In one embodiment five stripes and five five cross points are used. Each blade has five serial links coupled to each of the five cross points respectively.
In one example implementation, the serial pipe coupling a blade to switching fabric is a 50 Gb/s serial pipe made up of five 10 Gb/s serial links. Each of the 10 Gb/s serial links is coupled to a respective cross point and carries a serial data stream. The serial data stream includes a data slice of a wide cell that corresponds to one stripe.
In one embodiment of the present invention, each blade has a backplane interface adapter (BIA). The BIA has three traffic processing flow paths. The first traffic processing flow path extends in traffic flow direction from local packet processors toward a switching fabric. The second traffic processing flow path extends in traffic flow direction from the switching fabric toward local packet processors. A third traffic processing flow path carried local traffic from the first traffic processing flow path. This local traffic is sorted and routed locally at the BIA without having to go through the switching fabric.
The BIA includes one or more receivers, wide cell generators, and transmitters along the first path. The receivers receive narrow input cells carrying packets of data. These narrow input cells are output from packet processor(s) and/or from integrated bus translators (IBTs) coupled to packet processors. The BIA includes one or more wide cell generators. The wide cell generators generate wide striped cells carrying the packets of data received by the BIA in the narrow input cells. The transmitters transmit the generated wide striped cells in multiple stripes to the switching fabric.
According to the present invention, the wide cells extend across multiple stripes and include in-band control information in each stripe. In one embodiment, each wide cell generator parses each narrow input cell, checks for control information indicating a start of packet, encodes one or more new wide striped cells until data from all narrow input cells of the packet is distributed into the one or more new wide striped cells, and writes the one or more new wide striped cells into a plurality of send queues.
In one example, the BIA has four deserializer receivers, 56 wide cell generators, and five serializer transmitters. The four deserializer receivers receive narrow input cells output from up to eight originating sources (that is, up to two IBTs or packet processors per deserializer receiver). The 56 wide cell generators receive groups of the received narrow input cells sorted based on destination slot indentifier and originating source. The five serializer transmitters transmit the data slices of the wide cell that corresponds to the stripes.
According to a further feature, a BIA can also include a traffic sorter which sorts received narrow input cells based on a destination slot identifier. In one example, the traffic sorter comprises both a global/traffic sorter and a backplane sorter. The global/traffic sorter sorts received narrow input cells having a destination slot identifier that identifies a local destination slot from received narrow input cells having destination slot identifier that identifies global destination slots across the switching fabric. The backplane sorter further sorts received narrow input cells having destination slot identifiers that identify global destination slots into groups based on the destination slot identifier.
In one embodiment, the BIA also includes a plurality of stripe send queues and a switching fabric transmit arbitrator. The switching fabric transmit arbitrator arbitrates the order in which data stored in the stripe send queues is sent by the transmitters to the switching fabric. In one example, the arbitration proceeds in a round-robin fashion. Each stripe send queue stores a respective group of wide striped cells corresponding a respective originating source packet processor and a destination slot identifier. Each wide striped cell has one or more blocks across multiple stripes. During a processing cycle, the switching fabric transmit arbitrator selects a stripe send queue and pushes the next available cell (or even one or more blocks of a cell at time) to the transmitters. Each stripe of a wide cell is pushed to the respective transmitter for that stripe.
The BIA includes one or more receivers, wide/narrow cell translators, and transmitters along the second path. The receivers receive wide striped cells in multiple stripes from the switching fabric. The wide striped cells carry packets of data. The translators translate the received wide striped cells to narrow input cells carrying the packets of data. The transmitters then transmit the narrow input cells to corresponding destination packet processors or IBTs. In one example, the five deserializer receivers receive five subblocks of wide striped cells in multiple stripes. The wide striped cells carrying packets of data across the multiple stripes and including destination slot identifier information.
In one embodiment, the BIA further includes stripe interfaces and stripe receive synchronization queues. Each stripe interface sorts received subblocks in each stripe based on originating slot identifier information and stores the sorted received subblocks in the stripe receive synchronization queues.
The BIA further includes along the second traffic flow processing path an arbitrator, a striped-based wide cell assembler, and the narrow/wide cell translator. The arbitrator arbitrates an order in which data stored in the stripe receive synchronization queues is sent to the striped-based wide cell assembler. The striped-based wide cell assembler assembles wide striped cells based on the received subblocks of data. A narrow/wide cell translator then translates the arbitrated received wide striped cells to narrow input cells carrying the packets of data.
A second level of arbitration is also provided according to an embodiment of the present invention. The BIA further includes destination queues and a local destination transmit arbitrator in the second path. The destination queues store narrow cells sent by a local traffic sorter (from the first path) and the narrow cells translated by the translator (from the second path. The local destination transmit arbitrator arbitrates an order in which narrow input cells stored in the destination queues is sent to serializer transmitters. Finally, the serializer transmitters then that transmits the narrow input cells to corresponding IBTs and/or source packet processors (and ultimately out of a blade through physical ports).
According to a further feature of the present invention, system and method for encoding wide striped cells is provided. The wide cells extend across multiple stripes and include in-band control information in each stripe. State information, reserved information, and payload data may also be included in each stripe. In one embodiment, a wide cell generator encodes one or more new wide striped cells.
The wide cell generator encodes an initial block of a start wide striped cell with initial cell encoding information. The initial cell encoding information includes control information (such as, a special K0 character) and state information provided in each subblock of an initial block of a wide cell. The wide cell generator further distributes initial bytes of packet data into available space in the initial block. Remaining bytes of packet data are distributed across one or more blocks in of the first wide striped cell (and subsequent wide cells) until an end of packet condition is reached or a maximum cell size is reached. Finally, the wide cell generator further encodes an end wide striped cell with end of packet information that varies depending upon the degree to which data has filled a wide striped cell. In one encoding scheme, the end of packet information varies depending upon a set of end of packet conditions including whether the end of packet occurs at the end of an initial block, within a subsequent block after the initial block, at a block boundary, or at a cell boundary.
According to a further embodiment of the present invention, a method for interfacing serial pipes carrying packets of data in narrow input cells and a serial pipe carrying packets of data in wide striped cells includes receiving narrow input cells, generating wide striped cells, and transmitting blocks of the wide striped cells across multiple stripes. The method can also include sorting the received narrow input cells based on a destination slot identifier, storing the generated wide striped cells in corresponding stripe send queues based on a destination slot identifier and an originating source packet processor, and arbitrating the order in which the stored wide striped cells are selected for transmission.
In one example, the generating step includes parsing each narrow input cell, checking for control information that indicates a start of packet, encoding one or more new wide striped cells until data from all narrow input cells carrying the packet is distributed into the one or more new wide striped cells, and writing the one or more new wide striped cells into a plurality of send queues. The encoding step includes encoding an initial block of a start wide striped cell with initial cell encoding information, such as, control information and state information. Encoding can further include distributing initial bytes of packet data into available space in an initial block of a first wide striped cell, adding reserve information to available bytes at the end of the initial block of the first wide striped cell, distributing remaining bytes of packet data across one or more blocks in the first wide striped cell until an end of packet condition is reached or a maximum cell size is reached, and encoding an end wide striped cell with end of packet information. The end of packet information varies depending upon a set of end of packet conditions including whether the end of packet occurs at the end of an initial block, in any block after the initial block, at a block boundary, or at a cell boundary.
The method also includes receiving wide striped cells carrying packets of data in multiple stripes from a switching fabric, translating the received wide striped cells to narrow input cells carrying the packets of data, and transmitting the narrow input cells to corresponding source packet processors. The method further includes sorting the received subblocks in each stripe based on originating slot identifier information, storing the sorted received subblocks in stripe receive synchronization queues, and arbitrating an order in which data stored in the stripe receive synchronization queues is assembled. Additional steps are assembling wide striped cells in the order of the arbitrating step based on the received subblocks of data, translating the arbitrated received wide striped cells to narrow input cells carrying the packets of data, and storing narrow cells in a plurality of destination queues. In one embodiment, further arbitration is performed including arbitrating an order in which data stored in the destination queues is to be transmitted and transmitting the narrow input cells in the order of the further arbitrating step to corresponding source packet processors and/or IBTs.
Further embodiments, features, and advantages of the present inventions, as well as the structure and operation of the various embodiments of the present invention, are described in detail below with reference to the accompanying drawings.
The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate the present invention and, together with the description, further serve to explain the principles of the invention and to enable a person skilled in the pertinent art to make and use the invention.
In the drawings:
The present invention will now be described with reference to the accompanying drawings. In the drawings, like reference numbers indicate identical or functionally similar elements. Additionally, the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears.
A. Cross Point Architecture
B. Port Slice Operation with Wide Cell Encoding and Flow Control
C. Backplane Interface Adapter
D. Overall Operation of Backplane Interface Adapter
E. First Traffic Processing Path
F. Narrow Cell Format
G. Traffic Sorting
H. Wide Striped Cell Generation
I. Encoding Wide Striped Cells
J. Initial Block Encoding
K. End of Packet Encoding
L. Switching Fabric Transmit Arbitration
M. Cross Point Processing of Stripes
N. Second Traffic Processing Path
O. Cell Boundary Alignment
P. Packet Alignment
Q. Wide Striped Cell Size at Line Rate
R. IBT and Packet Processing
S. Narrow Cell and Packet Encoding Processes
T. Administrative Process and Error Control
U. Reset and Recovery Procedures
The present invention is a high-performance digital switch. Blades are coupled through serial pipes to a switching fabric. Serial link technology is used in the switching fabric. Serial data streams, rather than parallel data streams, are switched through a loosely striped switching fabric. Blades output serial data streams in the serial pipes. A serial pipe can be a number of serial links coupling a blade to the switching fabric. The serial data streams represent an aggregation of input serial data streams provided through physical ports to a respective blade. Each blade outputs serial data streams with in-band control information in multiple stripes to the switching fabric. In one embodiment, the serial data streams carry packets of data in wide striped cells across multiple loosely-coupled stripes. Wide striped cells are encoded. In-band control information is carried in one or more blocks of a wide striped cell.
In one implementation, each blade of the switch is capable of sending and receiving 50 gigabit per second full-duplex traffic across the backplane. This is done to assure line rate, wire speed and non-blocking across all packet sizes.
The high-performance switch according to the present invention can be used in any switching environment, including but not limited to, the Internet, an enterprise system, Internet service provider, and any protocol layer switching (such as, Layer 2, Layer 3, or Layers 4-7 switching).
The present invention is described in terms of this example environment. Description in these terms is provided for convenience only. It is not intended that the invention be limited to application in these example environments. In fact, after reading the following description, it will become apparent to a person skilled in the relevant art how to implement the invention in alternative environments known now or developed in the future.
II. Terminology
To more clearly delineate the present invention, an effort is made throughout the specification to adhere to the following term definitions as consistently as possible.
The terms “switch fabric” or “switching fabric” refer to a switchable interconnection between blades. The switch fabric can be located on a backplane, a blade, more than one blade, a separate unit from the blades, or on any combination thereof.
The term “packet processor” refers to any type of packet processor, including but not limited to, an Ethernet packet processor. A packet processor parses and determines where to send packets.
The term “serial pipe” refers to one or more serial links. In one embodiment, not intended to limit the invention, a serial pipe is a 10 Gb/s serial pipe and includes four 2.5 Gb/s serial links.
The term “serial link” refers to a data link or bus carrying digital data serially between points. A serial link at a relatively high bit rate can also be made of a combination of lower bit rate serial links.
The term “stripe” refers to one data slice of a wide cell. The term “loosely-coupled” stripes refers to the data flow in stripes which is autonomous with respect to other stripes. Data flow is not limited to being fully synchonized in each of the stripes, rather, data flow proceeds independently in each of the stripes and can be skewed relative to other stripes.
II. Digital Switch Architecture
An overview of the architecture of the switch 100 of the invention is illustrated in
In a preferred embodiment of the invention, switch 100 having 8 blades is capable of switching of 400 gigabits per second (Gb/s) full-duplex traffic. As used herein, all data rates are full-duplex unless indicated otherwise. Each blade 104 communicates data at a rate of 50 Gb/s over serial pipe 106.
Switch 100 is shown in further detail in
Each cross point 202A-202E is an 8-port cross point. In one example, each cross point 2202A-E receives eight 10 G streams of data. Each stream of data corresponds to a particular stripe. The stripe has data in a wide-cell format which includes, among other things, a destination port number (also called a destination slot number) and special in-band control information. The in-band control information includes special K characters, such as, a K0 character and K1 character. The K0 character delimits a start of new cell within a stripe. The K1 character delimits an end of a packet within the stripe. Such encoding within each stripe, allows each cross point 202A-202E to operate autonomously or independently of other cross points. In this way, the cross points 202A-202E and their associated stripes are loosely-coupled.
In each cross point 202, there are a set of data structures, such as data FIFOs (First in First out data structures). The data structures store data based on the source port and the destination port. In one embodiment, for an 8-port cross point, 56 data FIFOs are used. Each data FIFO stores data associated with a respective source port and destination port. Packets coming to each source port are written to the data FIFOs which correspond to a source port and a destination port associated with the packets. The source port is associated with the port (and port slice) on which the packets are received. The destination port is associated with a destination port or slot number which is found in-band in data sent in a stripe to a port.
In embodiments of the present invention, the switch size is defined as one cell and the cell size is defined to be either 8, 28, 48, 68, 88, 108, 128, or 148 bytes. Each port (or port slice) receives and sends serial data at a rate of 10 Gb/s from respective serial links. Each cross point 202A-202E has a 160 Gb/s switching capacity (160 Gb/s=10 Gb/s*8 ports*2 directions full-duplex). Such cell sizes, serial link data rate, and switching capacity are illustrative and not necessarily intended to limit the present invention. Cross-point architecture and operation is described further below.
In attempting to increase the throughput of switches, conventional wisdom has been to increase the width of data buses to increase the “parallel processing” capabilities of the switch and to increase clock rates. Both approaches, however, have met with diminishing returns. For example, very wide data buses are constrained by the physical limitations of circuit boards. Similarly, very high clock rates are limited by characteristics of printed circuit boards. Going against conventional wisdom, the inventors have discovered that significant increases in switching bandwidth could be obtained using serial link technology in the backplane.
In the preferred embodiment, each serial pipe 106 is capable of carrying full-duplex traffic at 50 Gb/s, and each serial link 204 is capable of carrying full-duplex traffic at 10 Gb/s. The result of this architecture is that each of the five cross points 202 combines five 10 gigabit per second serial links to achieve a total data rate of 50 gigabits per second for each serial pipe 106. Thus, the total switching capacity across backplane 102 for eight blades is 50 gigabits per second times eight times two (for duplex) or 800 gigabits per second. Such switching capacities have not been possible with conventional technology using synched parallel data buses in a switching fabric.
An advantage of such a switch having a 50 Gb/s serial pipe to backplane 102 from a blade 104 is that each blade 104 can support across a range of packet sizes four 10 Gb/s Ethernet packet processors at line rate, four Optical Channel OC-192C at line rate, or support one OC-768C at line rate. The invention is not limited to these examples. Other configurations and types of packet processors and can be used with the switch of the present invention as would be apparent to a person skilled in the art given this description.
Referring now to
Each packet processor 306 includes one or more physical ports. Each packet processor 306 receives inbound packets from the one or more physical ports, determines a destination of the inbound packet based on control information, provides local switching for local packets destined for a physical port to which the packet processor is connected, formats packets destined for a remote port to produce parallel data and switches the parallel data to an IBT 304. Each IBT 304 receives the parallel data from each packet processor 306. IBT 304 then converts the parallel data to at least one serial bit streams. IBT 304 provides the serial bit stream to BIA 302 via a pipe 308, described herein as one or more serial links. In a preferred embodiment, each pipe 308 is a 10 Gb/s XAUI interface.
In the example illustrated in
Also in the example of
BIA 302 receives the output of IBTs 304A-304D. Thus, BIA 302 receives 4 times 10 Gb/s of data. Or alternatively, 8 times 5 gigabit per second of data. BIA 302 runs at a clock speed of 156.25 MHz. With the addition of management overhead and striping, BIA 302 outputs 5 times 10 gigabit per second data streams to the five cross points 202 in backplane 102.
BIA 302 receives the serial bit streams from IBTs 304, determines a destination of each inbound packet based on packet header information, provides local switching between local IBTs 304, formats data destined for a remote port, aggregates the serial bit streams from IBTs 304 and produces an aggregate bit stream. The aggregated bit stream is then striped across the five cross points 202A-202E.
A. Cross Point Architecture
Port slice 402F is coupled to each of the seven other port slices 402A-402E and 402G-402H through links 420-426. Links 420-426 route data received in the other port slices 402A-402E and 402G-402H which has a destination port number (also called a destination slot number) associated with a port of port slice 402F (i.e. destination port number 5). Finally, port slice 402F includes a link 430 that couples the port associated with port slice 402F to the other seven port slices. Link 430 allows data received at the port of port slice 402F to be sent to the other seven port slices. In one embodiment, each of the links 420-426 and 430 between the port slices are buses to carry data in parallel within the cross point 202. Similar connections (not shown in the interest of clarity) are also provided for each of the other port slices 402A-402E, 402G and 402H.
Port slice 402F includes a receive synch FIFO module 515 coupled between deserializer receiver(s) 510 and accumulator 520. Receive synch FIFO module 515 stores data output from deserializer receivers 510 corresponding to port slice 402F. Accumulator 520 writes data to an appropriate data FIFO (not shown) in the other port slices 402A-402E, 402G, and 402H based on a destination slot or port number in a header of the received data.
Port slice 402F also receives data from other port slices 402A-402E, 402G, and 402H. This data corresponds to the data received at the other seven ports of port slices 402A-402E, 402G, and 402H which has a destination slot number corresponding to port slice 402F. Port slice 402F includes seven data FIFOs 530 to store data from corresponding port slices 402A-402E, 402G, and 402H. Accumulators (not shown) in the seven port slices 402A-402E, 402G, and 402H extract the destination slot number associated with port slice 402F and write corresponding data to respective ones of seven data FIFOs 530 for port slice 402F. As shown in
During operation, the FIFO RAMs accumulate data. After a data FIFO RAM has accumulated one cell of data, its corresponding FIFO controller generates a read request to FIFO read arbitrator 540. FIFO read arbitrator 540 processes read requests from the different FIFO controllers in a desired order, such as a round-robin order. After one cell of data is read from one FIFO RAM, FIFO read arbitrator 540 will move on to process the next requesting FIFO controller. In this way, arbitration proceeds to serve different requesting FIFO controllers and distribute the forwarding of data received at different source ports. This helps maintain a relatively even but loosely coupled flow of data through cross points 202.
To process a read request, FIFO read arbitrator 540 switches multiplexer 550 to forward a cell of data from the data FIFO RAM associated with the read request to dispatcher 560. Dispatcher 560 outputs the data to transmit synch FIFO 570. Transmit synch FIFO 570 stores the data until sent in a serial data stream by serializer transmitter(s) 580 to blade 104F.
B. Port Slice Operation with Wide Cell Encoding and Flow Control
According to a further embodiment, a port slice operates with respect to wide cell encoding and a flow control condition.
In step 2710, entries in receive synch FIFO 515 are managed. In one example, receive synch FIFO module 515 is an 8-entry FIFO with write pointer and read pointer initialized to be 3 entries apart. Receive synch FIFO module 515 writes 64-bit data from a SERDES deserialize receiver 510, reads 64-bit data from a FIFO with a clock signal and delivers data to accumulator 520, and maintains a three entry separation between read/write pointers by adjusting the read pointer when the separation becomes less than or equal to 1.
In step 2720, accumulator 520 receives two chunks of 32-bit data are received from receive synch FIFO 515. Accumulator 520 detects a special character K0 in the first bytes of first chunk and second chunk (step 2722). Accumulator 520 then extracts a destination slot number from the state field in the header if K0 is detected (step 2724).
As shown in
As shown in
In step 2746, a respective FIFO controller indicates to FIFO read arbitrator 540 if K0 has been read or FIFO RAM is empty. This indication is a read request for arbitration. In step 2748, a respective FIFO controller indicates to FIFO read arbitrator 540 whether K0 is aligned to the first 32-bit chunk or the second 32-bit chunk. When flow control from an output port is detected (such as when a predetermined flow control sequence of one or more characters is detected), FIFO controller stops requesting the FIFO read arbitrator 540 after the current cell is completely read from the FIFO RAM (step 2750).
As shown in
As shown in
C. Backplane Interface Adapter
To describe the structure and operation of the backplane interface adapter reference is made to components shown in
D. Overall Operation of Backplane Interface Adapter
As shown in
E. First Traffic Processing Path
Deserializer receiver(s) 602 receive narrow input cells carrying packets of data. These narrow input cells are output to deserializer receiver(s) 602 from packet processors and/or from integrated bus translators (IBTs) coupled to packet processors. In one example, four deserializer receivers 602 are coupled to four serial links (such as, links 308A-D, 318A-C described above in
F. Narrow Cell Format
G. Traffic Sorting
Traffic sorter 610 sorts received narrow input cells based on a destination slot identifier. Traffic sorter 610 routes narrow cells destined for the same blade as BIA 600 (also called local traffic) to destination queues 615. Narrow cells destined for other blades in a switch across the switching fabric (also called global traffic) are routed to wide cell generators 620.
H. Wide Striped Cell Generation
Wide cell generators 620 generate wide striped cells. The wide striped cells carry the packets of data received by BIA 600 in the narrow input cells. The wide cells extend across multiple stripes and include in-band control information in each stripe. In the interest of brevity, the operation of wide cell generators 620, 720 is further described with respect to a routine 1200 in
For each detected packet (step 1225), steps 1230-1240 are performed. In step 1230, wide cell generator 620, 720 encodes one or more new wide striped cells until data from all narrow input cells of the packet is distributed into the one or more new wide striped cells. This encoding is further described below with respect to routine 1400 and
In step 1230, wide cell generator 620 then writes the one or more new wide striped cells into a plurality of send queues 625. In the example of
I. Encoding Wide Striped Cells
According to a further feature of the present invention, system and method for encoding wide striped cells is provided. In one embodiment, wide cell generators 620, 720 each generate wide striped cells which are encoded (step 1230).
J. Initial Block Encoding
In step 1410, wide cell generator 620, 720 encodes an initial block of a start wide striped cell with initial cell encoding information. The initial cell encoding information includes control information (such as, a special K0 character) and state information provided in each subblock of an initial block of a wide striped cell.
In step 1420, wide cell generator(s) 620, 720 distribute initial bytes of packet data into available space in the initial block. In the example wide striped cell 1500 shown in
In step 1430, wide cell generator(s) 620, 720 distribute remaining bytes of packet data across one or more blocks in of the first wide striped cell (and subsequent wide cells). In the example wide striped cell 1500, maximum size of a wide striped cell is 160 bytes (8 blocks) which corresponds to a maximum of 148 bytes of data. In addition to the data bytes D0-D7 in the initial block, wide striped cell 1500 further has data bytes D8-D147 distributed in seven blocks (labeled in
In general, packet data continues to be distributed until an end of packet condition is reached or a maximum cell size is reached. Accordingly, checks are made of whether a maximum cell size is reached (step 1440) and whether the end of packet is reached (step 1450). If the maximum cell size is reached in step 1440 and more packet data needs to be distributed then control returns to step 1410 to create additional wide striped cells to carry the rest of the packet data. If the maximum cell size is not reached in step 1440, then an end of packet check is made (step 1450). If an end of packet is reached then the current wide striped cell being filled with packet data is the end wide striped cell. Note for small packets less than 148 bytes, than only one wide striped cell is needed. Otherwise, more than one wide striped cells are used to carry a packet of data across multiple stripes. When an end of packet is reached in step 1450, then control proceeds to step 1460.
K. End of Packet Encoding
In step 1460, wide cell generator(s) 620, 720 further encode an end wide striped cell with end of packet information that varies depending upon the degree to which data has filled a wide striped cell. In one encoding scheme, the end of packet information varies depending upon a set of end of packet conditions including whether the end of packet occurs in an initial cycle or subsequent cycles, at a block boundary, or at a cell boundary.
As shown in item 2 of
L. Switching Fabric Transmit Arbitration
In one embodiment, BIA 600 also includes switching fabric transmit arbitrator 630. Switching fabric transmit arbitrator 630 arbitrates the order in which data stored in the stripe send queues 625, 725 is sent by transmitters 640, 740 to the switching fabric. Each stripe send queue 625, 725 stores a respective group of wide striped cells corresponding to a respective originating source packet processor and a destination slot identifier. Each wide striped cell has one or more blocks across multiple stripes. During operation the switching fabric transmit arbitrator 630 selects a stripe send queue 625, 725 and pushes the next available cell to the transmitters 640, 740. In this way one full cell is sent at a time. (Alternatively, a portion of a cell can be sent.) Each stripe of a wide cell is pushed to the respective transmitter 640, 740 for that stripe. In one example, during normal operation, a complete packet is sent to any particular slot or blade from a particular packet processor before a new packet is sent to that slot from different packet processors. However, the packets for the different slots are sent during an arbitration cycle. In an alternative embodiment, other blades or slots are then selected in a round-robin fashion.
M. Cross Point Processing of Stripes Including Wide Cell Encoding
In on embodiment, switching fabric 645 includes a number n of cross point switches 202 corresponding to each of the stripes. Each cross point switch 202 (also referred to herein as a cross point or cross point chip) handles one data slice of wide cells corresponding to one respective stripe. In one example, five cross point switches 202A-202E are provided corresponding to five stripes. For clarity,
The operation of a cross point 202 and in particular a port slice 402F is now described with respect to an embodiment where stripes further include wide cell encoding and a flow control indication.
Port slice 402F also receives data from other port slices 402A-402E, 402G, and 402H. This data corresponds to the data received at the other seven ports of port slices 402A-402E, 402G, and 402H which has a destination slot number corresponding to port slice 402F. Port slice 402F includes seven data FIFOs 530 to store data from corresponding port slices 402A-402E, 402G, and 402H. Accumulators (not shown) in the seven port slices 402A-402E, 402G, and 402H extract the destination slot number associated with port slice 402F and write corresponding data to respective ones of seven data FIFOs 530 for port slice 402F. As shown in
During operation, the FIFO RAMs accumulate data. After a data FIFO RAM has accumulated one cell of data, its corresponding FIFO controller generates a read request to FIFO read arbitrator 540. FIFO read arbitrator 540 processes read requests from the different FIFO controllers in a desired order, such as a round-robin order. After one cell of data is read from one FIFO RAM, FIFO read arbitrator 540 will move on to process the next requesting FIFO controller. In this way, arbitration proceeds to serve different requesting FIFO controllers and distribute the forwarding of data received at different source ports. This helps maintain a relatively even but loosely coupled flow of data through cross points 202.
To process a read request, FIFO read arbitrator 540 switches multiplexer 550 to forward a cell of data from the data FIFO RAM associated with the read request to dispatcher 560. Dispatcher 560 outputs the data to transmit synch FIFO 570. Transmit synch FIFO 570 stores the data until sent in a serial data stream by serializer transmitter(s) 580 to blade 104F.
Cross point operation according to the present invention is described farther below with respect to a further embodiment involving wide cell encoding and flow control.
N. Second Traffic Processing Path
As shown in
Translators 680 translate the received wide striped cells to narrow input cells carrying the packets of data. Serializer transmitters 692 transmit the narrow input cells to corresponding source packet processors or IBTs.
BIA 600 further includes stripe interfaces 660 (also called stripe interface modules), stripe receive synchronization queues (685), and controller 670 coupled between deserializer receivers 650 and a controller 670. Each stripe interface 660 sorts received subblocks in each stripe based on source packet processor identifier and originating slot identifier information and stores the sorted received subblocks in the stripe receive synchronization queues 685.
Controller 670 includes an arbitrator 672, a striped-based wide cell assembler 674, and an administrative module 676. Arbitrator 672 arbitrates an order in which data stored in stripe receive synchronization queues 685 is sent to striped-based wide cell assembler 674. Striped-based wide cell assembler 674 assembles wide striped cells based on the received subblocks of data. A narrow/wide cell translator 680 then translates the arbitrated received wide striped cells to narrow input cells carrying the packets of data. Administrative module 676 is provided to carry out flow control, queue threshold level detection, and error detection (such as, stripe synchronization error detection), or other desired management or administrative functionality.
A second level of arbitration is also provided according to an embodiment of the present invention. BIA 600 further includes destination queues 615 and a local destination transmit arbitrator 690 in the second path. Destination queues 615 store narrow cells sent by traffic sorter 610 (from the first path) and the narrow cells translated by the translator 680 (from the second path). Local destination transmit arbitrator 690 arbitrates an order in which narrow input cells stored in destination queues 690 is sent to serializer transmitters 692. Finally, serializer transmitters 692 then transmit the narrow input cells to corresponding IBTs and/or source packet processors (and ultimately out of a blade through physical ports).
In the second traffic path, deserializer receiver 950 is coupled to cross clock domain synchronizer 952. Deserializer receiver 950 converts serial data slices of a stripe (e.g., subblockes) to parallel data. Cross clock domain synchronizer 952 synchronizes the parallel data.
Stripe interface 960 has a decoder 962 and sorter 964 to decode and sort received subblocks in each stripe based on source packet processor identifier and originating slot identifier information. Sorter 964 then stores the sorted received subblocks in stripe receive synchronization queues 965. Five groups of 56 stripe receive synchronization queues 965 are provided in total. This allows one queue to be dedicated for each group of subblocks received from a particular source per global blade (up to 8 source packet processors per blade for seven blades not including the current blade).
Arbitrator 672 arbitrates an order in which data stored in stripe receive synchronization queues 685 sent to striped-based wide cell assembler 674. Striped-based wide cell assembler 674 assembles wide striped cells based on the received subblocks of data. A narrow/wide cell translator 680 then translates the arbitrated received wide striped cells to narrow input cells carrying the packets of data as described above in
Destination queues include local destination queues 982 and backplane traffic queues 984. Local destination queues 982 store narrow cells sent by local traffic sorter 716. Backplane traffic queues 984 store narrow cells translated by the translator 680. Local destination transmit arbitrator 690 arbitrates an order in which narrow input cells stored in destination queues 982, 984 is sent to serializer transmitters 992. Finally, serializer transmitters 992 then transmit the narrow input cells to corresponding IBTs and/or source packet processors (and ultimately out of a blade through physical ports).
O. Cell Boundary Alignment
P. Packet Alignment
Q. Wide Striped Cell Size at Line Rate
In one example, a wide cell has a maximum size of eight blocks (160 bytes) which can carry a 148 bytes of payload data and 12 bytes of in-band control information. Packets of data for full-duplex traffic can be carried in the wide cells at a 50 Gb/sec rate through the digital switch.
R. IBT and Packet Processing
The integrated packet controller (IPC) and integrated giga controller (IGC) functions are provided with a bus translator, described above as the IPC/IGC Bus Translator (IBT) 304. In one embodiment, the IBT is an ASIC that bridges one or more IPC/IC ASIC. In such an embodiment, the IBT translates two ⅘ gig parallel stream into one 10 Gbps serial stream. The parallel interface can be the backplane interface of the IPC/IGC ASICs. The one 10 Gbps serial stream can be further processed, for example, as described herein with regard to interface adapters and striping.
Additionally, IBT 304 can be configured to operate with other architectures as would be apparent to one skilled in the relevant art(s) based at least on the teachings herein. For example, the IBT 304 can be implemented in packet processors using 10 GE and OC-192 configurations. The functionality of the IBT 304 can be incorporated within existing packet processors or attached as an add-on component to a system.
In
More specifically, the bus translator 1702 translates data 1704 into data 1706 and data 1706 into data 104. The data 1706 is received by transceiver(s) 1710 is forwarded to a translator 1712. The translator 1712 parses and encodes the data 1706 into a desired format.
Here, the translator 1712 translates the data 1706 into the format of the data 1704. The translator 1712 is managed by an administration module 1718. One or more memory pools 1716 store the information of the data 1706 and the data 1704. One or more clocks 1714 provide the timing information to the translation operations of the translator 1712. Once the translator 1712 finishes translating the data 1706, it forwards the newly formatted information as the data 1704 to the transceiver(s) 1708. The transceiver(s) 1708 forward the data 1704.
As one skilled in the relevant art would recognize based on the teachings described herein, the operational direction of bus translator 1702 can be reversed and the data 1704 received by the bus translator 1702 and the data 1706 forwarded after translation.
For ease of illustration, but without limitation, the process of translating the data 1706 into the data 1704 is herein described as receiving, reception, and the like. Additionally, for ease of illustration, but without limitation, the process of translating the data 1704 into the data 1706 is herein described as transmitting, transmission, and the like.
In
The packet decoders 1810 receive the packets from the receivers 1808. The packet decoders 1810 parse the information from the packets. In one embodiment, as is described below in additional detail, the packet decoders 1810 copy the payload information from each packet as well as the additional information about the packet, such as time and place of origin, from the start of packet (SOP) and the end of packet (EOP) sections of the packet. The packet decoders 1810 forward the parsed information to memory pool(s) 1812. In one embodiment, the bus translator 1802 includes more than one memory pool 1812. In an alternative embodiment, alternate memory pool(s) 1818 can be sent the information. In an additional embodiment, the packet decoder(s) 1810 can forward different types of information, such as payload, time of delivery, origin, and the like, to different memory pools of the pools 1812 and 1818.
Reference clock 1820 provides timing information to the packet decoder(s) 1810. In one embodiment, reference clock 1820 is coupled to the IPC/IGC components sending the packets through the connections 1804a-n. In another embodiment, the reference clock 1820 provides reference and timing information to all the parallel components of the bus translator 1802.
Cell encoder(s) 1814 receives the information from the memory pool(s) 1812. In an alternative embodiment, the cell encoder(s) 1814 receives the information from the alternative memory pool(s) 1818. The cell encoder(s) 1814 formats the information into cells.
In the description that follows, these cells are also referred to as narrow cells. Furthermore, the cell encoder(s) 1814 can be configured to format the information into one or more cell types. In one embodiment, the cell format is a fixed size. In another embodiment, the cell format is a variable size.
The cell format is described in detail below with regard to cell encoding and decoding processes of
The cell encoder(s) 1814 forwards the cells to transmitter(s) 1816. The transmitter(s) 1816 receive the cells and transmit the cells through interface connections 1806a-n.
Reference clock 1828 provides timing information to the cell encoder(s) 1814. In one embodiment, reference clock 1828 is coupled to the interface adapter components receiving the cells through the connections 1806a-n. In another embodiment, the reference clock 1828 provides reference and timing information to all the serial components of the bus translator 1802.
Flow controller 1822 measures and controls the incoming packets and outgoing cells by determining the status of the components of the bus translator 1802 and the status of the components connected to the bus translator 1802. Such components are previously described herein and additional detail is provided with regard to the interface adapters of the present invention.
In one embodiment, the flow controller 1822 controls the traffic through the connection 1806 by asserting a ready signal and de-asserting the ready signal in the event of an overflow in the bus translator 1802 or the IPC/IGC components further connected.
Administration module 1824 provides control features for the bus translator 1802. In one embodiment, the administration module 1824 provides error control and power-on and reset functionality for the bus translator 1802.
The cell decoders 1912 receive the cells from the synchronization module 1910. The cell decoders 1912 parse the information from the cells. In one embodiment, as is described below in additional detail, the cell decoders 1912 copy the payload information from each cell as well as the additional information about the cell, such as place of origin, from the slot and state information section of the cell.
In one embodiment, the cell format can be fixed. In another embodiment, the cell format can be variable. In yet another embodiment, the cells received by the bus translator 1902 can be of more than one cell format. The bus translator 1902 can be configured to decode these cell format as one skilled in the relevant art would recognize based on the teachings herein. Further details regarding the cell formats is described below with regard to the cell encoding processes of the present invention.
The cell decoders 1912 forward the parsed information to memory pool(s) 1914. In one embodiment, the bus translator 1902 includes more than one memory pool 1914. In an alternative embodiment, alternate memory pool(s) 1916 can be sent the information. In an additional embodiment, the cell decoder(s) 1912 can forward different types of information, such as payload, time of delivery, origin, and the like, to different memory pools of the pools 1914 and 1916.
Reference clock 1922 provides timing information to the cell decoder(s) 1912. In one embodiment, reference clock 1922 is coupled to the interface adapter components sending the cells through the connections 1904a-n. In another embodiment, the reference clock 1922 provides reference and timing information to all the serial components of the bus translator 1902.
Packet encoder(s) 1918 receive the information from the memory pool(s) 1914. In an alternative embodiment, the packet encoder(s) 1918 receive the information from the alternative memory pool(s) 1916. The packet encoder(s) 1918 format the information into packets.
The packet format is determined by the configuration of the IPC/IGC components and the requirements for the system.
The packet encoder(s) 1918 forwards the packets to transmitter(s) 1920. The transmitter(s) 1920 receive the packets and transmit the packets through interface connections 1906a-n.
Reference clock 1928 provides timing information to the packet encoder(s) 1918. In one embodiment, reference clock 1928 is coupled to the IPC/IGC components receiving the packets through the connections 1906a-n. In another embodiment, the reference clock 1928 provides reference and timing information to all the parallel components of the bus translator 1902.
Flow controller 1926 measures and controls the incoming cells and outgoing packets by determining the status of the components of the bus translator 1902 and the status of the components connected to the bus translator 1902. Such components are previously described herein and additional detail is provided with regard to the interface adapters of the present invention.
In one embodiment, the flow controller 1926 controls the traffic through the connection 1906 by asserting a ready signal and de-asserting the ready signal in the event of an overflow in the bus translator 1902 or the IPC/IGC components farther connected.
Administration module 1924 provides control features for the bus translator 1902. In one embodiment, the administration module 1924 provides error control and power-on and reset functionality for the bus translator 1902.
In
In terms of packet processing, packets are received by the bus translator 2002 by receivers 2012. The packets are processed into cells and forwarded to a serializer/deserializer (SERDES) 2026. SERDES 2026 acts as a transceiver for the cells being processed by the bus translator 2002. The SERDES 2026 transmits the cells via interface connection 2006.
In terms of cell processing, cells are received by the bus translator 2002 through the interface connection 2008 to the SERDES 2026. The cells are processed into packets and forwarded to transmitters 2036. The transmitters 2036 forward the packets to the IPC/IGC components through interface connections 2010a-n.
The reference clocks 2040 and 2048 are similar to those previously described in
The above-described separation of serial and parallel operations is a feature of embodiments of the present invention. In such embodiments, the parallel format of incoming and leaving packets at ports 2014a-n and 2038a-b, respectively, is remapped into a serial cell format at the SERDES 2026.
Furthermore, according to embodiments of the present invention, the line rates of the ports 2014a-n have a shared utilization limited only by the line rate of output 2006. Similarly for ports 2038a-n and input 2008.
The remapping of parallel packets into serial cells is described in further detail herein, more specifically with regard to
In
Administration module 2140 operates as previously described. As shown, the administration module 2140 includes an administration control element and an administration register. The administration control element monitors the operation of the bus translator 2102 and provides the reset and power-on functionality as previously described with regard to
The reference clocks 2134 and 2136 are similar to those previously described in
As shown in
Additionally, memory pool 2130 includes two pairs of FIFOs. The memory pool 2130 performs as previously described memory pools in
Interface connections 2106 and 2108 connect previously described interface adapters to the bus translator 2102 through the SERDES 2124. In one embodiment, the connections 2106 and 2108 are serial links. In another embodiment, the serial links are divided four lanes.
In one embodiment, the bus translator 2102 is an IBT 304 that translates one or more 4 Gbps parallel IPC/IGC components into four 3.125 Gbps serial XAUI interface links or lanes. In one embodiment, the back planes are the IPC/IGC interface connections. The bus translator 2102 formats incoming data into one or more cell formats.
In one embodiment, the cell format can be a four byte header and a 32 byte data payload. In a further embodiment, each cell is separated by a special K character into the header. In another embodiment, the last cell of a packet is indicated by one or more special K1 characters.
The cell formats can include both fixed length cells and variable length cells. The 36 bytes (4 byte header plus 32 byte payload) encoding is an example of a fixed length cell format. In an alternative embodiment, cell formats can be implemented where the cell length exceeds the 36 bytes (4 bytes+32 bytes) previously described.
In
In one embodiment, the cell encoder 2160 merges the data from each of the packet decoders 2150a-b into one 10 Gbps data stream to the interface adapter. The cell encoder 2160 merges the data by interleaving the data at each cell boundary. Each cell boundary is determined by the special K characters.
According to one embodiment, the received packets are 32 bit aligned, while the parallel interface of the SERDES elements is 64 bit wide.
In practice it can be difficult to achieve line rate for any packet length. Line rate means maintaining the same rate of output in cells as the rate at which packets are being received. Packets can have a four byte header overhead (SOP) and a four byte tail overhead (EOP). Therefore, the bus translators 2102 must parse the packets without the delays of typical parsing and routing components. More specifically, the bus translators 2102 formats parallel data inot cell format using special K characters, as described in more detail below, to merge state information and slot information (together, control information) in band with the data streams. Thus, in one embodiment, each 32 bytes of cell data is accompanied by a four byte header.
In an additional embodiment, as shown in
Although a separate native mode data path is not shown for cell to packet translation, one skilled in the relevant art would recognize how to accomplish it based at least on the teachings described herein. For example, by configuring two FIFOs for dedicated storage of 100 Gbps link information. In one embodiment, however, the bus translator 2102 processes native mode and non-native mode data paths in a shared operation as shown in
In an additional embodiment, where there is a zero body cell format being received by the interface adapter or BIA, the IBT 304 holds one last data transfer for each source slot. When it receives the EOP with the zero body cell format, the last one or two transfers are released to be transmitted from the parallel interface.
S. Narrow Cell and Packet Encoding Processes
According to one embodiment of the present invention, the cell includes a special character K02190; a control information 2194; optionally, one or more reserved 2196a-b; and data 2198a-n. In an alternate embodiment, data 2198a-n can contain more than D0-D31.
In one embodiment, the four rows or slots indicated in
As previously described herein, the IBT 304 transmits and receives cells to and from the BIA 302 through the XAUI interface. The IBT 304 transmits and receives packets to and from the IPC/IGC components, as well as other controller components (i.e., 10 GE packet processor) through a parallel interface. The packets are segmented into cells which consist of a four byte header followed by 32 bytes of data. The end of packet is signaled by K1 special character on any invalid data bytes within four byte of transfer or four K1 on all XAUI lanes. In one embodiment, each byte is serialized onto one XAUI lane. The following table illustrates in a right to left formation a byte by byte representation of a cell according to one embodiment of the present invention:
The packets are formatted into cells that consist of a header plus a data payload. The 4 bytes of header takes one cycle or row on four XAUI lanes. It has K0 special character on Lane0 to indicate that current transfer is a header. The control information starts on Lane1 of a header.
In one embodiment, the IBT 304 accepts two IPC/IGC back plane buses and translates them into one 10 Gbps serial stream.
In
In step 2204, the IBT 304 determines the port types through which it will be receiving packets. In one embodiment, the ports are configured for 4 Gbps traffic from IPC/IGC components. The process immediately proceeds to step 2206.
In step 2206, the IBT 304 selects a cell format type based on the type of traffic it will be processing. In one embodiment, the IBT 304 selects the cell format type based in part on the port type determination of step 2204. The process immediately proceeds to step 2208.
In step 2208, the IBT 304 receives one or more packets from through its ports from the interface connections, as previously described. The rate at which packets are delivered depends on the components sending the packets. The process immediately proceeds to step 2210.
In step 2210, the IBT 304 parses the one or more packets received in step 2208 for the information contained therein. In one embodiment, the packet decoder(s) of the IBT 304 parse the packets for the information contained within the payload section of the packet, as well as the control or routing information included with the header for that each given packet. The process immediately proceeds to step 2212.
In step 2212, the IBT 304 optionally stores the information parsed in step 2210. In one embodiment, the memory pool(s) of the IBT 304 are utilized to store the information. The process immediately proceeds to step 2214.
In step 2214, the IBT 304 formats the information into one or more cells. In one embodiment, the cell encoder(s) of the IBT 304 access the information parsed from the one or more packets. The information includes the data being trafficked as well as slot and state information (i.e., control information) about where the data is being sent. As previously described, the cell format includes special characters which are added to the information. The process immediately proceeds to step 2216.
In step 2216, the IBT 304 forwards the formatted cells. In one embodiment, the SERDES of the IBT 304 receives the formatted cells and serializes them for transport to the BIA 302 of the present invention. The process continues until instructed otherwise.
In
In step 2304, the IBT 304 determines the port types through which it will be receiving packets. The process immediately proceeds to step 2306.
In step 2306, the IBT 304 determines if the port type will, either individually or in combination, exceed the threshold that can be maintained. In other words, the IBT 304 checks to see if it can match the line rate of incoming packets without reaching the internal rate maximum. If it can, then the process proceeds to step 2310. In not, then the process proceeds to step 2308.
In step 2308, given that the IBT 304 has determined that it will be operating at its highest level, the IBT 304 selects a variable cell size that will allow it to reduce the number of cells being formatted and forwarded in the later steps of the process. In one embodiment, the cell format provides for cells of whole integer multiples of each of the one or more packets received. In another embodiment, the IBT 304 selects a cell format that provides for a variable cell size that allows for maximum length cells to be delivered until the packet is completed. For example, if a given packet is 2.3 cell lengths, then three cells will be formatted, however, the third cell will be a third that is the size of the preceding two cells. The process immediately proceeds to step 2312.
In step 2310, given that the IBT 304 has determined that it will not be operating at its highest level, the IBT 304 selects a fixed cell size that will allow the IBT 304 to process information with lower processing overhead. The process immediately proceeds to step 2312.
In step 2312, the IBT 304 receives one or more packets. The process immediately proceeds to step 2314.
In step 2314, the IBT 304 parses the control information from each of the one or more packets. The process immediately proceeds to step 2316.
In step 2316, the IBT 304 determines the slot and state information for each of the one or more packets. In one embodiment, the slot and state information is determined in part from the control information parsed from each of the one or more packets. The process immediately proceeds to step 2318.
In step 2318, the IBT 304 stores the slot and state information. The process immediately proceeds to step 2320.
In step 2320, the IBT 304 parses the payload of each of the one or more packets for the data contained therein. The process immediately proceeds to step 2322.
In step 2322, the IBT 304 stores the data parsed from each of the one or more packets. The process immediately proceeds to step 2324.
In step 2324, the IBT 304 accesses the control information. In one embodiment, the cell encoder(s) of the IBT 304 access the memory pool(s) of the IBT 304 to obtain the control information. The process immediately proceeds to step 2326.
In step 2326, the IBT 304 accesses the data parsed from each of the one or more packets. In one embodiment, the cell encoder(s) of the IBT 304 access the memory pool(s) of the IBT 304 to obtain the data. The process immediately proceeds to step 2328.
In step 2328, the IBT 304 constructs each cell by inserting a special character at the beginning of the cell currently being constructed. In one embodiment, the special character is K0. The process immediately proceeds to step 2330.
In step 2330, the IBT 304 inserts the slot information. In one embodiment, the IBT 304 inserts the slot information into the next lane, such as space 2194. The process immediately proceeds to step 2332.
In step 2332, the IBT 304 inserts the state information. In one embodiment, the IBT 304 inserts the state information into the next lane after the one used for the slot information, such as reserved 2196a. The process immediately proceeds to step 2334.
In step 2334, the IBT 304 inserts the data. The process immediately proceeds to step 2336.
In step 2336, the IBT 304 determines if there is additional data to be formatted. For example, if there is remaining data from a given packet. If so, then the process loops back to step 2328. If not, then the process immediately proceeds to step 2338.
In step 2338, the IBT 304 inserts the special character that indicated the end of the cell transmission (of one or more cells). In one embodiment, when the last of a cells is transmitted, the special character is K1. The process proceeds to step 2340.
In step 2340, the IBT 304 forwards the cells. The process continues until instructed otherwise.
In
In step 2404, the IBT 304 receives one or more cells. In one embodiment, the cells are received by the SERDES of the IBT 304 and forwarded to the cell decoder(s) of the IBT 304. In another embodiment, the SERDES of the IBT 304 forwards the cells to a synchronization buffer or queue that temporarily holds the cells so that their proper order can be maintained. These steps are described below with regard to steps 2406 and 2408. The process immediately proceeds to step 2406.
In step 2406, the IBT 304 synchronizes the one or more cells into the proper order. The process immediately proceeds to step 2408.
In step 2408, the IBT 304 optionally checks the one or more cells to determine if they are in their proper order.
In one embodiment, steps 2506, 2508, and 2510 are performed by a synchronization FIFO. The process immediately proceeds to step 2410.
In step 2410, the IBT 304 parses the one or more cells into control information and payload data. The process immediately proceeds to step 2412.
In step 2412, the IBT 304 stores the control information payload data. The process immediately proceeds to step 2414.
In step 2414, the IBT 304 formats the information into one or more packets. The process immediately proceeds to step 2416.
In step 2416, the IBT 304 forwards the one or more packets. The process continues until instructed otherwise.
In
In step 2504, the IBT 304 receives one or more cells. The process immediately proceeds to step 2506.
In step 2506, the IBT 304 optionally queues the one or more cells. The process immediately proceeds to step 2508.
In step 2508, the IBT 304 optionally determines if the cells are arriving in the proper order. If so, then the process immediately proceeds to step 2512. If not, then the process immediately proceeds to step 2510.
In step 2510. The IBT 304 holds one or more of the one or more cells until the proper order is regained. In one embodiment, in the event that cells are lost, the IBT 304 provide error control functionality, as described herein, to abort the transfer and/or have the transfer re-initiated. The process immediately proceeds to step 2514.
In step 2512, the IBT 304 parses the cell for control information. The process immediately proceeds to step 2514.
In step 2514, the IBT 304 determines the slot and state information. The process immediately proceeds to step 2516.
In step 2516, the IBT 304 stores the slot and state information. The process immediately proceeds to step 2518.
In one embodiment, the state and slot information includes configuration information as shown in the table below:
In one embodiment, the IBT 304 has configuration registers. They are used to enable Backplane and IPC/IGC destination slots.
In step 2518, the IBT 304 parses the cell for data. The process immediately proceeds to step 2520.
In step 2520, the IBT 304 stores the data parsed from each of the one or more cells. The process immediately proceeds to step 2522.
In step 2522, the IBT 304 accesses the control information. The process immediately proceeds to step 2524.
In step 2524, the IBT 304 access the data. The process immediately proceeds to step 2526.
In step 2526, the IBT 304 forms one or more packets. The process immediately proceeds to step 2528.
In step 2528, the IBT 304 forwards the one or more packets. The process continues until instructed otherwise.
T. Administrative Process and Error Control
In
In step 2604, the IBT 304 determines the status of its internal components. The process immediately proceeds to step 2606.
In step 2606, the IBT 304 determines the status of its links to external components. The process immediately proceeds to step 2608.
In step 2608, the IBT 304 monitors the operations of both the internal and external components. The process immediately proceeds to step 2610.
In step 2610, the IBT 304 monitors the registers for administrative commands. The process immediately proceeds to step 2612.
In step 2612, the IBT 304 performs resets of given components as instructed. The process immediately proceeds to step 2614.
In step 2614, the IBT 304 configures the operations of given components. The process continues until instructed otherwise.
In one embodiment, any errors are detected on the receiving side of the BIA 302 are treated in a fashion identical to the error control methods described herein for errors received on the Xpnt 202 from the BIA 302. In operational embodiments where the destination slot cannot be know under certain conditions by the BIA 302, the following process is followed:
In the event that an error is detected on the receiving side of the IBT 304, it is treated as if the error was seen by the BIA 302 from IBT 304. The following process will be used:
The following reset procedure will be followed to get the SERDES in sync. An external reset will be asserted to the SERDES core when a reset is applied to the core. The duration of the reset pulse for the SERDES need not be longer than 10 cycles. After reset pulse, the transmitter and the receiver of the SERDES will sync up to each other through defined procedure. It is assumed that the SERDES will be in sync once the core comes out of reset. For this reason, the reset pulse for the core must be considerably greater than the reset pulse for the SERDES core.
The core will rely on software interaction to get the core in sync. Once the BIA 302, 600, IBT 304, and Xpnt 202 come out of reset, they will continuously send lane synchronization sequence. The receiver will set a software visible bit stating that its lane is in sync. Once software determines that the lanes are in sync, it will try to get the stripes in sync. This is done through software which will enable continuously sending of stripe synchronization sequence. Once again, the receiving side of the BIA 302 will set a bit stating that it is in sync with a particular source slot. Once software determines this, it will enable transmit for the BIA 302, XPNT 202 and IBT 304.
IV. Control Logic
Functionality described above with respect to the operation of switch 100 can be implemented in control logic. Such control logic can be implemented in software, firmware, hardware or any combination thereof.
While specific embodiments of the present invention have been described above, it should be understood that they have been presented by way of example only, and not limitation. It will be understood by those skilled in the art that various changes in form and details maybe made therein without departing from the spirit and scope of the invention as defined in the appended claims. Thus, the breadth and scope of the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.
This application is a continuation application of U.S. application Ser. No. 11/804,977, filed May 21,2007, which is a continuation of application Ser. No. 09/855,038, filed May 15, 2001, now U.S. Pat. No. 7,236,490, issued Jun. 26, 2007, which claimed the benefit of Provisional. App. 60/249,871, filed Nov. 17, 2000, both of which are incorporated herein by reference in their respective entireties. Other related applications include U.S. application Ser. No. 09/988,066 (published on Aug. 8, 2002 as U.S. Publication No. 2002/0105966), Ser. No. 09/855,015 (now U.S. Pat. No. 7,356,030, issued Apr. 8, 2008), Ser. No. 10/810,301 (now U.S. Pat. No. 7,203,194, issued Apr. 10, 2007), and Ser. No. 11/724,965. Related patents include U.S. Pat. Nos. 6,697,368, 6,735,218, 6,901,072, 7,203,194, and 7,206,238.
| Number | Name | Date | Kind |
|---|---|---|---|
| 3866175 | Seifert, Jr. et al. | Feb 1975 | A |
| 4628480 | Floyd | Dec 1986 | A |
| 4667323 | Engdahl et al. | May 1987 | A |
| 4683564 | Young et al. | Jul 1987 | A |
| 4698748 | Juzswik et al. | Oct 1987 | A |
| 4723243 | Joshi et al. | Feb 1988 | A |
| 4754482 | Weiss | Jun 1988 | A |
| 4791629 | Burns et al. | Dec 1988 | A |
| 4794629 | Pastyr et al. | Dec 1988 | A |
| 4807280 | Posner et al. | Feb 1989 | A |
| 4876681 | Hagiwara et al. | Oct 1989 | A |
| 4896277 | Vercellotti et al. | Jan 1990 | A |
| 4985889 | Frankish et al. | Jan 1991 | A |
| 5101404 | Kunimoto et al. | Mar 1992 | A |
| 5136584 | Hedlund | Aug 1992 | A |
| 5195181 | Bryant et al. | Mar 1993 | A |
| 5208856 | Leduc et al. | May 1993 | A |
| 5224108 | McDysan et al. | Jun 1993 | A |
| 5231633 | Hluchyj et al. | Jul 1993 | A |
| 5280582 | Yang et al. | Jan 1994 | A |
| 5282196 | Clebowicz | Jan 1994 | A |
| 5287477 | Johnson et al. | Feb 1994 | A |
| 5301192 | Henrion | Apr 1994 | A |
| 5307345 | Lozowick et al. | Apr 1994 | A |
| 5323386 | Wiher et al. | Jun 1994 | A |
| 5365512 | Combs et al. | Nov 1994 | A |
| 5377189 | Clark | Dec 1994 | A |
| 5390173 | Spinney et al. | Feb 1995 | A |
| 5392279 | Taniguchi | Feb 1995 | A |
| 5406643 | Burke et al. | Apr 1995 | A |
| 5408469 | Opher et al. | Apr 1995 | A |
| 5430442 | Kaiser et al. | Jul 1995 | A |
| 5436893 | Barnett | Jul 1995 | A |
| 5461615 | Henrion | Oct 1995 | A |
| 5490258 | Fenner | Feb 1996 | A |
| 5506840 | Pauwels et al. | Apr 1996 | A |
| 5521923 | Willmann et al. | May 1996 | A |
| 5546385 | Caspi et al. | Aug 1996 | A |
| 5550816 | Hardwick et al. | Aug 1996 | A |
| 5563948 | Diehl et al. | Oct 1996 | A |
| 5566170 | Bakke et al. | Oct 1996 | A |
| 5598410 | Stone | Jan 1997 | A |
| 5600795 | Du | Feb 1997 | A |
| 5619497 | Gallagher et al. | Apr 1997 | A |
| 5640504 | Johnson, Jr. | Jun 1997 | A |
| 5646878 | Samra | Jul 1997 | A |
| 5663952 | Gentry, Jr. | Sep 1997 | A |
| 5663959 | Nakagawa | Sep 1997 | A |
| 5666353 | Klausmeier et al. | Sep 1997 | A |
| 5721819 | Galles et al. | Feb 1998 | A |
| 5732080 | Ferguson et al. | Mar 1998 | A |
| 5740176 | Gupta et al. | Apr 1998 | A |
| 5745708 | Weppler et al. | Apr 1998 | A |
| 5751710 | Crowther et al. | May 1998 | A |
| 5802287 | Rostoker et al. | Sep 1998 | A |
| 5815146 | Youden et al. | Sep 1998 | A |
| 5818816 | Chikazawa et al. | Oct 1998 | A |
| 5835496 | Yeung et al. | Nov 1998 | A |
| 5838684 | Wicki et al. | Nov 1998 | A |
| 5862350 | Coulson | Jan 1999 | A |
| 5867675 | Lomelino et al. | Feb 1999 | A |
| 5870538 | Manning et al. | Feb 1999 | A |
| 5872769 | Caldara et al. | Feb 1999 | A |
| 5872783 | Chin | Feb 1999 | A |
| 5875200 | Glover et al. | Feb 1999 | A |
| 5907566 | Benson et al. | May 1999 | A |
| 5907660 | Inoue et al. | May 1999 | A |
| 5909686 | Muller et al. | Jun 1999 | A |
| 5915094 | Kouloheris et al. | Jun 1999 | A |
| 5920566 | Hendel et al. | Jul 1999 | A |
| 5920886 | Feldmeier | Jul 1999 | A |
| 5936939 | Des Jardins et al. | Aug 1999 | A |
| 5936966 | Ogawa et al. | Aug 1999 | A |
| 5956347 | Slater | Sep 1999 | A |
| 5999528 | Chow et al. | Dec 1999 | A |
| 6000016 | Curtis et al. | Dec 1999 | A |
| 6016310 | Muller et al. | Jan 2000 | A |
| 6023471 | Haddock et al. | Feb 2000 | A |
| 6035414 | Okazawa et al. | Mar 2000 | A |
| 6038288 | Thomas et al. | Mar 2000 | A |
| 6067298 | Shinohara | May 2000 | A |
| 6067606 | Holscher et al. | May 2000 | A |
| 6076115 | Sambamurthy et al. | Jun 2000 | A |
| 6081522 | Hendel et al. | Jun 2000 | A |
| 6088356 | Hendel et al. | Jul 2000 | A |
| 6094434 | Kotzur et al. | Jul 2000 | A |
| 6104696 | Kadambi et al. | Aug 2000 | A |
| 6104700 | Haddock et al. | Aug 2000 | A |
| 6108306 | Kalkunte et al. | Aug 2000 | A |
| 6118787 | Kalkunte et al. | Sep 2000 | A |
| 6125417 | Bailis et al. | Sep 2000 | A |
| 6128666 | Muller et al. | Oct 2000 | A |
| 6144668 | Bass et al. | Nov 2000 | A |
| 6147996 | Laor et al. | Nov 2000 | A |
| 6151301 | Holden | Nov 2000 | A |
| 6151497 | Yee et al. | Nov 2000 | A |
| 6154446 | Kadambi et al. | Nov 2000 | A |
| 6157643 | Ma | Dec 2000 | A |
| 6160809 | Adiletta et al. | Dec 2000 | A |
| 6172990 | Deb et al. | Jan 2001 | B1 |
| 6178520 | DeKoning et al. | Jan 2001 | B1 |
| 6181699 | Crinion et al. | Jan 2001 | B1 |
| 6185222 | Hughes | Feb 2001 | B1 |
| 6195335 | Calvignac et al. | Feb 2001 | B1 |
| 6201492 | Amar et al. | Mar 2001 | B1 |
| 6222845 | Shue et al. | Apr 2001 | B1 |
| 6243667 | Kerr et al. | Jun 2001 | B1 |
| 6249528 | Kothary | Jun 2001 | B1 |
| 6263374 | Olnowich et al. | Jul 2001 | B1 |
| 6272144 | Berenbaum et al. | Aug 2001 | B1 |
| 6304903 | Ward | Oct 2001 | B1 |
| 6320859 | Momirov | Nov 2001 | B1 |
| 6333929 | Drottar et al. | Dec 2001 | B1 |
| 6335932 | Kadambi et al. | Jan 2002 | B2 |
| 6335935 | Kadambi et al. | Jan 2002 | B2 |
| 6343072 | Bechtolsheim et al. | Jan 2002 | B1 |
| 6351143 | Guccione et al. | Feb 2002 | B1 |
| 6356550 | Williams | Mar 2002 | B1 |
| 6356942 | Bengtsson et al. | Mar 2002 | B1 |
| 6363077 | Wong et al. | Mar 2002 | B1 |
| 6366557 | Hunter | Apr 2002 | B1 |
| 6369855 | Chauvel et al. | Apr 2002 | B1 |
| 6421352 | Manaka et al. | Jul 2002 | B1 |
| 6424658 | Mathur | Jul 2002 | B1 |
| 6424659 | Viswanadham et al. | Jul 2002 | B2 |
| 6427185 | Ryals et al. | Jul 2002 | B1 |
| 6430190 | Essbaum et al. | Aug 2002 | B1 |
| 6457175 | Lerche | Sep 2002 | B1 |
| 6460088 | Merchant | Oct 2002 | B1 |
| 6463063 | Bianchini, Jr. et al. | Oct 2002 | B1 |
| 6466608 | Hong et al. | Oct 2002 | B1 |
| 6470436 | Croft et al. | Oct 2002 | B1 |
| 6473433 | Bianchini, Jr. et al. | Oct 2002 | B1 |
| 6477174 | Dooley et al. | Nov 2002 | B1 |
| 6480477 | Treadaway et al. | Nov 2002 | B1 |
| 6490280 | Leung | Dec 2002 | B1 |
| 6493347 | Sindhu et al. | Dec 2002 | B2 |
| 6496502 | Fite et al. | Dec 2002 | B1 |
| 6505281 | Sherry | Jan 2003 | B1 |
| 6510138 | Pannell | Jan 2003 | B1 |
| 6522656 | Gridley | Feb 2003 | B1 |
| 6532229 | Johnson et al. | Mar 2003 | B1 |
| 6532234 | Yoshikawa et al. | Mar 2003 | B1 |
| 6535504 | Johnson et al. | Mar 2003 | B1 |
| 6549519 | Daines et al. | Apr 2003 | B1 |
| 6553370 | Andreev et al. | Apr 2003 | B1 |
| 6556208 | Congdon et al. | Apr 2003 | B1 |
| 6567404 | Wilford | May 2003 | B1 |
| 6577631 | Keenan et al. | Jun 2003 | B1 |
| 6587432 | Putzolu et al. | Jul 2003 | B1 |
| 6591302 | Boucher et al. | Jul 2003 | B2 |
| 6601186 | Fox et al. | Jul 2003 | B1 |
| 6606300 | Blanc et al. | Aug 2003 | B1 |
| 6628650 | Saite et al. | Sep 2003 | B1 |
| 6633952 | Winograd et al. | Oct 2003 | B2 |
| 6636483 | Pannell | Oct 2003 | B1 |
| 6643269 | Fan et al. | Nov 2003 | B1 |
| 6654342 | Dittia et al. | Nov 2003 | B1 |
| 6654346 | Mahalingaiah et al. | Nov 2003 | B1 |
| 6654370 | Quirke et al. | Nov 2003 | B1 |
| 6654373 | Lie et al. | Nov 2003 | B1 |
| 6658002 | Ross et al. | Dec 2003 | B1 |
| 6661791 | Brown | Dec 2003 | B1 |
| 6671275 | Wong et al. | Dec 2003 | B1 |
| 6678248 | Haddock et al. | Jan 2004 | B1 |
| 6681332 | Byrne et al. | Jan 2004 | B1 |
| 6683872 | Saito et al. | Jan 2004 | B1 |
| 6687217 | Chow et al. | Feb 2004 | B1 |
| 6687247 | Wilford et al. | Feb 2004 | B1 |
| 6691202 | Vasquez et al. | Feb 2004 | B2 |
| 6696917 | Heitner et al. | Feb 2004 | B1 |
| 6697359 | George | Feb 2004 | B1 |
| 6697368 | Chang et al. | Feb 2004 | B2 |
| 6700894 | Shung | Mar 2004 | B1 |
| 6708000 | Nishi et al. | Mar 2004 | B1 |
| 6721229 | Cole | Apr 2004 | B1 |
| 6721268 | Ohira et al. | Apr 2004 | B1 |
| 6721313 | Van Duyne | Apr 2004 | B1 |
| 6721338 | Sato | Apr 2004 | B1 |
| 6731875 | Kartalopoulos | May 2004 | B1 |
| 6735218 | Chang et al. | May 2004 | B2 |
| 6745277 | Lee et al. | Jun 2004 | B1 |
| 6751224 | Parruck et al. | Jun 2004 | B1 |
| 6754881 | Kuhlmann et al. | Jun 2004 | B2 |
| 6765866 | Wyatt | Jul 2004 | B1 |
| 6775706 | Fukumoto et al. | Aug 2004 | B1 |
| 6778546 | Epps et al. | Aug 2004 | B1 |
| 6781990 | Puri et al. | Aug 2004 | B1 |
| 6785793 | Aboulenein et al. | Aug 2004 | B2 |
| 6792484 | Hook | Sep 2004 | B1 |
| 6792502 | Pandya et al. | Sep 2004 | B1 |
| 6798740 | Senevirathne et al. | Sep 2004 | B1 |
| 6804220 | Odenwalder et al. | Oct 2004 | B2 |
| 6804731 | Chang et al. | Oct 2004 | B1 |
| 6807179 | Kanuri et al. | Oct 2004 | B1 |
| 6807363 | Abiko et al. | Oct 2004 | B1 |
| 6810046 | Abbas et al. | Oct 2004 | B2 |
| 6813243 | Epps et al. | Nov 2004 | B1 |
| 6813266 | Chiang et al. | Nov 2004 | B1 |
| 6816467 | Muller et al. | Nov 2004 | B1 |
| 6831923 | Laor et al. | Dec 2004 | B1 |
| 6831932 | Boyle et al. | Dec 2004 | B1 |
| 6836808 | Bunce et al. | Dec 2004 | B2 |
| 6839346 | Kametani | Jan 2005 | B1 |
| 6842422 | Bianchini | Jan 2005 | B1 |
| 6842791 | Navada et al. | Jan 2005 | B2 |
| 6854117 | Roberts | Feb 2005 | B1 |
| 6859438 | Haddock et al. | Feb 2005 | B2 |
| 6865153 | Hill et al. | Mar 2005 | B1 |
| 6901072 | Wong | May 2005 | B1 |
| 6912637 | Herbst | Jun 2005 | B1 |
| 6920154 | Achler | Jul 2005 | B1 |
| 6925516 | Struhsaker et al. | Aug 2005 | B2 |
| 6934305 | Duschatko et al. | Aug 2005 | B1 |
| 6937606 | Basso et al. | Aug 2005 | B2 |
| 6946948 | McCormack et al. | Sep 2005 | B2 |
| 6957258 | Maher, III et al. | Oct 2005 | B2 |
| 6959007 | Vogel et al. | Oct 2005 | B1 |
| 6973092 | Zhou et al. | Dec 2005 | B1 |
| 6978309 | Dorbolo | Dec 2005 | B1 |
| 6980552 | Belz et al. | Dec 2005 | B1 |
| 6982974 | Saleh et al. | Jan 2006 | B1 |
| 6990102 | Kaniz et al. | Jan 2006 | B1 |
| 6993032 | Dammann et al. | Jan 2006 | B1 |
| 7005812 | Mitchell | Feb 2006 | B2 |
| 7009968 | Ambe et al. | Mar 2006 | B2 |
| 7012919 | So et al. | Mar 2006 | B1 |
| 7020814 | Ryle et al. | Mar 2006 | B2 |
| 7080238 | Van Hoof et al. | Jul 2006 | B2 |
| 7082133 | Lor et al. | Jul 2006 | B1 |
| 7103041 | Speiser et al. | Sep 2006 | B1 |
| 7120744 | Klein | Oct 2006 | B2 |
| 7126956 | Scholten | Oct 2006 | B2 |
| 7176911 | Kidono et al. | Feb 2007 | B1 |
| 7185141 | James et al. | Feb 2007 | B1 |
| 7185266 | Blightman et al. | Feb 2007 | B2 |
| 7187687 | Davis et al. | Mar 2007 | B1 |
| 7190696 | Manur et al. | Mar 2007 | B1 |
| 7191277 | Broyles | Mar 2007 | B2 |
| 7193997 | Van Lunteren et al. | Mar 2007 | B2 |
| 7203194 | Chang et al. | Apr 2007 | B2 |
| 7206283 | Chang et al. | Apr 2007 | B2 |
| 7212536 | Mackiewich | May 2007 | B2 |
| 7218637 | Best et al. | May 2007 | B1 |
| 7219293 | Tsai et al. | May 2007 | B2 |
| 7228509 | Dada et al. | Jun 2007 | B1 |
| 7236490 | Chang et al. | Jun 2007 | B2 |
| 7237058 | Srinivasan | Jun 2007 | B2 |
| 7249306 | Chen | Jul 2007 | B2 |
| 7266117 | Davis | Sep 2007 | B1 |
| 7277425 | Sikdar | Oct 2007 | B1 |
| 7283547 | Hook et al. | Oct 2007 | B1 |
| 7286534 | Kloth | Oct 2007 | B2 |
| 7324509 | Ni | Jan 2008 | B2 |
| 7355970 | Lor | Apr 2008 | B2 |
| 7356030 | Chang et al. | Apr 2008 | B2 |
| 7366100 | Anderson et al. | Apr 2008 | B2 |
| 7391769 | Rajkumar et al. | Jun 2008 | B2 |
| 7428693 | Obuchi et al. | Sep 2008 | B2 |
| 7443856 | Lodha et al. | Oct 2008 | B2 |
| 7468975 | Davis | Dec 2008 | B1 |
| 7512127 | Chang et al. | Mar 2009 | B2 |
| 7518998 | Sakamoto et al. | Apr 2009 | B2 |
| 7561590 | Walsh | Jul 2009 | B1 |
| 7596139 | Patel et al. | Sep 2009 | B2 |
| 7613991 | Bain | Nov 2009 | B1 |
| 7636369 | Wong | Dec 2009 | B2 |
| 7649885 | Davis et al. | Jan 2010 | B1 |
| 7657703 | Singh | Feb 2010 | B1 |
| 7738450 | Davis | Jun 2010 | B1 |
| 7817659 | Wong et al. | Oct 2010 | B2 |
| 7830884 | Davis | Nov 2010 | B2 |
| 20010001879 | Kubik et al. | May 2001 | A1 |
| 20010007560 | Masuda et al. | Jul 2001 | A1 |
| 20010026551 | Horlin | Oct 2001 | A1 |
| 20010048785 | Steinberg | Dec 2001 | A1 |
| 20010053150 | Clear et al. | Dec 2001 | A1 |
| 20020001307 | Nguyen et al. | Jan 2002 | A1 |
| 20020012585 | Kalkunte et al. | Jan 2002 | A1 |
| 20020054594 | Hoof et al. | May 2002 | A1 |
| 20020054595 | Ambe et al. | May 2002 | A1 |
| 20020069294 | Herkersdorf et al. | Jun 2002 | A1 |
| 20020073073 | Cheng | Jun 2002 | A1 |
| 20020085499 | Toyoyama et al. | Jul 2002 | A1 |
| 20020087788 | Morris | Jul 2002 | A1 |
| 20020089937 | Venkatachary et al. | Jul 2002 | A1 |
| 20020089977 | Chang et al. | Jul 2002 | A1 |
| 20020091844 | Craft et al. | Jul 2002 | A1 |
| 20020091884 | Chang et al. | Jul 2002 | A1 |
| 20020105966 | Patel et al. | Aug 2002 | A1 |
| 20020126672 | Chow et al. | Sep 2002 | A1 |
| 20020131437 | Tagore-Brage | Sep 2002 | A1 |
| 20020141403 | Akahane et al. | Oct 2002 | A1 |
| 20020146013 | Karlsson et al. | Oct 2002 | A1 |
| 20020161967 | Kirihata et al. | Oct 2002 | A1 |
| 20030009466 | Ta et al. | Jan 2003 | A1 |
| 20030033435 | Hanner | Feb 2003 | A1 |
| 20030043800 | Sonksen et al. | Mar 2003 | A1 |
| 20030043848 | Sonksen | Mar 2003 | A1 |
| 20030048785 | Calvignac et al. | Mar 2003 | A1 |
| 20030074657 | Bramley, Jr. | Apr 2003 | A1 |
| 20030081608 | Barri et al. | May 2003 | A1 |
| 20030095548 | Yamano | May 2003 | A1 |
| 20030103499 | Davis et al. | Jun 2003 | A1 |
| 20030103500 | Menon et al. | Jun 2003 | A1 |
| 20030108052 | Inoue et al. | Jun 2003 | A1 |
| 20030110180 | Calvignac et al. | Jun 2003 | A1 |
| 20030115403 | Bouchard et al. | Jun 2003 | A1 |
| 20030120861 | Calle et al. | Jun 2003 | A1 |
| 20030128668 | Yavatkar et al. | Jul 2003 | A1 |
| 20030137978 | Kanetake | Jul 2003 | A1 |
| 20030152084 | Lee et al. | Aug 2003 | A1 |
| 20030152096 | Chapman | Aug 2003 | A1 |
| 20030156586 | Lee et al. | Aug 2003 | A1 |
| 20030159086 | Arndt | Aug 2003 | A1 |
| 20030165160 | Minami et al. | Sep 2003 | A1 |
| 20030169470 | Alagar et al. | Sep 2003 | A1 |
| 20030174719 | Sampath et al. | Sep 2003 | A1 |
| 20030177221 | Ould-Brahim et al. | Sep 2003 | A1 |
| 20030198182 | Pegrum et al. | Oct 2003 | A1 |
| 20030215029 | Limberg | Nov 2003 | A1 |
| 20030223424 | Anderson et al. | Dec 2003 | A1 |
| 20030227943 | Hallman et al. | Dec 2003 | A1 |
| 20040022263 | Zhao et al. | Feb 2004 | A1 |
| 20040028060 | Kang | Feb 2004 | A1 |
| 20040054867 | Stravers et al. | Mar 2004 | A1 |
| 20040062246 | Boucher et al. | Apr 2004 | A1 |
| 20040088469 | Levy | May 2004 | A1 |
| 20040128434 | Khanna et al. | Jul 2004 | A1 |
| 20040141504 | Blanc | Jul 2004 | A1 |
| 20040179548 | Chang et al. | Sep 2004 | A1 |
| 20040190547 | Gordy et al. | Sep 2004 | A1 |
| 20040208177 | Ogawa | Oct 2004 | A1 |
| 20040223502 | Wybenga et al. | Nov 2004 | A1 |
| 20040264380 | Kalkunte et al. | Dec 2004 | A1 |
| 20050010630 | Doering et al. | Jan 2005 | A1 |
| 20050041684 | Reynolds et al. | Feb 2005 | A1 |
| 20050089049 | Chang et al. | Apr 2005 | A1 |
| 20050097432 | Obuchi et al. | May 2005 | A1 |
| 20050132132 | Rosenbluth et al. | Jun 2005 | A1 |
| 20050138276 | Navada et al. | Jun 2005 | A1 |
| 20050144369 | Jaspers | Jun 2005 | A1 |
| 20050152324 | Benveniste | Jul 2005 | A1 |
| 20050175018 | Wong | Aug 2005 | A1 |
| 20050185652 | Iwamoto | Aug 2005 | A1 |
| 20050193316 | Chen | Sep 2005 | A1 |
| 20050201387 | Willis | Sep 2005 | A1 |
| 20050226236 | Klink | Oct 2005 | A1 |
| 20050246508 | Shaw | Nov 2005 | A1 |
| 20050249124 | Elie-Dit-Cosaque et al. | Nov 2005 | A1 |
| 20060031610 | Liav et al. | Feb 2006 | A1 |
| 20060034452 | Tonomura et al. | Feb 2006 | A1 |
| 20060077891 | Smith et al. | Apr 2006 | A1 |
| 20060092829 | Brolin et al. | May 2006 | A1 |
| 20060092929 | Chun | May 2006 | A1 |
| 20060114876 | Kalkunte | Jun 2006 | A1 |
| 20060146374 | Ng et al. | Jul 2006 | A1 |
| 20060165089 | Klink | Jul 2006 | A1 |
| 20060209685 | Rahman et al. | Sep 2006 | A1 |
| 20060221841 | Lee et al. | Oct 2006 | A1 |
| 20060268680 | Roberts et al. | Nov 2006 | A1 |
| 20070038798 | Bouchard et al. | Feb 2007 | A1 |
| 20070088974 | Chandwani et al. | Apr 2007 | A1 |
| 20070179909 | Channasagara | Aug 2007 | A1 |
| 20070208876 | Davis | Sep 2007 | A1 |
| 20070253420 | Chang et al. | Nov 2007 | A1 |
| 20070258475 | Chinn et al. | Nov 2007 | A1 |
| 20070288690 | Wang et al. | Dec 2007 | A1 |
| 20080002707 | Davis | Jan 2008 | A1 |
| 20080031263 | Ervin et al. | Feb 2008 | A1 |
| 20080037544 | Yano et al. | Feb 2008 | A1 |
| 20080049742 | Bansal et al. | Feb 2008 | A1 |
| 20080069125 | Reed et al. | Mar 2008 | A1 |
| 20080092020 | Hasenplaugh et al. | Apr 2008 | A1 |
| 20080095169 | Chandra et al. | Apr 2008 | A1 |
| 20080181103 | Davies | Jul 2008 | A1 |
| 20080205407 | Chang et al. | Aug 2008 | A1 |
| 20080307288 | Ziesler et al. | Dec 2008 | A1 |
| 20090175178 | Yoon et al. | Jul 2009 | A1 |
| 20090279423 | Suresh et al. | Nov 2009 | A1 |
| 20090279440 | Wong et al. | Nov 2009 | A1 |
| 20090279441 | Wong et al. | Nov 2009 | A1 |
| 20090279541 | Wong et al. | Nov 2009 | A1 |
| 20090279542 | Wong et al. | Nov 2009 | A1 |
| 20090279546 | Davis | Nov 2009 | A1 |
| 20090279548 | Davis et al. | Nov 2009 | A1 |
| 20090279549 | Ramanathan et al. | Nov 2009 | A1 |
| 20090279558 | Davis et al. | Nov 2009 | A1 |
| 20090279559 | Wong et al. | Nov 2009 | A1 |
| 20090282148 | Wong et al. | Nov 2009 | A1 |
| 20090282322 | Wong et al. | Nov 2009 | A1 |
| 20090287952 | Patel et al. | Nov 2009 | A1 |
| 20090290499 | Patel et al. | Nov 2009 | A1 |
| 20100046521 | Wong | Feb 2010 | A1 |
| 20100061393 | Wong | Mar 2010 | A1 |
| 20100100671 | Singh | Apr 2010 | A1 |
| 20100135313 | Davis | Jun 2010 | A1 |
| 20100161894 | Singh | Jun 2010 | A1 |
| 20100246588 | Davis | Sep 2010 | A1 |
| Number | Date | Country |
|---|---|---|
| 1380127 | Jan 2004 | EP |
| 2003289359 | Oct 2003 | JP |
| 2004-537871 | Dec 2004 | JP |
| WO 0184728 | Nov 2001 | WO |
| WO 02041544 | May 2007 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20090279561 A1 | Nov 2009 | US |
| Number | Date | Country | |
|---|---|---|---|
| 60249871 | Nov 2000 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 11804977 | May 2007 | US |
| Child | 12372390 | US | |
| Parent | 09855038 | May 2001 | US |
| Child | 11804977 | US |
