Patent Application
20070104210
1. Field of the Invention
The invention relates to a method and apparatus for high performance switching in local area communications networks. In particular, the invention relates to a process of scheduling of packets and allowing for a shaping of the flows of the packets so that all of the output bandwidth can be fairly and fully utilized according to set requirements.
2. Description of the Related Art
A switching system may include one or more network devices, such as an Ethernet switching chip, each of which includes several modules that are used to process information that is transmitted through the device. Specifically, the device includes at least one ingress module, a Memory Management Unit (MMU) and at least one egress module. The ingress module includes switching functionality for determining to which destination port a packet should be directed. The MMU is used for storing packet information and performing resource checks. The egress module is used for performing packet modification and for transmitting the packet to at least one appropriate destination port. One of the ports on the device may be a CPU port that enables the device to send and receive information to and from external switching/routing control entities or CPUs. Some devices also include a CPU processing module through which the device interfaces with external CPU ports.
In the processing of datagrams, such as packets, certain packets may receive preferential treatment when compared to other packets. As such, certain packets may be assigned a higher Quality of Service (QoS), such that those packets are given preferred treatment. This preferred treatment may be given, for example, to packets where time sensitive receipt of those packets is important. In many prior art systems, many QoS states are assigned, so that varying degrees of handling and prioritization can be provided. However, even if a small amount of bandwidth is allocated to a particular QoS state and is not used, that bandwidth is “wasted,” in that it could be utilized by other resources. Thus, there is a need in the prior art for systems that allow for dynamic management of buffers and scheduling of data transmission to allow for efficient utilization of all resources of a network device.
For the present invention to be easily understood and readily practiced, various embodiments will now be described, for purposes of illustration and not limitation, in conjunction with the following figures:
Shaping of the egress packet bandwidth, according to embodiments of the present invention, utilizes a combination of minimum bandwidth guarantee/maximum bandwidth allowable shaping and weighted deficit round robin scheduling scheme to select a packet from different CoS queues, for a given output port. For each queue, during minimum bandwidth guarantee shaping, the scheduler will be selected based on round robin scheduling or strict priority scheduling, based on a separate minimum bandwidth strict priority register.
After satisfying minimum bandwidth guarantees, each queue is entered into a maximum bandwidth allowable region, where the scheduler will use either weighted deficit round robin (WDRR) or strict priority (SP) to pick a packet from different CoS queues. A weight register is used to store the weight value that is used in that process. According to embodiments of the instant invention, when the weight is zero, this queue is treated as strict priority. When weight is non-zero, it is used to determine a WDRR quantum for each queue. The packet length information is feed back into the minimum/maximum bucket to account for transmitted bandwidth that is actually used. The same information is also feed back into a WDRR credit count to adjust actual bandwidth usage. The WDRR credit count is based on a fixed unit of some number of bytes, i.e., cells, to count transmit bandwidth. The packet length is first sent to a residue counter to calculate the positive or negative residue of actual data and a cell size. When the scheduler schedules out a cell, which may, for example, be 128 bytes, it will increment the WDRR credit counter and adjust the value of a cell granularity accumulation of positive or negative residue in residue counter. This is method accounts for accurate transmit bandwidth.
Device 100 can also include a number of ports to send and receive data, such as Port 0 to PortX, 108A-108X, and Port X+1 to Port Y, 109A-109X. The ports can be separated and are serviced by different ingress and egress port modules to support the dual-pipeline structure. One or more internal fabric high speed ports, for example a highspeed port, or more external Ethernet ports may be configured from the above-discussed ports. The network device can also include a CPU port 110 and a CPU processing module 11 to communicate with an external CPU. High speed ports are used to interconnect various network devices in a system and thus form an internal switching fabric for transporting packets between external source ports and one or more external destination ports. As such, high speed ports are not externally visible outside of a system that includes multiple interconnected network devices. CPU port 110 can be used to send and receive packets to and from external switching/routing control entities or CPUs. Device 100 interfaces with external/off-chip CPUs through a CPU processing module 111, which interfaces with a PCI bus that connects device 100 to an external CPU.
Network traffic also enters and exits device 100 through external ports 108A-108X and 109A-109X. Specifically, traffic in device 100 is routed from an external source port to one or more unique destination ports. In one embodiment of the invention, device 100 supports physical Ethernet ports and logical (trunk) ports.
Once a packet enters device 100 on a source port 109A-109X or 108A-108X, the packet is transmitted to one of the ingress modules 102A or 102B for processing. Packets may enter device 100 from a XBOD or a GBOD. The XBOD is a block that has one 10GE/12G MAC and supports packets from high speed ports and the GBOD is a block that has 12 10/100/1G MAC and supports packets from other ports.
The architecture of the network device provides for the ability to process data received quickly and also allows for a flexibility of processing. A part of this flexibility comes from the pipeline structure that is used to process packets once they are received. Data from the packet and attributes of that packet move through the modules of the network device, discussed above, in a pipeline structure. Each stage in the pipeline structure requires a set number of clock cycles and the packets are processed in order. Therefore, the packet is parsed, table lookups are performed, a decision routing process is performed and the packet is modified, before being sent out on an egress port. Each stage in the pipeline performs its function so that the overall function of the network device is achieved.
Arbiter 202 collects packet data and control information from MMU 104 and read/write requests to registers and memories from the CPU and synchronizes the packet data and control information from MMU 104 and writes the requests from the CPU in a holding register. Based on the request type from the CPU, arbiter 202 generates pipeline register and memory access instructions and hardware table initialization instructions. After arbiter 202 collects packet data, CPU requests and hardware table initialization messages, it generates an appropriate instruction. According to an embodiment, arbiter 202 generates a Start Cell Packet instruction, an End Cell of Packet instruction, a Middle Cell of Packet instruction, a Start-End Cell of Packet instruction, a Register Read Operation instruction, a Register Write Operation instruction, a Memory Read Operation instruction, a Memory Write Operation instruction, a Memory Reset Write Operation instruction, a Memory Reset Write All Operation instruction and a No Operation instruction. Egress pipeline resources associated Start Cell Packet instructions and Start-End Cell of Packet instructions are given the highest priority by arbiter 204. End Cell of Packet instructions, Middle Cell of Packet instructions, Register Read Operation instructions, Register Write Operation instructions, Memory Read Operation instructions and Memory Write Operation instruction receive the second highest priority from arbiter 204. Memory Reset Write Operation instructions and Memory Reset Write All Operation instructions receive the third highest priority from arbiter 204. No Operation instructions receive the lowest priority from arbiter 204.
After receiving an instruction from arbiter 204, the parser 206 parses packet data associated with the Start Cell of Packet instruction and the Start-End Cell of Packet instruction using the control information and a configuration register transmitted from arbiter 206. According to an embodiment, the packet data is parsed to obtained L4 and L3 fields which appear in the first 148 bytes of the packet. Table lookup stage 208 then receives all packet fields and register values from parser 206.
The Egress Scheduler (ES) is a pipeline structure processor. It is implemented as two identical blocks, one for each pipeline, discussed above. Another function inside the ES block is port tdm scheduling for two MMU Egress Pipelines. One pipeline of that architecture of the MMU 301 is illustrated in
The Egress Scheduler performs the following functions: Port TDM scheduling, QM and SC Queues, Strict Priority Scheduling (SP), Round Robin Priority Scheduling (RR), Weight Round Robin Priority Scheduling (WRR), Weight Deficit Round Robin Scheduling (WDRR), Min/Max Shaping Support, Egress Port Metering Mask Support, COS Based Flow Control Support, CPU COS Mask Support, and Register Read/Write Support. Each of the function is described below.
The port TDM scheduling can be implemented in different TDM schedules, depending on the port configurations of the network device. The basic rule for this TDM scheduling is for each 12G HG port, every 8 cycles need to get service. For each 10G Ethernet port, every 10 cycles need to get service. It is noted that each pipeline can be in its own TDM mode, which is programmable in ESTDMCONFIG registers.
The present invention, according to certain embodiments, supports two new queues for each egress port: QM (Queue Management) and SC (System Control). These two new queues are set to highest priority to be scheduled to each egress port, according to at least one embodiment. Within these two queues, QM is higher priority than SC. QM queue is mainly to service end to end flow control packets in order to pass through this type of packet flow through each device within the same fabric. SC queue is mainly to service one switch CPU to another switch CPU's system control packet. This type of traffic need to service as soon as possible in order let peer end's CPU can react on urgent congestion condition.
When a port's arbitration mode is selected as Strict Priority (SP) Scheduling Mode, the queue selected from this port will be following by strict priority fashion. This means, when a higher priority queue is not empty, ES block will keep selecting this queue till it finally empty. If ES block is currently selecting a low priority queue and suddenly a higher queue becomes not empty, ES will swim back to select higher priority queue.
When a port's arbitration mode is selected as Round Robin Scheduling Mode, the queue selected from this port will be followed in a Round Robin fashion. Each round starts from CoS7 and finishes at CoS0. Those specific CoS queues are illustrated in
When a port's arbitration mode is selected as Weighted Round Robin Scheduling Mode, the queue selected from this port is basically following by Round Robin fashion but with a weighted packet credit for each queue. For every packet scheduled, a credit will be incremented from a weight credit counter. The ES block will keep selecting CoS queues in a round robin fashion until any CoS queue's credit count matched its weighted credit and it is dropped from the round.
Each sub round starts from CoS7 and finishes at CoS0. When the ES block is selecting CoS0, next time it will start from CoS7 again. When all queue's weight credit counter matches its weighted credit limit. A round of WRR is finished and all queue start all over again. Within a round, if a queue becomes empty, its credit counter will be kept until the current round completed. Therefore, if a queue is constantly empty and each empty state only lasts for a short period, its credit counter will still be valid.
During each round, if there is any QM or SC queue becomes non-empty, the ES block will suspend round robin scheduling and remember at which point it left off. Then, the ES block will service these two queues first and when both two queues are back to empty, ES block will resume the round robin scheduling and start from the point that it left from. If a queue's weight register is programmed as 0, this queue will be treated as priority scheduling and will not participate in WRR selection. If any one of zero-weighted queue is not empty, ES block will not start or resume normal WRR round.
When a port's arbitration mode is selected to be Weighted Deficit Round Robin (WDRR), the backlogged and eligible CoS queues associated with the port are serviced according to a byte-accurate relative bandwidth sharing scheme. This behavior is accomplished by having the scheduler allow the CoS queues to send a weighted amount of traffic in rounds. This is illustrated in
A WDRR round is over when all CoS queues are in at least one of the following conditions:
The WeightedUseLimit is calculated by multiplying the CoS queue's associated weight (1-127) by the port's MTU_QUANTA_SELECT. The MTU_QUANTA_SELECT can be selected to match the expected max frame length. Four possible options, for certain embodiments, for MTU_QUANTA_SELECT are 2,048 bytes, 4,096 bytes, 8,192 bytes and 16,384 bytes.
During a round, the scheduler does not continue to service a single CoS queue at a time until its CurrentUseCount is greater than its WeightedUseLimit. Rather, the scheduler allows interleaving of packet transmissions between CoS queue transmissions to minimize latency. Consequently, a single WDRR round is actually composed of one or more sub rounds where each of the backlogged and eligible CoS queues are serviced in round robin order. The ES block will continue selecting a CoS queue as a part of the round robin order until the CoS queue's CurrentUseCount is greater than its WeightedUseLimit and it drops out of the round. Each sub round starts from CoS7 and finishes at CoS0. When the ES block is selecting CoS0, next time it will start from CoS7 again.
Although the WDRR accounting is in cells, the bandwidth sharing is actually byte-accurate. This is accomplished by also keeping track of the actual number of bytes sent using a ResidueCount 315, in
The way the ResidueCount operates is as follows. For every cell's worth of packet being scheduled, i.e., for every 128 bytes for example, the CurrentUseCount may be incremented by 1. If the EP modified length is not equal to a single cell size, then the ResidueCount may be adjusted. The difference between the cell size and the actual EP modified cell length may then be added to the ResidueCount. Depending on the state of the ResidueCount, the CurrentUseCount and the ResidueCount may both be further adjusted. When the ResidueCount is negative, it implies that more was sent than currently accounted for. When the ResidueCount is positive, this implies that less was sent than currently accounted for, as documented in the p-code.
During each round, if either the QM or SC queue becomes non-empty, ES block will suspend round robin scheduling while taking note of where it has left off. Then, the ES block will then service these two queues first and when both of these queues are empty, ES block will resume the round robin scheduling and start from where it had left off. As discussed above, the QM queue has higher priority over the SC queue.
The WDRR scheduler interacts with the minimum bandwidth metering and maximum bandwidth shaping that is available per CoS queue. Min/Max BW mechanisms associated with each CoS queue results in the identification of scheduling groups. These groups are illustrated in
Within the MinBW Group 410, there exist two subgroups, the Strict Priority group 412 and the Round Robin group 414. Within the ExcessBW Group 420, there exist two subgroups, the Strict Priority group 422 and the Non-Strict Priority Group. 426 The servicing order of these groups is also shown in
Next, the set of queues that have not yet had their minimum BW requirement satisfied are serviced (MinBW group, 410). First, the Strict Priority subgroup 412 is serviced in SP order according to their CoS queue number (higher the CoS queue number, the higher the priority). Secondly, the RoundRobin subgroup 414 is serviced when the Strict Priority subgroup is empty. These CoS queues are serviced in packet round robin order. There exists a per-CoS queue register that is used to indicate how to treat this queue in terms of the two subgroups (Q_SP_Min_Group) within the MinBW group 410. If this register is ‘1’, then the CoS queue is serviced in the StrictPriority subgroup in this MinBW group. Otherwise, the CoS queue is serviced in the Round Robin subgroup. Next, the ExcessBW group 420 is serviced. First, the Strict Priority subgroup 422 is serviced. Next the non-Strict Priority subgroup 426 is serviced according to the scheduler mode. In this section, this mode is assumed to be WDRR. Membership between these two subgroups of the ExcessBW group 420 is determined by the value of the WDRR weight. If a queue's weight register is programmed as 0, this queue will be treated as a member of the Strict Priority subgroup and will not participate in WDRR selection.
The overall functionality of the present scheduling process are further illustrated in
A user may configure the MTU_QUANTA_SELECT to be smaller than the actual MTU associated with this port. The requirement for this misconfiguration case is that the scheduler will continue to service packets. In this implementation, such a misconfiguration may result in momentary “bubbles” in operation and consequently the scheduler becomes non-work conserving. However, the scheduler is able to operate at a rate faster than line rate. Consequently, the non-work conserving behavior is not visible to an outside observer. At present, the worst case scenario identified is if a single CoS queue has associated with it a weight of 1 and an MTU_QUANTA_SELECT of 2 kbytes. If the actual MTU of the system is 16 kbytes, then in the worst case, the scheduler must refresh the CoS queue's CurrrentUseCount over 8 rounds before it is able to be scheduled again. However, given the MMU is able to schedule faster than line rate and given the egress port is able to buffer up to 14 cells, this misconfiguration issue should not be visible to an outside observer.
The user may configure the WDRR configuration registers while servicing live traffic. The requirement is that operation must continue without interruption in service. In this implementation, whenever a WDRR configuration register is written to (regardless of whether the value is actually changed), the CurrentUseCount is reset to 0 and all COS queues become eligible for the subsequent WDRR round. The registers that trigger this behavior include the Q_Weight_Select and the MTU_QUANTA_SELECT.
A Min/Max Shaping Mask is based its queue's minimum Bandwidth requirement and maximum allowable bandwidth. When any one of queue's minimum bandwidth requirement is not satisfied, this queue has a higher priority than other queues which satisfied their minimum bandwidth requirement. If more than one queue is not satisfied its minimum requirement, these queues will based on either Strict Priority or Round Robin method to pick one queue. When any one is satisfied, this queue will be dropped out from this minimum list.
When all of queues have satisfied their minimum bandwidth requirements, all the queues will be entered into a state to gain maximum allowable bandwidth. Still, between each eligible queue, the ES block will, based on either Strict Priority, Round Robin, Weighted Round Robin or Weighted Deficit Round Robin Scheduling method, pick one queue for distributing the excess bandwidth. After each queue satisfies its maximum bandwidth requirement, it will be dropped from max list. During the time of transmission, some queues may switch between min group and max group because of their minimum bandwidth condition has been met. When there is a switch between min/max group, the ES will only trigger a credit reset on min group rather than reset on all queue, to prevent max queue's bandwidth distribution being disturbed.
Before the ES block selects a queue, each queue's empty status will pass through a min/max shaping mask to find out if the current port is in a minimum shaping state or in a maximum shaping state. This min/max shaping mask is only applied to Packet boundary; if this mask is set during a packet transmission, it will not take effect until the completion of the current packet transmission. It should be noted that QM and SC queues do not participate in min/max shaping. They are always governed by strict priority to get scheduled out. In general, when queues do not participate in min/max shaping, they are always scheduled out using strict priority.
The buckets discussed below are not in Egress Scheduler block. However, their function descriptions are listed in the following sections to give a broader view, since their behaviors are deeply correlated with Egress scheduler functionality.
Each queue of each port has its own Min Bucket (or Minimum Shaping Bucket) to count for satisfaction of a minimum bandwidth requirement. When a packet is sent from a queue, the length of this packet will be added into the Min Bucket. The Min Bucket stops counting new transmitted packet length when the bucket is over its High Threshold. This means that the bucket is saturated at the High Threshold. When a refresh cycle time has expired, refresh tokens are leaked away from each Min Bucket. Each refresh cycle is 7.8125 μs, according to one embodiment and, for every one token leaked away, it means this queue's Min Bucket is refreshed at rate of 64,000 bit per second, in that embodiment.
When tokens inside Min Bucket are over Low Threshold, the queue's Out_Profile_Flag will be set. And when Bucket is under its Low Threshold, its Out_Profile_Flag will be reset. This flag is used by Egress Scheduler to determine in which region the current queue is at, and apply proper scheduler arbitrations on a port. In Min not Satisfied region, Egress Scheduler is applying RR+SP arbitration scheme In Min Satisfied and Max not Satisfied region, Egress Scheduler is applying any kind of arbitration scheme. (WDRR, WRR, RR+SP).
The Min Bucket accounts for all the tokens transmitted within the minimum and maximum region to make sure each queue is not under its minimum bandwidth requirement. When a queue constantly transmits greater than its minimum bandwidth, this Min Bucket will get saturated at High Threshold and will not drop down below Low Threshold. That means this queue is only operate at Max Region after it satisfies its minimum bandwidth requirement. In order to resolve this accounting issue, a new restriction is added to Min Bucket.
The Min Bucket can only add tokens when a packet is transmitted within the MinBW Region. If a packet is transmitted half way when Min Bucket is over its Low Threshold, the Min Bucket will continue add tokens until this packet transmission is completed. When Min Bucket is over its Low Threshold, its Out_Profile flag will be set. The Egress Scheduler will not treat this queue as having a minimum Bandwidth requirement being satisfied. After a couple of refreshes, the Min Bucket will drop below its Low Threshold, and Out_Profile flag will reset to in profile status. The Egress Scheduler will treat this queue as the minimum Bandwidth requirement not being satisfied again.
Each queue of each port also has its own Max Bucket (or Maximum Shaping Bucket) to count for satisfaction of the maximum bandwidth requirement. When a packet sent from a queue when the queue is still under maximum bandwidth not satisfied, the length of this packet will be added into Max Bucket. When a refresh cycle time is expired, refresh tokens are leaked away from each Min Bucket. Each refresh cycle is a fixed amount of time, which may be, for example, 7.8125 micro seconds, according to a particular embodiment. Therefore, for every one token leaked away, it means this queue's Max Bucket is refreshed at a fixed number of bits/sec, which may be 64,000 bits per second for an embodiment of the invention.
When tokens inside Max Bucket are over the Threshold, the queue's Out_Profile_Flag will be set. And when the Max Bucket is under its Low Threshold, its Out_Profile_Flag will be reset. This flag is used by the Egress Scheduler to determine if a given queue is using all of its allowable bandwidth. When set to Out-Profile status, this queue will not be able to participate in scheduler arbitration until its flag is reset to In-Profile Status.
The Egress Port Metering Mask is based on its port's predefined Bandwidth requirement on maximum allowable bandwidth. When a port's maximum bandwidth requirement is not satisfied, this port is allowed to do transmission. If a port's transmission takes away too much bandwidth, i.e. more than its allowable, this egress metering mask will get set. When this mask is set, no more packets can be allowed to be sent out from MMU. The MMU waits until this egress port metering mask gets reset and resumes transmission. The assertion of the Egress Port Metering flag will retain each queue's credit count and hold them until flag is de-asserted and resumes scheduling. This mask is only applied to the packet boundary; i.e. if this mask is set during a packet transmission, it will not take effect until the completion of the current packet transmission.
Each port has one Port Metering Bucket (or Port Shaping Bucket) to provide satisfaction of the maximum bandwidth requirement for a port. When a packet is sent from any queue of a port, when this port has not satisfied its port maximum bandwidth, the length of this packet will be added into Port Metering Bucket. When a refresh cycle time has expired, refresh tokens are leaked away from each Port Metering Bucket. Each refresh cycle is 7.8125 us, according to one embodiment. For every one token leaked away, it means this port Metering Bucket is refreshed at rate of 64,000 bit per second, in that embodiment.
When tokens of Port Metering Bucket are over the Port Metering Threshold, the port's Out_Profile_Flag will be set. And when tokens inside the Port Metering Bucket are under the Port Metering Threshold, the port's Out_Profile_Flag will be reset. This flag is used by the Egress Scheduler to determine whether a given port is able to transmit or not. However, this flag is used only at the start of a new packet transmission. During the actual packet transmission, the Egress scheduler will not stop transmission, even this Flag is switched from In-Profile to Out-Profile status. Thereafter, the port will not be able to resume transmission until its flag is reset to In-Profile status.
One of average skill in the art will also recognize that the functional building blocks, and other illustrative blocks, modules and components herein, can be implemented as illustrated or by discrete components, application specific integrated circuits, processors executing appropriate software and the like or any combination thereof.
Moreover, although described in detail for purposes of clarity and understanding by way of the aforementioned embodiments, the present invention is not limited to such embodiments. It will be obvious to one of average skill in the art that various changes and modifications may be practiced within the spirit and scope of the invention, as limited only by the scope of the appended claims.
This application claims priority of U.S. Provisional Patent Application Ser. No. 60/735,251, filed on Nov. 10, 2005 and U.S. Provisional Patent Application No. 60/772,886, filed on Feb. 14, 2006. The subject matter of the earlier filed applications is hereby incorporated by reference.
| Number | Date | Country | |
|---|---|---|---|
| 60735251 | Nov 2005 | US | |
| 60772886 | Feb 2006 | US |
