Patent Application
20050068798
Committed Access Rate (CAR) or Committed Information Rate (CIR) is the data rate that an access provider guarantees will be available on a connection. CAR is a way to provide Quality of Service (QoS) in an IP/Ethernet network. By providing CAR to a targeted QoS group in the IP/Ethernet network, a preserved and guaranteed bandwidth specified in a predetermined service level agreement (SLA) can be provided to that targeted QoS group rather than merely providing a best effort service. The ability to provide QoS in the IP/Ethernet network is important for supporting real time applications and for deploying a pure IP network in areas where most of the existing infrastructure may be based on ATM or SONET.
Currently, CAR is generally only available in large and expensive networking systems. Thus, CAR is not cost effective and thus is generally not available in an enterprise network. However, to support the QoS in the IP/Ethernet network, it may be preferable to deploy CAR in the IP/Ethernet network from end to end and not merely within the core of the network.
In addition, the increased demand to support real-time or interactive audio and video applications in an enterprise IP/Ethernet network is a key driving force for providing QoS in an enterprise network. Currently, supervision is used to prevent network congestion. However, supervision does not solve potential congestion problems in an enterprise network when dealing with multicast (Mcast) traffic. For example, in a N−1 (i.e., N ports sending traffic to 1 port) situation, Mcasts can cause a large burst in packet flow in a very short period of the time. The large burst makes it difficult to mix audio, video and data traffic together without limiting the quality of the audio/video traffic or separating the real-time traffic from data traffic.
One way of addressing the issue of large bursts is through the use of packet memory reservations. In packet memory resource reservation, a set amount of memory is allocated in an attempt to guarantee bandwidth for a particular type of network traffic.
For example, 50% of the available bandwidth in a network switch may be preserved for a targeted QoS group and the remaining 50% may be reserved for other types of network traffic. With packet memory reservation, when one type of traffic reaches the capacity of its allocated memory, packets from that traffic are dropped. However, the other type of traffic may still have available memory but because that memory is preserved, the available memory cannot be utilized. Under utilization of packet memory space leads to the bandwidth for the system to be underutilized which also makes the system inefficient.
Furthermore, if the traffic reaching capacity is Mcast traffic, it would be undesirable to drop Mcast packets because such dropping may limit the quality of real-time audio/video traffic.
Systems and methods for committed access rate (CAR) system architecture in an IP/Ethernet network with optional dynamic packet memory reservation are disclosed. It should be appreciated that the present invention can be implemented in numerous ways, including as a process, an apparatus, a system, a device, a method, or a computer readable medium such as a computer readable storage medium or a computer network wherein program instructions are sent over optical or electronic communication lines. Several inventive embodiments of the present invention are described below to enable any person skilled in the art to make and use the invention. Descriptions of specific embodiments and applications are provided only as examples and various modifications will be readily apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the invention. Thus, the present invention is to be accorded the widest scope encompassing numerous alternatives, modifications and equivalents consistent with the principles and features disclosed herein. For purpose of clarity, details relating to technical material that is known in the technical fields related to the invention have not been described in detail so as not to unnecessarily obscure the present invention.
The method generally includes classifying each received packet into a quality of service (QoS) group using the packet header information, defining a traffic transmission rate profile using, for example, a token bucket model to measure and check the traffic rate profile of the incoming packet against a corresponding service level agreement (SLA), marking the packet as in profile or out of profile packet, and performing packet buffer memory reservation to guarantee storage for in profile CAR packets.
The packet classification may be performed via a content addressable memory (CAM), or via a multi-bank ternary CAM (T-CAM). The token bucket model can be realized in hardware and facilitates in controlling CAR packets as well as input rate limiting (IRL) packets and output rate limiting (ORL) packets. A CAR packet is in profile if it is within the corresponding SLA such that the in profile CAR packet receives congestion-free service. A CAR packet is out of profile if the SLA is exceeded and is provided with best effort service and/or dropped. IRL and ORL in profile packets receive best effort service whereas IRL and ORL out of profile packets are dropped.
Buffer memory reservation may be via static memory reservation in which memory space is statically partitioned between CAR packets and non-CAR packets. Alternatively, the buffer memory reservation may be via dynamic memory reservation in which packet buffer memory is dynamically allocated between CAR packets and non-CAR packets and a push-out mechanism (e.g., head-drop) is employed to push out non-CAR packets when the network traffic is congested. Examples of push out mechanisms include head drop which refers to dropping the oldest packets and tail drop which refers to dropping the newest packets. Separate multicast queues and thresholds can optionally be defined for multicast packets and a multicast counter can be provided to facilitate tracking of multicast packets.
A network device, e.g., a router or a switch, for providing committed access rate (CAR) in an IP/Ethernet network generally includes a control pipe configured to classify each received packet into a quality of service (QoS) group using packet header information. The control pipe is further configured to define a traffic transmission profile using a token bucket model to define the traffic behavior for a given traffic flow and measuring against a corresponding SLA, to mark the packet as in profile or out of profile, and to perform packet buffer memory reservation to guarantee storage space for in profile CAR packets. The network device also includes a transmit queue in communication with the control pipe and a packet buffer memory in communication with the transmit queue. The transmit queue includes transmit queue entries and transmit queue entry memory. The packet buffer memory is configured to receive and store received packets. The control pipe is configured to perform packet buffer memory reservation to guarantee transmit queue and packet buffer memory space for in profile CAR packets.
CAR classifies traffic into different QoS groups based on SLA and gives each QoS group a predetermined service in terms of bandwidth and resource allocation. In other words, it makes conformed CAR traffics immune from congestion due to other traffics in the network. The CAR network device 100 is able to mix various types of network traffic (e.g., CAR, IRL, ORL, etc.) with low cost and high quality.
As shown in
When a packet arrives at the network device 100, the control pipe 102 utilizes information in the packet header for processing. In particular, the control pipe 102 performs a packet classification function to classify incoming packet traffic into different QoS groups using the information available in the packet header. For example, the packet header may contain any combination of data such as L2 source address, L2 destination address, IP source address, IP destination address, VLAN Tag, TCP socket numbers, and/or various other packet header information. The level of packet classification capabilities depends on where the CAR network device 100 is deployed within the network and the type of the application.
Packet classification is configured in hardware and is determined in the control pipe 102 of the network device 100 via the CAM 110. The CAM 110 is optionally a multi-bank Ternary CAM (T-CAM) as the T-CAM permits partial-match retrieval and is useful for packet classification. However, any other suitable addressable memory may be used. The multi-bank TCAM may provide classification for L2/L3/MPLS packets separately. Packet matches with programmed fields with content in the TCAM are marked and assigned a unique pointer for further packet rule lookups and packet processing.
After packet classification and identification, the control pipe 102 performs traffic rate profile check. In one embodiment, a token bucket model is used to measure and check the traffic rate profile of the incoming packet against the SLA. The configurable parameters used in the token bucket model include token refill rate r, token size s and burst size b. Thus, the long-term average rate is r*s and the burst size b maps to the maximum storage requirement in the network device 100. The token bucket model assumes that the outgoing bandwidth is at least equal to or greater than the average rate r*s, which can be controlled by Weighted Fair Queuing (WFQ) in the output stage. WFQ is a technique for selecting packets from multiple queues. WFQ avoids the problem of starvation that can arise when strict priorities are used. Otherwise, the storage requirement would be unbounded. The same token bucket model may be used to define CAR, ORL (output or outbound rate limiting) and/or IRL (input or inbound rate limiting) traffic. A counter to track the usage per flow and a memory element to store the available space may be used to realize the token bucket model in hardware. A pointer assigned in the packet classification stage is used to reference the current usage in the memory.
It is noted that although the token model may be utilized, any other suitable mechanism to measure incoming traffic rate against configured traffic rate profile (resource over time) may be employed. In addition, any suitable modifications to the token model as described may be employed. For example, two cascading token buckets may be employed in which the first token bucket measures incoming CAR traffic rate against configured traffic rate profile and marks the packet as in profile or out of profile. The out of profile bucket may then be passed to a second, preferably larger token bucket that measures the out of profile packet against a more relaxed traffic rate profile configuration. The second token bucket determines whether the out of profile packet receives best of effort service or simply dropped.
Once the traffic rate profile of the incoming packet has been checked and measured against the SLA using, e.g., the token bucket model, the packet can be categorized as an in profile or an out of profile packet. CAR packets within the confirmed SLA, i.e., if token is available, are in profile packets and are treated as committed packets and enjoy congestion-free service. CAR packets exceeding the SLA are out of profile packets and may be dropped and/or treated as best effort packets. IRL and ORL in profile packets receive best effort service while IRL and ORL out of profile packets are dropped. Services for the two classes of packets for CAR, ORL and IRL traffic are summarized in TABLE 1 below.
The network device performs the resource reservation function by managing the packet buffer memory 106, and transmit queue entries (TxE) and transmit queue (TxQ) links of the transmit queue 104.
Each transmit queue entry 120 contains a transmit queue link 122 and a transmit update entry memory address 124. Within a given transmit queue 104, the transmit queue link 122 of each transmit queue entry 120 points to the next transmit queue entry, as indicated by arrows from transmit queue link 122A to transmit queue entry 120B, from transmit queue link 122B to transmit queue entry 120C, and from transmit queue link 122C to transmit queue entry 120D. A transmit queue entry 120 is consumed when the corresponding packet is either sent or dropped (e.g., pushed out such as by being head-dropped).
The transmit update entry memory address 124 of each transmit queue entry 120 points to a location in the transmit queue edit memory 126 that contains information for packet header updates as well as the address of the packet in packet memory. Each transmit update entry memory address 124 points to a transmit update entry 128 in the transmit queue edit memory 126. Any of the transmit update entries 128 may be pointed to by multiple transmit queue entries 120 such as may be the case with a multicast packet. For example, in
As is evident, implementation of CAR seeks to guarantee a minimum packet memory space for CAR packets. This guarantee of memory space for CAR packets may be achieved by utilizing static packet buffer memory reservation in which a separate packet buffer memory space is reserved for each CAR flow. Static reservation is a way to partition the packet buffer memory space between CAR and non-CAR traffic. Thus, non-CAR traffic will not be allowed to utilize the packet buffer memory space reserved for CAR even when there is available memory space in the space reserved for CAR traffic.
To utilize the available memory space for increased efficiency while guaranteeing memory space for CAR packets, the guarantee of memory space for CAR packets is preferably achieved using dynamic rather than static memory reservation of the packet buffer memory space between CAR and non-CAR traffic flows. The dynamic memory reservation of the packet buffer memory space is made depending on the traffic rate profile and the current usage of the memories. Dynamic memory reservation preferably employs a push-out mechanism (e.g., head-drop) for non-CAR packets. Thus when memory is not congested, all memory space is eligible for non-CAR packets to utilize. However, during times of network congestion, a push-out head-drop mechanism frees memory space for CAR packets. Because non-CAR packets can be pushed out upon detection of network congestion, the memory space occupied by non-CAR packets are effectively seen as free memory space for CAR packets. In contrast with static memory reservation, dynamic memory reservation eliminates the need for hard boundaries to restrict non-CAR packets.
Dynamic memory reservation of packet buffer memory space will be described in more detail with reference to
The multicast (Mcast) packets portion 138 preferably has a statically configured amount of space so as to ensure the quality of Mcast traffic such as streaming and/or interactive audio/video traffic. Because the packet buffer memory 106 utilized by each Mcast packet can only be made available when all corresponding Mcast transmit queue entries 120 have been either transmitted or dropped, pushing out Mcast links in the transmit queue 104 does not necessarily free the space in the packet buffer memory 106. Thus, best effort multicast (Mcast) packets are preferably separated from best-effort unicast (Ucast) packets, i.e., packets coming from and going to a single network. In addition to separating the multicast traffic, the memory space allocated for multicast packets is preferably limited to a predefined maximum or threshold packet memory space. As shown in
Such separation of multicast traffic allows dynamic packet buffer memory allocation as will be described in more detail below to improve the efficiency of packet memory utilization without limiting the quality of multicast traffic. The multicast packet threshold facilitates in tracking segments used by multicast packets. When the multicast threshold is exceeded, the network device preferably tail drops incoming requests to the multicast queue.
Packet buffer memory space is dynamically allocated for the non-CAR unicast packets portion 140. In particular, the network device dynamically allocates (loans) memory reserved for CAR packets and/or multicast packets to non-CAR unicast packets when these two memories are not being fully utilized by CAR packets and/or by multicast packets, respectively. In other words, when the CAR-packet network traffic is not congested, i.e., when memory reserved for CAR packets is not being fully utilized by CAR packets, the network device may dynamically allocate (loan) memory reserved for CAR packets to non-CAR unicast packets. Similarly, when the multicast network traffic is not congested, i.e., when memory reserved for multicast packets is not being fully utilized by multicast packets, the network device may dynamically allocate (loan) memory reserved for multicast packets to non-CAR unicast packets. Such dynamic memory allocation allows non-CAR packets to utilize memory spaced otherwise reserved for CAR packets and/or multicast packets when space is available in either or both of these portions of the packet buffer memory 106. As shown in
On the other hand, when the network device packet memory space becomes congested, a push out mechanism is preferably implemented to push out non-CAR unicast packets from the network device to free up space for incoming CAR packets and/or multicast packets. For example, a head drop mechanism may be implemented. The push out mechanism thus returns memory space previously dynamically allocated (loaned) to non-CAR packets back to CAR or multicast packets. Note that non-CAR unicast packets are preferably sent to separate transmit queues so that they are more accessible for head drop when necessary. As is evident, because a non-CAR unicast packets can be pushed out of the memory space reserved for CAR and/or multicast packets upon detection of network congestion, the memory space occupied by the non-CAR packets in the CAR memory space are effectively free memory space for CAR and multicast packets. Therefore, hard boundaries to restrict non-CAR packets are unnecessary and may be eliminated to thereby improve efficiency.
Referring again to
The control pipe 102 also includes the optional non-CAR counter 114 which may be employed to measure non-CAR packet memory usage. However, the non-CAR counter 114 is not necessary for packet memory management. The control pipe 102 further includes the multicast counter 116 to ensure that the threshold for multicast packets is not exceeded. Although not shown, a free space counter may be employed to track the number of free segments in memory. A predetermined number of memory segments should be kept free to allow for a finite reaction time for the network device (time that it takes a packet to be processed in the control pipe). The free segments portion of memory 132 is shown in
The push-out based dynamic memory allocation mechanism facilitates in supporting more CAR QoS agreements while dedicating less packet buffer memory to meet those QoS agreements. In other words the allocation mechanism provides the ability to support CAR QoS agreements with a low-cost silicon network device or switch by using a relatively small amount of embedded packet buffer (cache) memory. In one embodiment, the embedded packet buffer (cache) memory can be approximately 1-2 MB in size but any other suitable memory size may be employed. The memory allocation mechanism also allows for the ability to share CAR and non-CAR memory resources while at the same time guaranteeing availability of resources for CAR packets whenever it is needed.
In addition to the non-CAR unicast packet push out mechanism, the control pipe preferably also detects network congestion to begin head-dropping and tail-dropping packets. To detect network congestion, the free memory space in the packet memory is monitored. If the free memory space crosses a predetermined threshold, the push out process will begin. The threshold only needs to match the push out speed in the PMM. For example, if the PMM takes 30 clocks to start wire-speed (full speed) dropping, the threshold only needs to trigger before the free memory space falls below a level requires to store packets that may arrive over a 30 clock period. This makes most of the memory available for storing the packets rather than reserving an unnecessarily large amount of memory space as a buffer zone in order for the packet dropping mechanism to function properly.
Preferably, the control pipe detects network congestion by implementing two buffer congestion thresholds MAX and HIGH. The control pipe head and tail drops non-CAR unicast packets when the HIGH buffer congestion threshold is crossed. When the MAX threshold is also crossed, the control pipe preferably implements a more aggressive packet selection for dropping than is the case when the HIGH threshold is crossed.
If the incoming packet is determined not to be an in-profile CAR packet or an in-profile multicast packet at 182, then the packet is a non-CAR packet. In which case, the non-CAR packet is queued at 190 and is subject to be pushed out (e.g., head dropped).
As is evident, the CAR architecture facilitates in guaranteeing minimum packet memory space and transmit queue entries for CAR packets, sharing memory across as many traffic classes as possible such as by providing dynamic rather than fixed boundary between CAR and non-CAR memory spaces, providing separate queue and threshold for multicast packets, and providing the capability to provide best effort service for out of profile CAR packets.
With the above-described CAR architecture, the network switch can handle all types of network traffic and address supervision problems encountered with networks where Mcast burst issues are common. The CAR mechanism lowers the costs in supervising a network for congestion yet allows higher quality of service for QoS traffic groups. Such architecture facilitates in providing CAR in a low-cost enterprise network device.
While various embodiments are described and illustrated herein, it will be appreciated that they are merely illustrative and that modifications can be made to these embodiments without departing from the spirit and scope of the invention. Thus, the invention is intended to be defined only in terms of the following claims.
