Congestion avoidance techniques are essential to the operation of networks and network devices. One such technique known in the art as “Random Early Discard” or “RED” is described in a publication by S. Floyd and V. Jacobson entitled “Random Early Detection Gateways for Congestion Avoidance,” (Transactions on Networking, August 1993), which is hereby incorporated by reference for all purposes.
The basic principle behind RED is to control the average length of a network device's (e.g., a router's) output queue in order to avoid long-term congestion. To achieve this goal, RED must work tightly coupled with transport protocols, such as TCP, which are equipped with their own congestion avoidance mechanisms and are thus capable to react to congestion indications generated by RED routers.
It is responsibility of the transport protocol to take the appropriate countermeasures when it detects packets marked by RED. One explicit method of marking packets in this context is described in RFC 3168, “The Addition of Explicit Congestion Notification (ECN) to IP” (K. Ramakrishnan et al., September 2001), which is hereby incorporated by reference. When TCP is being used in the absence of an explicit method of marking packets, packets can only be “marked” by discarding them, with TCP interpreting the loss of packets as a congestion indication. When packet drops are detected, TCP sources immediately reduce their transmission rate, causing a reduction of the traffic volume at the congested router(s). Discarding packets is also a useful means to control average queue size when non-reactive transport protocols such as UDP are exploited.
As noted in the Background section of the AQM Application, the RED algorithm presents scalability issues and other challenges. Moreover, as the speed of network traffic increases, controlling network congestion in an acceptable manner becomes increasingly challenging. This is true in part because it is not economically feasible to increase buffer sizes in proportion to the higher network speeds. High speed, coupled with proportionally smaller buffer sizes and low latency, causes buffers to fill up very quickly when congestion arises.
Some exemplary high-speed, low latency networks having relatively small buffers, which will be referred to herein as Data Center Ethernet (“DCE”) or the like, are described in U.S. patent application Ser. No. 11/084,587, entitled “Ethernet Extension for the Data Center” and filed on Mar. 18, 2005, to U.S. patent application Ser. No. 11/078,992, entitled “Fibre Channel Over Ethernet” and filed on Mar. 10, 2005 and to U.S. patent application Ser. No. 11/094,877, entitled “Network Device Architecture for Consolidating Input/Output and Reducing Latency” and filed on Mar. 30, 2005, (the “DCE Applications”), all of which are incorporated by reference for all purposes.
DCE networks are a challenging environment for congestion management because of their high speed (minimum 10 Gbps) and low latency (few microseconds of round trip). Also, in certain cases, such networks make use of 802.3X link-level flow control to guarantee zero packet loss to applications. If link-level flow-control is being used, congestion spreads almost instantly.
Prior art congestion control techniques such as RED and ECN have been shown to work poorly with small buffers because of the extremely compressed dynamics exhibited by such buffers. In fact, under congestion conditions a buffer in a DCE network fills up instantly when such techniques are employed, causing RED or ECN to work in the region of maximum drop/mark probability. This, in turn, causes the traffic flows to slow down more than necessary, which causes a loss of throughput.
More advanced congestion control mechanisms tailored for networks characterized by operational parameters similar to DCE have been considered. One such mechanism is Fibre Channel Congestion Control (“FCC”), a congestion management mechanism for Fibre Channel networks that is described in co-pending U.S. patent application Ser. No. 10/777,886, entitled “End-to-End Congestion Control in a Fibre Channel Network” and filed on Feb. 11, 2004, which is a continuation-in-part of co-pending U.S. patent application Ser. No. 10/026,583, entitled “Methods and Apparatus for Network Congestion Control” and filed on Dec. 18, 2001, both of which are incorporated herein by reference for all purposes.
While quite effective at controlling congestion when it arises, FCC uses a conservative, time-driven rate recovery process to accelerate traffic flows when congestion is improving. Therefore, FCC may take a longer-than-optimal time to recover the original rate of traffic flows in congested high-speed, low-latency networks such as DCE networks.
Many of the congestion management challenges of DCE networks are shared by other networks, including but not limited to Fibre Channel networks and high-speed Ethernet. It would be very desirable to implement methods and devices that address at least some of the shortcomings of the prior art.
The present invention provides improved methods and devices for managing network traffic. Preferred implementations of the invention allow congestion to be pushed from congestion points in the core of a network to reaction points, which may be edge devices, host devices or components thereof. Preferably, rate limiters shape individual flows of the reaction points that are causing congestion. Parameters of these rate limiters are preferably tuned based on feedback from congestion points, e.g., in the form of backward congestion notification (“BCN”) messages. In some implementations, such BCN messages include congestion change information and at least one instantaneous measure of congestion. The instantaneous measure(s) of congestion may be relative to a threshold of a particular queue and/or relative to a threshold of a buffer that includes a plurality of queues.
Some implementations of the invention provide a congestion management method that includes the following steps: detecting network congestion at a first congestion point of a network; identifying a first congested entity of the network; calculating feedback information regarding a congestion level of the congested entity; and sending a first feedback message to a first reaction point of the network. The reaction point is associated with one or more traffic flows causing the congestion, at least in part. The feedback message includes the feedback information and identity data for the congested entity.
The feedback information may comprise an instantaneous measure of congestion and congestion change information. The instantaneous measure of congestion and the congestion change information may be determined with reference to a predetermined threshold of a queue. The predetermined threshold may decrease as a number of active virtual output queues (“VOQs”) in a buffer of a congestion point increases and the first predetermined threshold may increase as the number of active VOQs in the buffer decreases.
The first feedback message may be an indication to slow down a traffic flow, an indication to speed up a traffic flow or an indication to stop a traffic flow. The first feedback message preferably identifies a particular flow. The congested entity may be a queue.
The detecting step may involve sampling a frame and determining whether a sampled frame includes data that is responsive to a feedback message. When it is determined that the sampled frame includes responsive data, the method may also include these steps: determining that the responsive data identify the first congested entity; determining that the occupancy of a queue to which the sampled frame will be added is currently above a first predetermined threshold; and sending a second feedback message to a source address of the sampled frame. The second feedback message comprises an indication to slow down a traffic flow.
When it is determined that the sampled frame includes responsive data, the method may also include these steps: determining that the responsive data identify the first congested entity; determining that the occupancy of a queue to which the sampled frame will be added is currently above a second predetermined threshold; and sending a second feedback message to a source address of the sampled frame. The second feedback message comprises an indication to stop a traffic flow.
When it is determined that the sampled frame includes responsive data, the method may also include these steps: determining that the responsive data identify the first congested entity; determining that the occupancy of a buffer of the congestion point is above a buffer congestion threshold; and sending a second feedback message to a source address of the sampled frame. The second feedback message comprises an indication that the occupancy of the buffer is above the buffer congestion threshold.
When it is determined that the sampled frame does not include responsive data, the method may further comprise the steps of determining that the occupancy of a queue to which the sampled frame will be added is currently below a first predetermined threshold and determining not to send a second feedback message to a source address of the sampled frame.
Alternative methods of the invention control rates of traffic injected into a network. One such method includes these steps: receiving a first feedback message from a congestion point of a network, the first feedback message comprising an instantaneous measure of congestion for the congestion point, congestion change information for the congestion point and identity data for the congestion point; calculating a feedback signal based, at least in part, on information in the first feedback message; and adjusting a flow rate of traffic addressed to the congestion point according to the feedback signal.
The first feedback message may identify a particular flow. The congested entity may comprise a queue. The calculating step may involve calculating the feedback signal based on the instantaneous measure of congestion and the congestion change information for the congestion point.
The first feedback message may also comprise an indication that the occupancy of a buffer of the congestion point is above a buffer congestion threshold. If so, the calculating step may involve calculating the maximum negative value of the feedback signal.
The method may include the step of adding a tag to each frame sent to the congestion point. The tag includes data responsive to the first feedback message.
All of the foregoing methods, along with other methods of the present invention, may be implemented by software, firmware and/or hardware. For example, at least some methods of the present invention may be implemented by computer programs embodied in machine-readable media. Some aspects of the invention can be implemented by network devices or portions thereof, such as an ingress port of an edge network device or an egress port of a host device's network interface card.
In this application, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be obvious, however, to one skilled in the art, that the present invention may be practiced without some or all of these specific details. In other instances, well known process steps have not been described in detail in order not to obscure the present invention.
The present invention provides congestion management methods and devices that are particularly suitable for network devices, such as switches and routers. Some aspects of the present invention are particularly suitable for implementing a Data Center Ethernet (“DCE”) solution, which simplifies the connectivity of data centers and provides a high bandwidth, low latency network for carrying Ethernet and storage traffic. Some exemplary DCE methods and devices are described in the DCE Applications, which have been incorporated by reference herein. However, the present invention has wide applicability outside of the DCE context and is suitable for Fibre Channel networks, IP networks, etc, potentially any kind of packet switched network.
In this example, core switch 140 is a “congestion point” that detects the congestion condition. According to preferred implementations of the invention, as soon as a congestion point detects congestion, it starts sending explicit feedback messages to the reaction points associated with the traffic flows causing such congestion. Such feedback messages will sometimes be referenced herein as backwards congestion notification (“BCN”) messages, BCN frames, or the like. In some such implementations, the feedback message is an Ethernet frame, which may have a format similar to that of the frame depicted in
In this example, core switch 140 causes “slow-down” BCN messages 180 and 190 to be sent towards end nodes 115 and 135. These messages will also be referred to herein as a “negative BCN feedback messages” or the like. Such messages (and other BCN messages that are described below) are processed at “reaction points,” where congestion mitigation measures are put into place. The reaction points could be edge switches 110 and 130, or, in some implementations, end nodes 115 and 135.
The processing of a negative BCN feedback message will result in the instantiation of a filter/rate limiter (or a further slow down of the one(s) already instantiated, if any) at the reaction point. The purpose of the rate limiter is to slow down a congesting traffic flow to mitigate congestion at the core switch. If congestion should improve (or dissipate completely), “speed-up” messages (also referred to herein as “positive BCN feedback messages” or the like) will cause the rate limiters to increase their rate to avoid wasting bandwidth at the congestion point.
However, core switch 140 has detected congestion. First, core switch 140 has sent negative BCN feedback message 220 to a reaction point (edge switch 110), indicating that edge switch 110 should slow down its rate of transmission. Preferably, negative BCN feedback message 220 includes sufficient detail to allow edge switch 110 to identify a particular traffic flow (i.e., a layer 2 flow, a layer 3 flow, or a layer 4 flow) that needs to be slowed. A BCN frame is generated by a congestion point by sampling incoming traffic, e.g., as described below. In this example, core switch 140 has subsequently sent a “stop” BCN message 230 to edge switch 110. As described in more detail below, a “stop” BCN message 230 will cause a reaction point to stop transmitting data (preferably on a specified data flow) for a period of time.
One exemplary BCN frame is depicted in
In this example, field 320 is an IEEE 802.1Q tag that carries the VLAN of the sampled frame and the Priority field indicating the highest priority. Field 320 will indicate a null VLAN in two instances: (1) if the sampled frame did not carry an 802.1Q tag or (2) if the VLAN field of such tag indicated a null value. Field 325 identifies the frame as being a BCN feedback message, in this example by indicating a predetermined EtherType This EtherType could be any of the currently unassigned EtherTypes, e.g., as per http://www.iana.org/assignments/ethernet-numbers. These EtherTypes are assignable by the IEEE Standards Department, 445 Hoes Lane, P.O. Box 1331, Piscataway, N.J. 08855-1331.
Version field 330 indicates the version of the BCN protocol. In this example, three bits following version field 330 change the semantics of the BCN message when they are set. The meaning of these bits will be described below. Q bit 331 indicates that Qdelta is saturated. In the example described below, Qdelta is saturated when its value is either equal to −2Qeq or −2Qeq. M bit 332 indicates a condition of mild congestion, whereas S bit 333 indicates a condition of severe congestion. Reserved bits in field 335 are not used in this example. Instead, they are set to zero on transmission and ignored on reception. Future versions of the BCN protocol may redefine all or some of the reserved bits.
Field 340 indicates a congestion point identifier (“CPID”). A primary purpose of the CPID is to identify a congested entity in the network. In this example, the congested entity is a queue of core switch 140. This information is sent to a reaction point in order to create an association between the congested entity and the reaction point.
The contents of timestamp field 350 and unit field 352 are copied from the homonymous fields of a Rate Limited Tag (“RLT”) of the sampled frame. RLTs will be described below with reference to
Qoff field 355 and Qdelta field 360 contain quantitative feedback information conveyed by the congestion point to the reaction point. The use of such fields will be described below with reference to
Field 365 of BCN frame 305 consists of the first N bytes of the sampled frame. N is a configurable parameter, and it has a minimum value is such that the resulting BCN frame is always guaranteed to be as large as, or larger than, a minimum-sized frame of the type used to implement the invention (e.g., a minimum-sized Ethernet frame of 64 bytes). For example, in the case of BCN frame 305 of
BCN frame 505 includes outer destination address field 510, which indicates the outer source address of the sampled frame. Field 515 indicates the outer source address of the congestion point, which is a hierarchical MAC address in this example. Field 520 indicates the outer S-Tag (the outer IEEE 802.1Q tag) of the sampled frame. Field 525 indicates that frame 505 is a MAC-in-MAC frame.
Field 530 indicates the inner destination address, which is the inner source address of the sampled frame. Fields 535 through 580 correspond generally with fields 315 through 370 of BCN frame 305. The VLAN field of the inner and outer S-Tags 540 and 520 (a.k.a. B-Tag in 802.1ah) should be the same as the VLAN field of the 802.1Q field of the sampled frame. The priority field of the outer S-Tag 520 should be set to the highest level of priority, while the same field of the inner S-Tag 540 is the priority field of the sampled packet.
When edge switch 110 receives a BCN frame from congestion point 140 and such message is intended to cause a congestion mitigation action to be undertaken on a particular data flow (e.g., the installation of a rate limiter or the slowing down of an existing one), edge switch 110 stores a CPID in a local register associated with such data flow. All the frames 240 belonging to that flow that are subsequently injected by edge switch 110 in the network will carry a Rate Limited Tag (“RLT”) containing the CPID.
One exemplary rate-limited frame 400 is illustrated in
Fields 410 through 427, shown in bold in
CPID field 415 indicates the congestion point to which the RLT pertains. This information may be used to complete the association between a reaction point and the corresponding congestion point. One important purpose of this association is to prevent a reaction point from receiving positive feedback from multiple congestion points for the same flow. Preferably, a congestion point will generate BCN feedback messages only on flows whose frames that carry an RLT tag with a CPID matching its own ID. As noted above, when a reaction point receives a BCN frame from a congestion point and such message causes a congestion mitigation action to be undertaken on a particular data flow, the reaction point associates the CPID with the data flow, e.g. by saving the CPID in a local register associated with such data flow. Field 420 is reserved.
Timestamp field 425 may be used to estimate the round trip time between the reaction point and the congestion point with which it is associated. Each time a reaction point inserts an RLT tag in a frame it is going to transmit, the current value of a local free running timer is copied into timestamp field 425. Unit field 427 indicates the time units used by the free running timer. The resolution of this free running timer may be, for example, a value in the range 1 μs to 100 μs. As noted above, when a frame having an RLT tag is sampled by a congestion point, the contents of timestamp field 425 and unit field 427 are copied and inserted in timestamp field 350 and unit field 352 of a BCN frame generated by the congestion point.
Exemplary methods for congestion detection and for generating BCN frames at a congestion point will now be described with reference to
Incoming frames are sampled with a certain probability P 620. P 620 is a configurable parameter, the selection of which is a tradeoff between the usefulness of more frequent congestion detection and the overhead required for more frequent sampling and computation. In some preferred implementations, P 620 is in the range of 0.001 to 0.1; in some such implementations, P 620 is 0.01. The values of Qeq 610, Qsc 615 and P 620 should be established before the other steps shown in
In step 625, a congestion point determines whether or not to sample a frame. If no frame is sampled, no BCN frame will be generated at that moment. When a frame is sampled, the process continues to step 635, wherein the sampled frame is evaluated.
In this example, when the length of queue 605 is below Qeq, the treatment of sampled frames will differ according to whether the sampled frame carries an RLT tag having a CPID that identifies the congestion point. If the sampled frame does not carry such an RLT tag and the length of the queue below Qeq, no BCN Frame is generated (message generation scheme 640) and sent (step 642). However, if the sampled frame does carry such an RLT tag and the length of the queue below Qeq, a BCN Frame is generated (message generation scheme 660) and sent (step 642).
In other words, in this implementation, if the sampled frame carries an RLT tag the congestion point generates a BCN frame irrespective of the current queue length if and only if its congestion point identifier matches the CPID field in the RLT tag. When such a match occurs, the timestamp field of the RLT tag is copied into the corresponding field of the BCN Frame.
In this implementation, when the queue length is above Qeq, the Congestion Point will generate either a regular BCN feedback message or a “stop” BCN feedback message irrespective of the CPID field in the RLT tag. In this example, if the length of queue 605 is ≧Qeq and is ≦Qsc, a negative BCN feedback message is generated whether or not the packet carries an RLT tag, and whether or not the CPID of the RLT tag (if any) matches the congestion point ID. A “stop” BCN feedback message is generated when the length of the queue is >Qsc.
In this example, a BCN feedback message includes two fields, Qoff and Qdelta. Qoff is an instantaneous measure of congestion, which in this example is the offset of the current queue length with respect to the equilibrium threshold Qeq. Here, Qoff is saturated at +Qeq and −Qeq. Here, a BCN feedback message also includes congestion change information. Here, the congestion change information is Qdelta, which is the change in length of the queue since the last sampled frame. In this example, Qdelta is saturated at +2Qeq and −2Qeq. When Qdelta saturates, the Q bit in the BCN Frame is set. A “stop” BCN feedback message is indicated by zero values for Qoff and Qdelta. In fact, since a BCN message is not generated when a frame is sampled and Qoff and Qdelta are both zero, this combination may be used to identify a “stop” BCN message.
Qdelta may be calculated according to at least two methods. In the first method, Qdelta is the difference between the current queue length and the queue length at the previous time of sampling. In a second method, Qdelta is the difference between the number of packets (or other data units) added to the queue and the number of packets (or other data units) removed from the queue since the last time of sampling. The first method is more accurate but requires that an indication of the previous queue length be stored in memory. The second method requires a smaller amount of state to be kept, but may be prone to error accumulation.
After BCN frames have been received indicating congestion that pertains to this reaction point (e.g., as in the state of edge device 110 in
According to some implementations of the invention, the rate control algorithm used by rate limiters 740 works according to a Feedback Signal Fb that is calculated, e.g., according to Equation (1):
Fb=(Qoff−w·Qdelta) Equation (1)
In Equation (1), w is a parameter used to weight the derivative component Qdelta (which is also referred to herein as the congestion change component or the like) more or less with respect to the offset component Qoff (which is also referred to herein as the instantaneous measure of congestion or the like). The values of Qoff and Qdelta are determined from BCN frames received by a reaction point. Based on the sign of the Feedback Signal Fb, in some implementations of the invention the rate R is increased or decreased as follows:
If Fb>0 R=R+Gi·Fb·Ru Equation (2)
If Fb<0 R=R·(1−Gd·|Fb|) Equation (3)
If Fb=0, R is unchanged. Here, Gi and Gd are the Increase Gain and Decrease Gain respectively, and Ru is the Rate Unit (i.e., the granularity of the rate adjustment) employed by the rate limiters. In one example, Gi=1, Ru=8 Mbps and Gd= 1/64. However, these values are merely exemplary and the variables of Equations (2) and (3) may be optimized according to the implementation. The calculations are preferably done in the reaction point. In alternative implementations, the calculations are done elsewhere, e.g., in the detection point. However, if the calculations are performed in a location other than the reaction point, the most effective use of timestamps will be inhibited.
It will be observed that in implementations that use equations in the general form of Equations (2) and (3) to control changes in R, the rates are decreased more aggressively when Fb<0 (a multiplicative decrease) than the rates are increased when Fb>0 (an additive increase). This is desirable in order to avoid filling the buffers of a congestion point too quickly due to a slow response to detected congestion or due to a too-rapid increase in flow when congestion is abating.
A limited number of filters/rate limiters may be available. There may be cases when all the filters have been used and a BCN message is received which should cause the instantiation of a new filter/rate limiter pair. In such cases, a number of actions may be taken, e.g.: (1) aggregate all the filters/rate limiters in a single filter/rate limiter that controls the entire traffic originated by and end system; (2) aggregate filters/rate limiters in an “intelligent” way, e.g., use the same filter/rate limiter for all the traffic flows sharing the same destination address, etc; or (3) aggregate filters/rate limiters in a “less intelligent” way, e.g., use the same filter/rate limiter for all the traffic flows sharing the same bucket based on an hash function of the frame header.
When a reaction point receives a BCN Frame, the difference between the current time and the time indicated in the timestamp field of the BCN Frame is calculated. This difference is the last measure of the round trip time between the reaction point and the congestion point. This measure may be averaged out, for example using an Exponential Weighted Moving Average similar to the one used by WRED, and used to dynamically adjust the value of some of the reaction parameters. For example, a reaction point may have a number of tables containing different values of the w, Gi, and Gd parameters precalculated based on different round-trip times. The current value of the averaged round-trip time may be used to select the table of parameters that best suite the current loop delay.
Once a rate limiter has been instantiated, it may be reclaimed once two conditions are satisfied: (1) the queue of the rate limiter is empty, and (2) its rate is at or above the line-rate. These two conditions are necessary to avoid out of order packet delivery.
Each rate limiter is associated with a timer that is reset every time a BCN Frame is received. If this timer expires, it means that the corresponding rate limiter has not received BCN Frames for the entire duration of the timeout period. This may happen, for example, because the traffic stream that that rate limiter was controlling has suddenly ended. Alternatively, this may occur because routing issues in the network are preventing BCN Frames from reaching the reaction point. To reclaim a rate limiter that may potentially be stale, various implementations of the invention employ a variety of solutions. In some implementations, the rate limiter is immediately freed up at the timeout expiration. In other implementations, the rate of the rate limiter starts automatically increasing when the timer expires. This increase may continue, for example, until the conditions for the filter reclaiming are met or BCN frames are eventually received. In other implementations, management software is notified (e.g., via an interrupt) of the anomaly and the management software is allowed to deal with the issue.
Rate limiters use a certain amount of buffer space to store frames held in their queues. Therefore, an active queue management mechanism may advantageously be used to prevent such buffers from overflowing. Traditional AQM techniques such as RED do not work well in such conditions because of the limited buffer and flow dynamics. An alternative AQM algorithm of the present invention may be implemented as follows. First, a threshold Qaqm is associated with the rate limiter queues. If the length of a rate limiter queue is below the Qaqm threshold, no action is taken. If the length of the rate limiter is above the Qaqm threshold, a packet is dropped/marked with a certain fixed probability (e.g., a probability in the range of 0.1 to 0.001).
If reactive and non-reactive flows (such as TCP and UDP flows) are sharing the same rate limiter queue, two separate packet counters are introduced. One packet counter is used for counting reactive packets in and the other for non-reactive packets stored in the queue. The AQM algorithm described in the previous paragraph could be implemented in the same way, except that for non-reactive flows the drop probability is 1.
An active filter 720 may change its association with a congestion point over time. The association can be changed when a negative BCN Frame is received from a congestion point different from the one currently associated with the filter. For example, if a traffic flow is subject to congestion at congestion point CP1 (and therefore is filtered and rate-controlled according to feedback from CP1) starts experiencing congestion at congestion point CP2, CP2 will generate negative a BCN frame for that flow, causing its filter to change association from CP1 to CP2. After some time, the negative feedback generated by one of the two congestion points will prevail and the filter will settle its association with that congestion point.
When a congestion point is subject to severe congestion, it may send a “stop” BCN feedback message. Such a message is also referred to herein as a “BCN0” message or the like because in some implementations a “stop” BCN feedback message is a BCN message with Qoff=0 and Qdelta=0.
Referring now to graph 805 of
After the timer expiration, Tmax is doubled and Rmin is halved, so that the next BCN0 will cause the random timer to have a longer duration and the rate limiter to restart from a slower rate, effectively realizing an exponential back-off. The initial values of Tmax and Rmin are restored upon the reception of the first positive feedback. During the timeout period, i.e., while the random timer is running, all BCN messages, including BCN0, must be ignored.
The same timer may be used if, for any reason, the rate of a rate limiter becomes smaller that Rmin. When this happens, the random timer is started. When it expires, the rate of the rate limiter is set to Rmin.
Special handling of the BCN message is required when any of the Q bits is set in the BCN Frame. When this bit is set, the Qdelta parameter is saturated at 2Qeq or −2Qeq. When this happens, a stronger rate adjustment must be performed because the system is working outside of the linear region. The saturation feedback signal is calculated as follow:
The rate adjustment is then performed as usual, i.e.:
The saturation feedback generates a rate adjustment twice as big as the maximum rate adjustment.
It will often be the case that a queue considered herein is part of a VOQ system wherein an unpredictable number of queues may be sharing a common buffer at any given time. In such circumstances, it may be beneficial to tune or modify the previously-described methods of the present invention according to the state of the VOQ system and the associated buffer. The larger the number of VOQs sharing the same physical or logical buffer, the lower the equilibrium threshold Qeq should be kept. Accordingly, some implementations of the invention provide a dynamic equilibrium threshold Qeq that responds to such conditions by decreasing Qeq as the number of active VOQs increases and increasing Qeq as the number of active VOQs decreases.
Moreover, the more that a common buffer is congested, the stronger the reaction implemented by the reaction points should be. In some implementations of the invention, the overall occupancy of a buffer will override the previously-described methods for implementing BCN messages according to indications from individual queues. One such implementation will now be described with reference to
In this example, when the occupancy of buffer 905 increases beyond mild congestion threshold (“Bmc”), the M bit will be set in the BCN frame (e.g., in reserved area 335 of frame 305 (see
When the severe congestion threshold (“Bsc”) is crossed, the S bit will be set in the BCN frame. If the reaction point detects that the S bit has been set, the reaction point will translate any corresponding BCN indication to be a “stop” BCN indication and will respond accordingly.
The interfaces 1068 are typically provided as interface cards (sometimes referred to as “line cards”). Generally, interfaces 1068 control the sending and receiving of data packets over the network and sometimes support other peripherals used with the network device 1060. Among the interfaces that may be provided are Fibre Channel (“FC”) interfaces, Ethernet interfaces, frame relay interfaces, cable interfaces, DSL interfaces, token ring interfaces, and the like. In addition, various very high-speed interfaces may be provided, such as fast Ethernet interfaces, Gigabit Ethernet interfaces, ATM interfaces, HSSI interfaces, POS interfaces, FDDI interfaces, ASI interfaces, DHEI interfaces and the like.
When acting under the control of appropriate software or firmware, in some implementations of the invention CPU 1062 may be responsible for implementing specific functions associated with the functions of a desired network device. According to some embodiments, CPU 1062 accomplishes all these functions under the control of software including an operating system (e.g. Linux, VxWorks, etc.), and any appropriate applications software.
CPU 1062 may include one or more processors 1063 such as a processor from the Motorola family of microprocessors or the MIPS family of microprocessors. In an alternative embodiment, processor 1063 is specially designed hardware for controlling the operations of network device 1060. In a specific embodiment, a memory 1061 (such as non-volatile RAM and/or ROM) also forms part of CPU 1062. However, there are many different ways in which memory could be coupled to the system. Memory block 1061 may be used for a variety of purposes such as, for example, caching and/or storing data, programming instructions, etc.
Regardless of network device's configuration, it may employ one or more memories or memory modules (such as, for example, memory block 1065) configured to store data, program instructions for the general-purpose network operations and/or other information relating to the functionality of the techniques described herein. The program instructions may control the operation of an operating system and/or one or more applications, for example.
Because such information and program instructions may be employed to implement the systems/methods described herein, the present invention relates to machine-readable media that include program instructions, state information, etc. for performing various operations described herein. Examples of machine-readable media include, but are not limited to, magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROM disks; magneto-optical media; and hardware devices that are specially configured to store and perform program instructions, such as read-only memory devices (ROM) and random access memory (RAM). The invention may also be embodied in a carrier wave traveling over an appropriate medium such as airwaves, optical lines, electric lines, etc. Examples of program instructions include both machine code, such as produced by a compiler, and files containing higher level code that may be executed by the computer using an interpreter.
Although the system shown in
Although illustrative embodiments and applications of this invention are shown and described herein, many variations and modifications are possible which remain within the concept, scope, and spirit of the invention, and these variations would become clear to those of ordinary skill in the art after perusal of this application.
Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and the invention is not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.
This application is a divisional of U.S. application Ser. No. 11/248,933, entitled “Methods and Devices for Backward Congestion Notification,” filed on Oct. 11, 2005, by Davide Bergamasco et al, which is incorporated herein by reference in its entirety for all purposes. The present application claims priority to and benefit of this application. This application is related to U.S. patent application Ser. No. 11/155,388, entitled “ACTIVE QUEUE MANAGEMENT METHODS AND DEVICES” and filed on Jun. 16, 2005 (the “AQM Application”), which is hereby incorporated by reference for all purposes.
Number | Name | Date | Kind |
---|---|---|---|
5402416 | Cieslak et al. | Mar 1995 | A |
5526350 | Gittins et al. | Jun 1996 | A |
5742604 | Edsall et al. | Apr 1998 | A |
5920566 | Hendel et al. | Jul 1999 | A |
5946313 | Allan et al. | Aug 1999 | A |
5974467 | Haddock et al. | Oct 1999 | A |
6021124 | Haartsen | Feb 2000 | A |
6078586 | Dugan et al. | Jun 2000 | A |
6104699 | Holender et al. | Aug 2000 | A |
6195356 | Anello et al. | Feb 2001 | B1 |
6201789 | Witkowski et al. | Mar 2001 | B1 |
6236652 | Preston et al. | May 2001 | B1 |
6333917 | Lyon et al. | Dec 2001 | B1 |
6397260 | Wils et al. | May 2002 | B1 |
6404768 | Basak et al. | Jun 2002 | B1 |
6414939 | Yamato | Jul 2002 | B1 |
6415323 | McCanne et al. | Jul 2002 | B1 |
6456590 | Ren et al. | Sep 2002 | B1 |
6456597 | Bare | Sep 2002 | B1 |
6459698 | Acharya | Oct 2002 | B1 |
6504836 | Li et al. | Jan 2003 | B1 |
6529489 | Kikuchi et al. | Mar 2003 | B1 |
6556541 | Bare | Apr 2003 | B1 |
6556578 | Silberschatz et al. | Apr 2003 | B1 |
6560198 | Ott et al. | May 2003 | B1 |
6587436 | Vu et al. | Jul 2003 | B1 |
6611872 | McCanne | Aug 2003 | B1 |
6636524 | Chen et al. | Oct 2003 | B1 |
6650623 | Varma et al. | Nov 2003 | B1 |
6671258 | Bonneau | Dec 2003 | B1 |
6721316 | Epps et al. | Apr 2004 | B1 |
6724725 | Dreyer et al. | Apr 2004 | B1 |
6785704 | McCanne | Aug 2004 | B1 |
6839794 | Schober | Jan 2005 | B1 |
6885633 | Mikkonen | Apr 2005 | B1 |
6888824 | Fang et al. | May 2005 | B1 |
6901593 | Aweya et al. | May 2005 | B2 |
6904507 | Gil | Jun 2005 | B2 |
6917986 | Mor et al. | Jul 2005 | B2 |
6922408 | Bloch et al. | Jul 2005 | B2 |
6934256 | Jacobson et al. | Aug 2005 | B1 |
6934292 | Ammitzboell | Aug 2005 | B1 |
6946313 | Gottfried | Sep 2005 | B2 |
6975593 | Collier et al. | Dec 2005 | B2 |
6976581 | Chen | Dec 2005 | B2 |
6990529 | Yang et al. | Jan 2006 | B2 |
6999462 | Acharya | Feb 2006 | B1 |
7016971 | Recio et al. | Mar 2006 | B1 |
7020715 | Venkataraman et al. | Mar 2006 | B2 |
7046631 | Giroux et al. | May 2006 | B1 |
7046666 | Bollay et al. | May 2006 | B1 |
7047666 | Hahn et al. | May 2006 | B2 |
7093024 | Craddock et al. | Aug 2006 | B2 |
7133405 | Graham et al. | Nov 2006 | B2 |
7133416 | Chamdani | Nov 2006 | B1 |
7158480 | Firoiu et al. | Jan 2007 | B1 |
7187688 | Garmire et al. | Mar 2007 | B2 |
7190667 | Susnow et al. | Mar 2007 | B2 |
7197047 | Latif et al. | Mar 2007 | B2 |
7209478 | Rojas et al. | Apr 2007 | B2 |
7209489 | Bailey et al. | Apr 2007 | B1 |
7221656 | Aweya et al. | May 2007 | B1 |
7225364 | Carnevale et al. | May 2007 | B2 |
7246168 | Bales | Jul 2007 | B1 |
7266122 | Hogg et al. | Sep 2007 | B1 |
7266598 | Rolia | Sep 2007 | B2 |
7277391 | Aweya et al. | Oct 2007 | B1 |
7286485 | Ouellette et al. | Oct 2007 | B1 |
7319669 | Kunz et al. | Jan 2008 | B1 |
7342934 | Mott et al. | Mar 2008 | B1 |
7349334 | Rider | Mar 2008 | B2 |
7349336 | Mathews et al. | Mar 2008 | B2 |
7359321 | Sindhu et al. | Apr 2008 | B1 |
7385997 | Gorti et al. | Jun 2008 | B2 |
7400590 | Rygh et al. | Jul 2008 | B1 |
7400634 | Higashitaniguchi et al. | Jul 2008 | B2 |
7406092 | Dropps et al. | Jul 2008 | B2 |
7436845 | Rygh et al. | Oct 2008 | B1 |
7469298 | Kitada et al. | Dec 2008 | B2 |
7486689 | Mott | Feb 2009 | B1 |
7525983 | Dropps et al. | Apr 2009 | B2 |
7529243 | Sodder et al. | May 2009 | B2 |
7561571 | Lovett et al. | Jul 2009 | B1 |
7564789 | Betker | Jul 2009 | B2 |
7564869 | Cafiero et al. | Jul 2009 | B2 |
7596627 | Cometto et al. | Sep 2009 | B2 |
7602720 | Bergamasco et al. | Oct 2009 | B2 |
7684326 | Nation et al. | Mar 2010 | B2 |
7721324 | Jackson | May 2010 | B1 |
7756027 | Reddy et al. | Jul 2010 | B1 |
7801125 | Kreeger et al. | Sep 2010 | B2 |
7826452 | Bishara et al. | Nov 2010 | B1 |
7830793 | Gai et al. | Nov 2010 | B2 |
7961621 | Bergamasco et al. | Jun 2011 | B2 |
7996971 | Newton | Aug 2011 | B2 |
8121138 | Miyabe | Feb 2012 | B2 |
8149710 | Bergamasco et al. | Apr 2012 | B2 |
8160094 | Cafiero et al. | Apr 2012 | B2 |
8238347 | DeSanti et al. | Aug 2012 | B2 |
8259720 | Farinacci et al. | Sep 2012 | B2 |
8385208 | Medina | Feb 2013 | B1 |
20010043564 | Bloch et al. | Nov 2001 | A1 |
20010048661 | Clear et al. | Dec 2001 | A1 |
20020023170 | Seaman et al. | Feb 2002 | A1 |
20020046271 | Huang | Apr 2002 | A1 |
20020085493 | Pekkala et al. | Jul 2002 | A1 |
20020085565 | Ku et al. | Jul 2002 | A1 |
20020103631 | Feldmann et al. | Aug 2002 | A1 |
20020141427 | McAlpine | Oct 2002 | A1 |
20020159385 | Susnow et al. | Oct 2002 | A1 |
20020188648 | Aweya et al. | Dec 2002 | A1 |
20020191640 | Haymes et al. | Dec 2002 | A1 |
20030002517 | Takajitsuko et al. | Jan 2003 | A1 |
20030016624 | Bare | Jan 2003 | A1 |
20030026267 | Oberman et al. | Feb 2003 | A1 |
20030037127 | Shah et al. | Feb 2003 | A1 |
20030037163 | Kitada et al. | Feb 2003 | A1 |
20030061379 | Craddock et al. | Mar 2003 | A1 |
20030084219 | Yao et al. | May 2003 | A1 |
20030091037 | Latif et al. | May 2003 | A1 |
20030115355 | Cometto et al. | Jun 2003 | A1 |
20030118030 | Fukuda | Jun 2003 | A1 |
20030128703 | Zhao et al. | Jul 2003 | A1 |
20030152063 | Giese et al. | Aug 2003 | A1 |
20030169690 | Mott | Sep 2003 | A1 |
20030193894 | Tucker et al. | Oct 2003 | A1 |
20030193942 | Gil | Oct 2003 | A1 |
20030195983 | Krause | Oct 2003 | A1 |
20030202536 | Foster et al. | Oct 2003 | A1 |
20030223416 | Rojas et al. | Dec 2003 | A1 |
20030223435 | Gil | Dec 2003 | A1 |
20030227893 | Bajic | Dec 2003 | A1 |
20040008675 | Basso et al. | Jan 2004 | A1 |
20040013088 | Gregg | Jan 2004 | A1 |
20040013124 | Peebles et al. | Jan 2004 | A1 |
20040024903 | Costatino et al. | Feb 2004 | A1 |
20040032856 | Sandstrom | Feb 2004 | A1 |
20040042448 | Lebizay et al. | Mar 2004 | A1 |
20040042477 | Bitar et al. | Mar 2004 | A1 |
20040076175 | Patenaude | Apr 2004 | A1 |
20040078621 | Talaugon et al. | Apr 2004 | A1 |
20040081203 | Sodder et al. | Apr 2004 | A1 |
20040100980 | Jacobs et al. | May 2004 | A1 |
20040120332 | Hendel | Jun 2004 | A1 |
20040156390 | Prasad et al. | Aug 2004 | A1 |
20040196809 | Dillinger et al. | Oct 2004 | A1 |
20040213243 | Lin et al. | Oct 2004 | A1 |
20040240459 | Lo et al. | Dec 2004 | A1 |
20050002329 | Luft et al. | Jan 2005 | A1 |
20050018606 | Dropps et al. | Jan 2005 | A1 |
20050025179 | McLaggan et al. | Feb 2005 | A1 |
20050044141 | Hameleers et al. | Feb 2005 | A1 |
20050047421 | Solomon | Mar 2005 | A1 |
20050060445 | Beukema et al. | Mar 2005 | A1 |
20050138243 | Tierney et al. | Jun 2005 | A1 |
20050141419 | Bergamasco et al. | Jun 2005 | A1 |
20050141568 | Kwak et al. | Jun 2005 | A1 |
20050157750 | Rabie et al. | Jul 2005 | A1 |
20050169188 | Cometto et al. | Aug 2005 | A1 |
20050169270 | Mutou et al. | Aug 2005 | A1 |
20050190752 | Chiou et al. | Sep 2005 | A1 |
20050226149 | Jacobson et al. | Oct 2005 | A1 |
20050238064 | Winter et al. | Oct 2005 | A1 |
20060002385 | Johnsen et al. | Jan 2006 | A1 |
20060023708 | Snively et al. | Feb 2006 | A1 |
20060059213 | Evoy | Mar 2006 | A1 |
20060087989 | Gai et al. | Apr 2006 | A1 |
20060098589 | Kreeger et al. | May 2006 | A1 |
20060098681 | Cafiero et al. | May 2006 | A1 |
20060101140 | Gai et al. | May 2006 | A1 |
20060126509 | Abi-Nassif et al. | Jun 2006 | A1 |
20060146832 | Rampal et al. | Jul 2006 | A1 |
20060171318 | Bergamasco et al. | Aug 2006 | A1 |
20060187832 | Yu | Aug 2006 | A1 |
20060193247 | Naseh et al. | Aug 2006 | A1 |
20060193252 | Naseh et al. | Aug 2006 | A1 |
20060195607 | Naseh et al. | Aug 2006 | A1 |
20060198323 | Finn | Sep 2006 | A1 |
20060209885 | Hain et al. | Sep 2006 | A1 |
20060215550 | Malhotra | Sep 2006 | A1 |
20060251067 | DeSanti et al. | Nov 2006 | A1 |
20070041321 | Sasaki et al. | Feb 2007 | A1 |
20070047443 | Desai et al. | Mar 2007 | A1 |
20070081454 | Bergamasco et al. | Apr 2007 | A1 |
20070115824 | Chandra et al. | May 2007 | A1 |
20070121617 | Kanekar et al. | May 2007 | A1 |
20070133539 | Kang et al. | Jun 2007 | A1 |
20070165632 | Zwiebel | Jul 2007 | A1 |
20070183332 | Oh et al. | Aug 2007 | A1 |
20070233887 | Nubani et al. | Oct 2007 | A1 |
20070280207 | Shimizu et al. | Dec 2007 | A1 |
20080069114 | Shimada et al. | Mar 2008 | A1 |
20080089247 | Sane et al. | Apr 2008 | A1 |
20080186968 | Farinacci et al. | Aug 2008 | A1 |
20080212595 | Figueira et al. | Sep 2008 | A1 |
20080259798 | Loh et al. | Oct 2008 | A1 |
20080273465 | Gusat et al. | Nov 2008 | A1 |
20090010162 | Bergamasco et al. | Jan 2009 | A1 |
20090052326 | Bergamasco et al. | Feb 2009 | A1 |
20090073882 | McAlpine et al. | Mar 2009 | A1 |
20090232138 | Gobara et al. | Sep 2009 | A1 |
20090252038 | Cafiero et al. | Oct 2009 | A1 |
20110007741 | Kreeger et al. | Jan 2011 | A1 |
20110222402 | Gai et al. | Sep 2011 | A1 |
20120195310 | Cafiero et al. | Aug 2012 | A1 |
20120300782 | Farinacci et al. | Nov 2012 | A1 |
20120307638 | Bergamasco et al. | Dec 2012 | A1 |
Number | Date | Country |
---|---|---|
1778079 | May 2004 | CN |
1 028 600 | Aug 2000 | EP |
1206099 | May 2002 | EP |
1 698 600 | Sep 2006 | EP |
WO2004064324 | Jul 2004 | WO |
WO2006047092 | May 2006 | WO |
WO2006047109 | May 2006 | WO |
WO2006.047194 | May 2006 | WO |
WO2006047223 | May 2006 | WO |
2006057730 | Jun 2006 | WO |
WO 2006063922 | Jun 2006 | WO |
WO 2007050250 | May 2007 | WO |
WO 2007121101 | Oct 2007 | WO |
WO 2008097730 | Jan 2008 | WO |
WO 2009006573 | Jan 2009 | WO |
Entry |
---|
A.K. Choudry, et al., “A New Buffer Management Scheme for Hierarchical Shared Memory Switches”, IEEE/ACM Transactions on Networking, 26 pp., 1997. |
A.K. Choudry, et al., “Dynamic Queue Length Thresholds for Shared-Memory Packet Switches”, IEEE/ACM Transactions on Networking, Apr. 1998. |
Cisco Systems, Inc., “Cisco data Center Network Architecture and Solutions Overview,” http://www.cisco.com/application/pdf/en/us/guest/netsol/ns377/c643/cdccont—0900aecd802c9a4f.pdf, 2006. |
F. Kamoun, et al., “Analysis of Shared Finite Storage in a Computer Network Node Environment Under General Traffic Conditions”, IEEE Transactions on Communications, Jul. 1990. |
Floyd et al., Random Early Detection Gateways for Congestion Avoidance, Lawrence Berkeley Laboratory, Univ. of California, IEEE/ACM Transactions on Networking, Aug. 1993, 22 pages. |
IEEE Standards 802.3ah™—2004, IEEE Computer Society, Sep. 7, 2004, 623 pages. |
IEEE Standards 802.3ak™—2004, IEEE Computer Society, Mar. 1, 2004, 52 pages. |
IEEE Standards 802.3™—2002, IEEE Computer Society, Mar. 8, 2002, 1538 pages. |
InfiniBand Arch, Spec, vol. 1. Oct. 24, 2000 Final. Infiniband SM Trade Association. |
InfiniBand Arch, Spec, vol. 2. Oct. 24, 2000 Final. Infiniband SM Trade Association. |
J. Mandavi, et al., “IPPM Metrics for Measuring Connectivity”, RFC 2678, pp. 1-9, Sep. 1999. |
J. Moy, OSPF Version 2 (RFC 2178), Network Working Group, Cascade Communications Corp., Jul. 1997, 211 pp. |
J. Postel, “Internet Control Message Protocol, DARPA Internet Program Protocol Specification”, RFC 792, pp. 1-15, Sep. 1981. |
K. Ramakrishnan et al., “The Addition of Explicit Congestion Notification (ECN to IP,” RFC 3168, Sep. 2001. |
MAC Control PAUSE Operation, 31B.1 PAUSE description, Annex 31B, IEEE Std 802.3, 1998 Edition, 11 pages. |
MAC Control PAUSE Operation, 31B.3.1 Transmit Operation, Annex 31B, IEEE Std 802.3ae-2002, 4 pages. |
MAC Control, IEEE Std 802.3-2002, Section Two, 9 pages. |
Mekkittikul et al., A Practical Scheduling Algorithm to Achieve 100% Throughput in Input-Queued Switches, Computer Systems Laboratory, Stanford University, 1998, 8 pages. |
Sancho et al.; “Analyzing the Influence of Virtual Lanes on the Performance on Infiniband Networks”; 2002; IEEE Proceeding of the International Parallel and Disctributed processing Symposium (IPDPS'02); pp. 1-10. |
Wei Cao Huawei Technologies: “IEEE 802.1ah Mode for Ethernet Over MPLS; draft-cao-pwe3-801-1ah-00.txt” IETF Standard-Workingdraft, Internet Engineering Task Force, IETF, CH, Oct. 16, 2006, XP015047518 ISSN: 000-0004. |
International Search Report and Written Opinion, dated Sep. 21, 2006, from PCT/US05/37069. |
International Search Report and Written Opinion, dated Oct. 16, 2006, from PCT/US05/37765. |
International Search Report and Written Opinion, dated Nov. 1, 2006, from PCT/US05/36700. |
International Search Report and Written Opinion, dated Jan. 16, 2007, from PCT/US05/37239. |
International Search Report and Written Opinion, dated Feb. 7, 2008, from PCT/US07/015506. |
International Search Report and Written Opinion, dated Feb. 20, 2007, from PCT/US05/37651. |
International Search Report and Written Opinion, dated Sep. 27, 2007, from PCT/US06/38858. |
International Search Report and Written Opinion, dated May 23, 2008, from PCT/US08/051986. |
International Search Report and Written Opinion, dated Jun. 4, 2008, PCT/US2007/066027. |
International Search Report and Written Opinion, dated Oct. 15, 2008, from PCT/US08/069154. |
CN Office Action mailed Jul. 31, 2009, in Chinese Application No. 200580034647.5. |
CN Second Office Action mailed Feb. 5, 2010, in Chinese Application No. 200580034647.5. |
CN Third Office Action mailed Aug. 11, 2010, in Chinese Application No. 200580034647.5. |
CN Fourth Office Action mailed Jan. 10, 2011, in Chinese Application No. 200580034647.5. |
CN Office Action mailed Aug. 8, 2008, in Chinese Application No. 200580035946. |
CN Second Office Action mailed Feb. 27, 2009, in Chinese Application No. 200580035946. |
CN Office Action mailed Jul. 18, 2008, in Chinese Application No. 200580034646.0. |
CN Second Office Action mailed Jan. 15, 2010, in Chinese Application No. 200580034646.0. |
CN First Office Action mailed Feb. 12, 2010, in Chinese Application No. 200580034955.8. |
CN Second Office Action mailed Aug. 11, 2010, in Chinese Application No. 200580034955.8. |
CN Third Office Action mailed Dec. 3, 2010, in Chinese Application No. 200580034955.8. |
CN Office Action mailed Apr. 3, 2009, in Chinese Application No. 200680032204. |
EPO Extended Search Report mailed Jul. 16, 2009, in EP Application No. 05810244.3. |
EPO Office Action mailed Oct. 1, 2009, in EP Application No. 05810244.3. |
EPO Extended Search Report mailed Jul. 13, 2009, in EP Application No. 05810800.2. |
EPO Office Action mailed Oct. 19, 2009, in EP Application No. 05810800.2. |
EPO Search Report mailed Mar. 19, 2010, in EP Application No. 08728248.9. |
EPO Office Action mailed Jun. 18, 2010, in EP Application No. 08728248.9. |
U.S. Appl. No. 10/777,886, entitled “End-to-End Congestion Control”, filed on Feb. 11, 2004. |
U.S. Appl. No. 60/621,396, filed Oct. 22, 2004. |
US Office Action mailed Mar. 31, 2008 in related U.S. Appl. No. 11/084,587. |
US Office Action mailed Oct. 28, 2008 in related U.S. Appl. No. 11/084,587. |
US Office Action mailed Apr. 22, 2009 in related U.S. Appl. No. 11/084,587. |
US Office Action mailed Nov. 23, 2009 in related U.S. Appl. No. 11/084,587. |
US Office Action mailed Jun. 24, 2010 in related U.S. Appl. No. 11/084,587. |
US Final Office Action mailed Nov. 26, 2010 in related U.S. Appl. No. 11/084,587. |
US Notice of Allowance mailed Feb. 22, 2011, in related U.S. Appl. No. 11/084,587. |
US Final Office Action mailed Jul. 11, 2008 in related U.S. Appl. No. 11/078,992. |
US Office Action mailed Oct. 23, 2008 in related U.S. Appl. No. 11/078,992. |
US Notice of Allowance mailed Mar. 23, 2009 in related U.S. Appl. No. 11/078,992. |
US Office Action mailed Jul. 3, 2008 in related U.S. Appl. No. 11/400,671. |
US Final Office Action mailed Mar. 17, 2009 in related U.S. Appl. No. 11/400,671. |
US Office Action mailed Jun. 22, 2009 in related U.S. Appl. No. 11/400,671. |
US Office Action mailed Dec. 9, 2009 in related U.S. Appl. No. 11/400,671. |
US Final Office Action mailed Jun. 11, 2010 in related U.S. Appl. No. 11/400,671. |
US Office Action mailed Mar. 17, 2011 in related U.S. Appl. No. 11/400,671. |
US Office Action mailed Apr. 13, 2011 in related U.S. Appl. No. 12/485,337. |
US Office Action mailed Feb. 21, 2008 in related U.S. Appl. No. 11/094,877. |
US Office Action mailed Jul. 28, 2008 in related U.S. Appl. No. 11/094,877. |
US Final Office Action mailed Dec. 10, 2008 in related U.S. Appl. No. 11/094,877. |
US Office Action mailed Apr. 7, 2009 in related U.S. Appl. No. 11/094,877. |
US Office Action mailed Nov. 4, 2009 in related U.S. Appl. No. 11/094,877. |
US Notice of Allowance mailed Apr. 23, 2010 in related U.S. Appl. No. 11/094,877. |
US Notice of Allowance mailed Jun. 28, 2010 in related U.S. Appl. No. 11/094,877. |
US Office Action mailed Jan. 24, 2008 in related U.S. Appl. No. 11/152,991. |
US Final Office Action mailed Sep. 8, 2008 in related U.S. Appl. No. 11/152,991. |
US Office Action mailed Mar. 4, 2009 in related U.S. Appl. No. 11/152,991. |
US Final Office Action mailed Aug. 18, 2009 in related U.S. Appl. No. 11/152,991. |
US Notice of Allowance mailed Dec. 31, 2009 in related U.S. Appl. No. 11/152,991. |
US Notice of Allowance mailed May 17, 2010 in related U.S. Appl. No. 11/152,991. |
US Office Action mailed May 29, 2008 in related U.S. Appl. No. 11/155,388. |
US Final Office Action mailed Sep. 15, 2008 in related U.S. Appl. No. 11/155,388. |
US Notice of Allowance mailed May 29, 2009 in related U.S. Appl. No. 11/155,388. |
US Notice of Allowance mailed Jul. 17, 2009 in related U.S. Appl. No. 11/155,388. |
US Office Action mailed May 14, 2009 in related U.S. Appl. No. 11/248,933. |
US Final Office Action mailed Dec. 28, 2009 in related U.S. Appl. No. 11/248,933. |
US Office Action mailed May 13, 2010 in related U.S. Appl. No. 11/248,933. |
US Notice of Allowance mailed Oct. 8, 2010 in related U.S. Appl. No. 11/248,933. |
US Notice of Allowance mailed Feb. 4, 2011, in related U.S. Appl. No. 11/248,933. |
US Office Action mailed Feb. 18, 2009 in related U.S. Appl. No. 11/490,806. |
US Office Action mailed Nov. 12, 2009 in related U.S. Appl. No. 11/490,806. |
US Final Office Action mailed Jun. 17, 2009 in related U.S. Appl. No. 11/490,806. |
US Final Office Action mailed Mar. 17, 2010 in related U.S. Appl. No. 11/490,806. |
US Non-Final Office Action mailed Jul. 22, 2010 in related U.S. Appl. No. 11/490,806. |
US Final Office Action mailed Dec. 16, 2010 in related U.S. Appl. No. 11/490,806. |
US Office Action mailed Apr. 15, 2009 in related U.S. Appl. No. 11/670,544. |
US Final Office Action mailed Oct. 22, 2009 in related U.S. Appl. No. 11/670,544. |
US Office Action mailed Oct. 19, 2009 in related U.S. Appl. No. 11/825,631. |
US Office Action mailed Apr. 28, 2010 in related U.S. Appl. No. 11/825,631. |
US Office Action mailed Oct. 29, 2010 in related U.S. Appl. No. 11/825,631. |
US Office Action mailed Oct. 19, 2009 in related U.S. Appl. No. 11/842,866. |
US Final Office Action mailed Apr. 2, 2010 in related U.S. Appl. No. 11/842,866. |
US Office Action mailed Oct. 29, 2010 in related U.S. Appl. No. 11/842,866. |
US Notice of Allowance dated May 19, 2011, issued in U.S. Appl. No. 11/084,587. |
US Office Action dated Feb. 15, 2013 issued in U.S. Appl. No. 13/112,824. |
US Notice of Allowanced dated Jun. 17, 2013 issued in U.S. Appl. No. 13/112,824. |
US Final Office Action dated Sep. 16, 2011 issued in U.S. Appl. No. 11/400,671. |
US Office Action dated Dec. 22, 2011 issued in U.S. Appl. No. 11/400,671. |
US Notice of Allowance dated Apr. 4, 2012 issued in U.S. Appl. No. 11/400,671. |
US Notice of Allowance dated Aug. 26, 2011 issued in U.S. Appl. No. 12/485,337. |
US Notice of Allowance dated Dec. 13, 2011 issued in U.S. Appl. No. 12/485,337. |
US Office Action dated Apr. 18, 2013 issued in U.S. Appl. No. 13/444,556. |
US Office Action dated Oct. 23, 2012 issued in U.S. Appl. No. 12/885,339. |
US Notice of Allowance dated May 10, 2013 issued in U.S. Appl. No. 12/885,339. |
US Notice of Allowance dated Nov. 23, 2010 issued in U.S. Appl. No. 11/248,933. |
US Notice of Allowance dated Nov. 18, 2011 issued in U.S. Appl. No. 11/670,544. |
US Notice of Allowance dated May 1, 2012 issued in U.S. Appl. No. 11/670,544. |
US Office Action dated May 26, 2011 issued in U.S. Appl. No. 11/825,631. |
US Notice of Allowance dated Nov. 25, 2011 issued in U.S. Appl. No. 11/825,631. |
US Office Action dated May 27, 2011 issued in U.S. Appl. No. 11/842,866. |
US Notice of Allowance dated Oct. 7, 2011 issued in U.S. Appl. No. 11/842,866. |
US Office Action dated Jun. 20, 2013 issued in U.S. Appl. No. 13/357,443. |
EPO Extended Search Report dated Jun. 1, 2011 issued in EP Application No. 05812799.4. |
India First Examination Report dated May 29, 2013, issued in IN Application No. 1348/KOLNP/2007. |
EPO Office Action dated Apr. 12, 2011, issued in EP Application No. 05810244.3. |
IN Office Action dated Apr. 12, 2013, issued in IN Application No. 1347/KOLN/2007. |
CN Third Office Action dated Jul. 6, 2011, in Chinese Application No. 200580034646.0. |
CN Fourth Office Action dated Dec. 31, 2011, in Chinese Application No. 200580034646.0. |
EPO Extended Search Report dated Nov. 12, 2012 issued in EP 05 851 226.0. |
IN Office Action dated Mar. 22, 2013 issued in IN Application No. 1345/KOLNP/2007. |
EPO Extended Search Report dated Nov. 8, 2012 issued in EP 05 810 293.0. |
EPO Office Action dated May 19, 2011 issued in EP Application No. 05810800.2. |
CN First Office Action dated Apr. 3, 2009 issued in CN 200680032204.7. |
CN Second Office Action dated Jan. 18, 2012 issued in Chinese Application No. 200880003496.0. |
CN First Office Action dated Aug. 11, 2011 issued in Chinese Application No. 200880003496.0. |
EPO Office Action dated Nov. 18, 2011, issued in EP Application No. 08728248.9. |
International Search Report and Written Opinion dated Oct. 15, 2008 issued in PCT/US2008/069154. |
International Preliminary Report on Patentability and Written Opinion dated Jan. 5, 2010 issued in PCT/US2008/069154. |
Number | Date | Country | |
---|---|---|---|
20110273983 A1 | Nov 2011 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 11248933 | Oct 2005 | US |
Child | 13101870 | US |