Patent Grant
9813359
Embodiments described herein relate generally to switch fabrics and more particularly, to sending data through switch fabrics (e.g., Clos networks).
Known switch fabrics can be multi-stage networks that provide connections between multiple input ports and multiple output ports. Thus, through a switch fabric, a first peripheral device operatively coupled to the switch fabric can send data to a second peripheral device operatively coupled to the switch fabric.
A three-stage Clos network, for example, has a middle stage connected between an input stage and an output stage. Each stage includes multiple modules. Each input stage module has multiple input ports and is operatively coupled to each middle stage module. Similarly, each output stage module has multiple output ports and is connected to each middle stage module.
As the data traverses the switch fabric, each stage determines to which subsequent stage to send the data. To make these decisions, a header (e.g., a packet header) can be used. Each stage of known Ethernet switch fabrics, for example, perform layer 2/layer 3 (L2/L3) packet forwarding, lookup and classification. In some known switch fabrics including more than a single stage, such forwarding, lookup and classification functions can significantly increase the end-to-end latency of the switch fabric. Further, adding additional peripheral devices and/or stages to the switch fabric can significantly increase the end-to-end latency of the switch fabric.
Additionally, known Ethernet switch fabrics often do not ensure that data packets sent from a first peripheral device to a second peripheral device traverse the switch fabric using the same path. Accordingly, packet order is not preserved at the output of the switch fabric and the second peripheral device reorders the data packets, causing further latency and increasing buffering requirements.
Thus, a need exists for a switch fabric that has a relatively low end-to-end latency when compared with known switch fabrics. Additionally, a need exists for a switch fabric that preserves packet ordering.
A method of sending data to a switch fabric includes assigning a destination port of an output module to a data packet based on at least one field in a first header of the data packet. A module associated with a first stage of the switch fabric is selected based on at least one field in the first header. A second header is appended to the data packet. The second header includes an identifier associated with the destination port of the output module. The data packet is sent to the module associated with the first stage. The module associated with the first stage is configured to send the data packet to a module associated with a second stage of the switch fabric based on the second header.
In some embodiments, a method of sending data to a switch fabric includes assigning a destination port of an output module (e.g., an edge device) to a data packet (or a data cell) based on at least one field in a first header of the data packet (or data cell). A module associated with a first stage of the switch fabric is selected based on at least one field in the first header. A second header is appended to the data packet (or data cell). The second header includes an identifier associated with the destination port of the output module. The data packet (or data cell) is sent to the module associated with the first stage. The module associated with the first stage is configured to send the data packet (or data cell) to a module associated with a second stage of the switch fabric based on the second header.
In some embodiments, the first header includes a destination Media Access Control (MAC) address, a destination internet protocol (IP) address, a source MAC address, a source IP address and/or a transfer protocol. A portion of the data in the first header can be used as an input to a hash function. The output of the hash function can identify which module associated with the first stage is selected.
In some embodiments, the second header includes a destination identifier, such as, for example, an identifier of a destination edge device, an identifier of a destination port on a destination edge device, and/or the like. Before entering the switch fabric, the destination identifier can be determined using the data in the first header and a lookup table, which can associate the destination MAC address and/or the destination IP address of a destination peripheral device with a destination port to which the destination peripheral device is coupled.
After the second header is appended to the data packet and the data packet is sent into the switch fabric, the modules associated with the switch fabric can use the destination identifier as an input to a hash function to determine to which module associated with the next stage of the switch fabric to send the data packet. Accordingly, the modules within the switch fabric need not use a lookup table to associate the destination MAC address and/or the destination IP address of the destination peripheral device with the destination port to which the destination peripheral device is coupled because the second header contains the result of such an association. Accordingly, the modules within the switch fabric take less time to route the data packet using the second header, than switch fabrics where the first header alone is used for routing within the switch fabric.
Additionally, in some embodiments, using the destination identifier to route the data packet through the switch fabric ensures that data packets sent from a same source peripheral device to the same destination peripheral device at different times will traverse the switch fabric using the same path as long the switch fabric system is operating in the same configuration at the different times (e.g., the hash functions used are the same, the peripheral devices are coupled to the switch fabric in the same manner, etc.). In such embodiments, this ensures that the order that the data packets are received by the destination peripheral device is the same as the order in which the data packets were sent by the source peripheral device.
In some embodiments, a switch fabric system includes multiple edge devices, multiple modules associated with a first stage of the switch fabric system, and multiple modules associated with a second stage of the switch fabric system. A first edge device from the set of edge devices is configured to receive a set of data packets. Each data packet from the set of data packets can include a first header. The first edge device is configured to append a second header to each data packet based on at least one field in the first header. The second header includes an identifier associated with a destination port of a second edge device from the set of edge devices. The first edge device is configured to send each data packet from the set of data packets to a module from the plurality of modules associated with the first stage based on the first header. The set of modules associated with the first stage of the switch fabric system is configured to send each data packet from the set of data packets to a module from the set of modules associated with the second stage based on the second header.
In some embodiments, a processor-readable medium stores code representing instructions configured to cause a processor to assign a destination port of an output module (e.g., an edge device) to a data packet based on an identifier of a destination node operatively coupled to the destination port of the output module through at least a switch fabric. The identifier of the destination node is within a first header of the data packet. A second header is appended to the data packet. The second header includes an identifier associated with the destination port of the output module. A module associated with a first stage of the switch fabric is selected using a result of a first hash function having as inputs a set of fields in the first header. The data packet is sent to the module associated with the first stage. The module associated with the first stage is configured to select a module associated with a second stage of the switch fabric using a result of a second hash function having as an input the identifier associated with the destination port of the output module from the second header.
As used herein, a switch fabric system can be a system that includes a switch fabric and devices coupled to the switch fabric. In some embodiments, for example, a switch fabric system can include multiple input/output modules (e.g., an edge device, an access switch, etc.) operatively coupled to the switch fabric such that the input/output modules can send data to and receive data from the switch fabric. Additionally, in some embodiments, the switch fabric system can include peripheral devices (e.g., servers, storage devices, gateways, workstations, etc.) operatively coupled to the input/output modules such that the peripheral devices can send data to and receive data from the switch fabric via the input/output modules. In such embodiments, for example, a first peripheral device can send data to a second peripheral device via the input/output modules and the switch fabric, as described in further detail herein.
As used herein, a switch fabric can be a network that includes multiple stages of switches that operatively connect one or more input devices (e.g., a first edge device) with one or more output devices (e.g., a second edge device). A switch fabric can be configured to receive a signal from an input device, forward the signal through the multiple stages of switches, and output the signal to an output device. Each switch of the multiple stages of switches routes the signal such that the signal arrives at its destination. Such a switch fabric can be referred to, for example, as a Clos network.
As used herein, a module that is within a switch fabric can be any assembly and/or set of operatively coupled electrical components that defines one or more switches within a stage of a switch fabric. An input/output module (e.g., an edge device, an access switch, etc.), for example, can be any assembly and/or set of operatively coupled electrical components configured to send data to and/or receive data from a switch fabric. In some embodiments, for example, an input/output module can be an access switch or an edge device configured receive data from a server, prepare data to enter into the switch fabric, and send the data to the switch fabric. In some embodiments, a module can include, for example, a memory, a processor, electrical traces, optical connectors, and/or the like.
As used in this specification, the singular forms “a,” “an” and “the” include plural referents unless the context clearly dictates otherwise. Thus, for example, the term “a module” is intended to mean a single module or a combination of modules.
The terms “first stage”, “second stage” and so on refer to portions, modules or nodes within a switch fabric. In some instances, these terms refer to a specific stage within a given switch fabric. For example, a three-stage Clos network includes three consecutive stages from ingress to egress; such a switch fabric has three stages that can be referred to as the “first stage” (the first stage with respect to the ingress to egress direction) through the “third stage” (the third and final stage with respect to the ingress to egress direction). For example,
In some embodiments, each module 112 of the first stage 140 is a switch (e.g., a packet switch, a frame switch and/or a cell switch). The switches are configured to redirect data (e.g., data packets, data cells, etc.) as it flows through the switch fabric 100. In some embodiments, for example, each switch includes multiple input ports operatively coupled to write interfaces on a memory buffer (not shown in
In alternate embodiments, each module of the first stage is a crossbar switch having input bars and output bars. Multiple switches within the crossbar switch connect each input bar with each output bar. When a switch within the crossbar switch is in an “on” position, the input is operatively coupled to the output and data can flow. Alternatively, when a switch within the crossbar switch is in an “off” position, the input is not operatively coupled to the output and data cannot flow. Thus, the switches within the crossbar switch control which input bars are operatively coupled to which output bars.
Each module 112 of the first stage 140 includes a set of input ports 160 configured to receive data (e.g., a signal, a cell of a packet, a data packet, etc.) as it enters the switch fabric 100. In this embodiment, each module 112 of the first stage 140 includes the same number of input ports 160.
Similar to the first stage 140, the second stage 142 of the switch fabric 100 includes modules 114. The modules 114 of the second stage 142 are structurally similar to the modules 112 of the first stage 140. Each module 114 of the second stage 142 is operatively coupled to each module 112 of the first stage 140 by a data path 120. Each data path 120 between a given module 112 of the first stage 140 and a given module 114 of the second stage 142 is configured to facilitate data transfer from the modules 112 of the first stage 140 to the modules 114 of the second stage 142.
The data paths 120 between the modules 112 of the first stage 140 and the modules 114 of the second stage 142 can be constructed in any manner configured to facilitate data transfer from the modules 112 of the first stage 140 to the modules 114 of the second stage 142. In some embodiments, for example, the data paths 120 are optical connectors between the modules. In other embodiments, the data paths are within a midplane. Such a midplane can be similar to that described in U.S. application Ser. No. 12/345,500, filed Dec. 29, 2008, and entitled “System Architecture for a Scalable and Distributed Multi-Stage Switch Fabric,” which is incorporated herein by reference in its entirety. Such a midplane can be used to connect each module of the second stage with each module of the first stage. In still other embodiments, two or more modules are contained within a single chip package and the data paths are electrical traces.
In some embodiments, the switch fabric 100 is a non-blocking Clos network. Thus, the number of modules 114 of the second stage 142 of the switch fabric 100 varies based on the number of input ports 160 of each module 112 of the first stage 140. In a rearrangeably non-blocking Clos network (e.g., a Benes network), the number of modules 114 of the second stage 142 is greater than or equal to the number of input ports 160 of each module 112 of the first stage 140. Thus, if n is the number of input ports 160 of each module 112 of the first stage 140 and m is the number of modules 114 of the second stage 142, m≧n. In some embodiments, for example, each module of the first stage has five input ports. Thus, the second stage has at least five modules. All five modules of the first stage are operatively coupled to all five modules of the second stage by data paths. Said another way, each module of the first stage can send data to any module of the second stage.
The third stage 144 of the switch fabric 100 includes modules 116. The modules 116 of the third stage 144 are structurally similar to the modules 112 of the first stage 140. The number of modules 116 of the third stage 144 is typically equivalent to the number of modules 112 of the first stage 140. Each module 116 of the third stage 144 includes output ports 162 configured to allow data to exit the switch fabric 100. Each module 116 of the third stage 144 includes the same number of output ports 162. Further, the number of output ports 162 of each module 116 of the third stage 144 is typically equivalent to the number of input ports 160 of each module 112 of the first stage 140.
Each module 116 of the third stage 144 is connected to each module 114 of the second stage 142 by a data path 124. The data paths 124 between the modules 114 of the second stage 142 and the modules 116 of the third stage 144 are configured to facilitate data transfer from the modules 114 of the second stage 142 to the modules 116 of the third stage 144.
The data paths 124 between the modules 114 of the second stage 142 and the modules 116 of the third stage 144 can be constructed in any manner configured to facilitate data transfer from the modules 114 of the second stage 142 to the modules 116 of the third stage 144. In some embodiments, for example, the data paths 124 are optical connectors between the modules. In other embodiments, the data paths are within a midplane. Such a midplane can be similar to that described in further detail herein. Such a midplane can be used to connect each module of the second stage with each module of the third stage. In still other embodiments, two or more modules are contained within a single chip package and the data paths are electrical traces.
The switch fabric 230 can be structurally and functionally similar to the switch fabric 100. Accordingly, the switch fabric includes modules F1-FN associated with a first stage 232 of the switch fabric 230, modules G1-GN associated with a second stage 234 of the switch fabric 230, and modules H1-HN associated with a third stage 236 of the switch fabric. Each module F1-FN associated with the first stage 232 is operatively coupled to each module G1-GN associated with the second stage 234 via data paths. Similarly, each module G1-GN associated with the second stage 234 is operatively coupled to each module H1-HN associated with the third stage 236. The data paths between the modules F1-FN associated with the first stage 232 and the modules G1-GN associated with the second stage 234 and/or the data paths between the modules G1-GN associated with the second stage 234 and the modules H1-HN associated with the third stage 236 can be constructed in any manner configured to facilitate data transfer. In some embodiments, for example, the data paths include optical connectors and optical fibers between the modules. In other embodiments, the data paths are within a midplane.
The modules F1-FN associated with a first stage 232 are configured to send data (e.g., data packets, data cells, etc.) to modules G1-GN associated with a second stage 234. As described in further detail herein, in some embodiments, a module F1-FN associated with the first stage 232 is configured to determine to which module G1-GN associated with the second stage 234 to send the data packet based on a header of the data packet (e.g., destination identifier 432 shown and described with respect to
The peripheral devices 270 can be, for example, servers, storage devices, gateways, workstations, and/or the like. The peripheral devices 270 can be operatively coupled to the edge devices 250 using any suitable connection. For example,
The edge devices 250 can be any devices configured to operatively couple peripheral devices 270 to the switch fabric 230. In some embodiments, for example, the edge devices 250 can be access switches, input/output modules, top-of-rack devices and/or the like. Edge devices E1 and E2 are schematically shown as source edge devices and edge device E3 is schematically shown as a destination edge device for illustration purposes only. Structurally, the edge devices 250 (including E1, E2, and E3) can function as source edge devices and destination edge devices. Accordingly, the edge devices 250 can send data to and receive data from the switch fabric 230.
The edge devices 250 can store Media Access Control (MAC) addresses for other edge devices 250 and/or peripheral devices 270 within the switch fabric system 200. For example, an edge device E3 can store the MAC addresses of the peripheral devices S5, S6 coupled to the edge device E3. Using the MAC addresses of the peripheral devices S5, S6, the edge device E3 can properly forward data packets to its destination when a data packet is received. In some embodiments, for example, an edge device 250 can be coupled to 48 peripheral devices 270 each running multiple virtual machines. If, for example, each peripheral device 270 is running 50 virtual machines, the edge device 250 will store 2400 MAC addresses of edge devices 250 to which it is coupled (e.g., source MAC address (SMACs)). In other embodiments, any number of peripheral devices running any number of virtual machines can be coupled to the edge device.
In some embodiments, the edge devices 250 also store multiple destination MAC addresses (DMACs). Such DMACs can be associated with peripheral devices 270 to which the peripheral devices 270 coupled to an edge device 250 can send data. For example, an edge device E1 can store the MAC addresses of the peripheral devices S5, S6 and associate the MAC addresses with a destination port of E3. Accordingly, the MAC address of peripheral device S5 is associated with destination port 252 and the MAC address of S6 is associated with destination port 253. In some embodiments, an edge device 250 can be coupled to 48 peripheral devices 250 each running 2400 virtual machines. If, for example, each of the 2400 virtual machines is sending data to or receiving data from a connection with 25 other virtual machines coupled to another edge device 250, the edge device 250 can store 60000 DMACs. In such an embodiment, each edge device 250 can store a total of 62400 MAC addresses (e.g., 60000 DMACs+2400 SMACs). In other embodiments, each edge device 250 can store any number of DMACs and/or SMACs.
In some embodiments, each edge device 250 includes a lookup table that associates the MAC addresses (e.g., the DMACs and the SMACs) with the port of an edge device 250 to which the peripheral device having the MAC address is coupled. For example, such a lookup table can associate S5 with port 252 and S6 with 253. In such embodiments, the edge device 250 can use the lookup table to determine how to forward the data packet, as described in further detail herein.
The edge devices 250 can be configured to prepare a data packet to enter the switch fabric 230. For example, the edge device 250 can be configured to forward, classify, and/or modify the packet encapsulation of a data packet prior to sending the data packet to the switch fabric 230. As described in further detail herein, in some embodiments, for example, a hash function using data stored within a header of a data packet (e.g., header portion 423 of
Additionally, as described in further detail herein, data within the header of a data packet (e.g., header portion 423 of
In use, for example, a peripheral device S1 can be configured to send a data packet to another peripheral device S6, via path 222 (e.g., via an edge device E1, the switch fabric 230 and an edge device E3).
The peripheral device S1 can send the data packet to the edge device E1. The data packet can be similar to the data packet 420 shown and described in
The edge device E1 receives the data packet 420 and parses the packet header portion 423 of the data packet 420. In some embodiments, the edge device E1 can use the destination MAC address in the packet header portion in conjunction with a lookup table stored at the edge device E1 to determine an identifier of the destination port (e.g., port 253) to which the peripheral device S6 is coupled and/or an identifier of a destination edge device E3 to which the peripheral device S6 is coupled. The lookup table can, for example, correlate the destination MAC address with the identifier of the destination port (e.g., port 253) and/or the identifier of the destination edge device E3. In some embodiments, for example, the identifier can be a port number, an address (e.g., MAC address, IP address, etc.), an internal unique identifier, an identifier of the second peripheral device itself, and/or any other suitable identifier used to identify the destination peripheral device's S6 position within the switch fabric system 200.
A destination identifier portion (e.g., a second header) containing an identifier associated with the destination port (e.g., port 253) and/or an identifier associated with a destination edge device E3 can be appended to the data packet 420. For example,
Using the information contained within the packet header portion 433 of the data packet 430, the edge device E1 can determine to which module F1-FN to send the data packet 430. While shown in
In some embodiments, for example, the edge device E1 can use a hash function using as inputs the destination MAC address, the destination IP address, the source MAC address, the source IP address, and/or the transfer protocol. Based on the inputs, the hash function can generate an identifier associated with a module (e.g., module F1) associated with the first stage 232 of the switch fabric 230. In some embodiments, the identifier generated by the hash function can be associated with the module F1 using a lookup table to determine to which output port of the edge device E1 the module F1 is coupled and/or the like. In other embodiments, the identifier produced from the hash function can be an identifier of an output port of the edge device E1 to which the module F1 is coupled. In still other embodiments, any other method of associating the identifier generated by the hash function with the module F1 can be used. Because the identifier associated with the module F1 is generated based on the information in the packet header portion 433 of the data packet 430, every data packet sent from peripheral S1 to peripheral S6 will be sent to the same module (e.g., module F1) associated with the first stage 232.
After the module F1 associated with the first stage 232 receives the data packet 430, it parses the destination identifier portion 432 of the data packet 430. Using the destination identifier within the destination identifier portion 432, the module F1 can determine to which module G2 associated with the second stage 234 of the switch fabric 230 to send the data packet 430. In some embodiments, for example, the module F1 can use a hash function using as an input the destination identifier 432. Based on the destination identifier 432, the hash function can generate an identifier associated with a module (e.g., module G2) associated with the second stage 234, and send the data packet 430 accordingly. In some embodiments, the identifier generated by the hash function can be associated with the module G1 using a lookup table to determine to which output port of the module F1 the module G1 is coupled and/or the like. In other embodiments, the identifier produced from the hash function can be an identifier of an output port of the module F1 to which the module G1 is coupled. In still other embodiments, any other method of associating the identifier generated by the hash function with the module G1 can be used. Because the result of the hash function is based on the destination identifier 432, all data packets 430 within the module F1 being sent to the peripheral device S6 will be sent by F1 to the same module G2 associated with the second stage 234.
The amount of time the module F1 takes to determine to which module to send the data packet 430 can be decreased by using a destination identifier portion 432 of the data packet 430 instead of the packet header portion 433. In some embodiments, for example, the destination identifier portion 433 can be smaller (e.g., fewer bytes of memory) and contain fewer fields than the packet header portion 433. Thus, parsing the destination identifier portion 432 can be faster than parsing the packet header portion 433. Additionally, using a hash function allows the module F1 to quickly determine to which module to send the data packet 430. Such a hash function can be easily implemented and allows for a quick identification of the appropriate module based on the destination identifier portion 432.
Additionally, because a destination MAC address is associated with a destination identifier (e.g., a destination port, a destination edge device E3, etc.) at the edge device E1, the module F1 can forward the data packet 430 without associating a destination MAC address with a destination identifier. This reduces the amount of time used by the module F1 when determining to which module G1-GN to send the data packet 430. This also reduces the amount of memory used by the module F1 because the module F1 need not store associations between a destination MAC address and destination identifiers. Further, because the association of a destination MAC address with a destination identifier is performed at the edge device E1 and the result stored in the destination identifier portion 432, the modules within the switch fabric 230 need not perform such an association. Moreover, the module F1 can forward the data packet 430 without performing standard layer 2/layer 3 (L2/L3) forwarding, lookup and classification functions (commonly used in Ethernet switch fabrics).
After the module G2 associated with the second stage 234 receives the data packet 430, it parses the destination identifier portion 432 of the data packet 430, similar to the module F1. Using the destination identifier within the destination identifier portion 432, the module G2 can determine to which module H1-HN associated with the third stage 236 of the switch fabric 230 to send the data packet 430. In some embodiments, for example, the module G2 can use a hash function using as an input the destination identifier. Based on the destination identifier, the hash function can generate an identifier associated with a module (e.g., module H2) associated with the third stage 236, and send the data packet 430 accordingly. Because the result of the hash function is based on the destination identifier, all data packets 430 within the module G2 being sent to the peripheral device S6 will be sent by G2 to the same module H2 associated with the second stage 234.
Similarly, after the module H2 associated with the third stage 236 receives the data packet 430, it parses the destination identifier portion 432 of the data packet 430, similar to the module F1. Using the destination identifier within the destination identifier portion 432, the module H2 can determine to which edge device E3 to send the data packet 430. In some embodiments, for example, the module H2 can use a hash function using as an input the destination identifier. Based on the destination identifier, the hash function can generate an identifier associated with an edge device E3, and send the data packet 430 accordingly. Because the result of the hash function is based on the destination identifier, all data packets 430 within the module H2 being sent to the peripheral device S6 will be sent by H2 to the same edge device E3.
After the edge device E3 receives the data packet 430, the edge device 430 can determine to which peripheral device S6 to send the data packet 430 and send the data packet 430 accordingly. In some embodiments, the edge device E3 can parse the destination identifier portion 432 of the data packet 430. If the destination identifier portion 432 includes an identifier of a specific port 253, the edge device E3 can send the data packet to the peripheral device S6 operatively coupled to the port 253. In other embodiments, the edge device E3 can parse the packet header portion 433. Using the stored SMACs and the destination MAC address in the packet header portion 433, the edge device E3 can determine to which port 253 the destination peripheral device S6 is coupled and send the data packet accordingly. In some embodiments, prior to sending the data packet 430 to the destination peripheral device S6, the destination identifier portion 432 is removed from the data packet 430. Accordingly, in such embodiments, the destination peripheral device S6 receives a data packet similar to the data packet 420 of
As discussed above, because the routing decision at the edge device E1 is based on a hash function using as inputs a destination MAC address, a destination IP address, a source MAC address, a source IP address and/or a transfer protocol (e.g., the data within a packet header portion of a data packet), each data packet sent from a first peripheral device (e.g., S1) to a second peripheral device (e.g., S6) is sent to the same module F1 associated with the first stage 232. Additionally, as discussed above, because the routing decisions in the switch fabric 230 (e.g., at modules F1, G2, and H2) are based on the a destination identifier portion 432 appended to the data packet 430 at the edge device E1, each data packet 430 sent from the first peripheral device (e.g., S1) to the second peripheral device (e.g., S6) traverses the same path 222 through the switch fabric 230 (e.g., from F1 to G2, from G2 to H2, and from H2 to E3). This ensures that each data packet sent from the first peripheral device (e.g., S1) to the second peripheral device (e.g., S6) traverses the switch fabric system 200 using the same path 222. Accordingly, the order in which packets are sent from the first peripheral device (e.g., S1) to the second peripheral device (e.g., S6) is preserved. Said another way, if a second data packet is sent from peripheral device S1 to peripheral device S6 after a first data packet is sent from peripheral device S1 to peripheral device S6, the first data packet will arrive at peripheral device S6 prior to the second data packet arriving at peripheral device S6.
The edge devices 550 include multiple cable connector ports 582 each configured to be coupled to an end portion of a cable 580. Through the cables 580, each edge device 550 can be operatively coupled to the switch fabric chassis 530. Each edge device 550 can function as a source edge device and a destination edge device. Accordingly, each edge device 550 can send data to and receive data from the switch fabric chassis 530.
The cables 580 can be constructed of any material suitable to transfer data between the edge devices 550 and the switch fabric chassis 530. In some embodiments, for example, each cable 580 is constructed of multiple optical fibers. In such an embodiment, each cable 580 can have, for example, twelve transmit and twelve receive fibers. The twelve transmit fibers of each cable 580 can include eight fibers for transmitting data, one fiber for transmitting a control signal, and three fibers for expanding the data capacity and/or for redundancy. Similarly, the twelve receive fibers of each cable 580 have eight fibers for receiving data, one fiber for receiving a control signal, and three fibers for expanding the data capacity and/or for redundancy. In other embodiments, any number of fibers can be contained within each cable. In some embodiments, for example, the cables 580 can be 40 gigabit (40 G) cables. The transmit and receive designations of the fibers are from the perspective of the edge devices 550. The designations are opposite if viewed from the perspective of the switch fabric chassis 530.
While shown in
The switch fabric chassis 530 includes multiple interface cards 560 (only a single interface card 560 from a set of multiple interface cards 560 is shown in
The interface card 560 includes multiple cable connector ports 584 and multiple 1st/3rd stage module systems 562. The cable connector ports 584 can be similar to the cable connector ports 582. Accordingly, each cable connector port 584 can be configured to receive an end of a cable 580. Via a cable connector port 582, a cable 580 and a cable connector port 584, an edge device 550 can be operatively coupled to an interface card 560.
Each 1st/3rd stage module system includes a module associated with a first stage of the switch fabric system 500 and a module associated with a third stage of the switch fabric system 500. The module associated with the first stage and the module associated with the third stage can be similar to the modules 232 and the modules 234, respectively, shown and described above with respect to
The 1st/3rd stage module systems 562 can be application-specific integrated circuits (ASICs) or chip packages having multiple ASICs. The 1st/3rd stage module systems 562 can be instances of the same ASIC or chip package. Said another way, the ASIC or chip package of each 1st/3rd stage module system 562 can be substantially similar (i.e., the same kind or type) to the ASIC or chip package of other 1st/3rd stage module systems 562. Thus, manufacturing costs can be decreased because multiple instances of a single ASIC or chip package can be produced.
The interface card 570 includes multiple 2nd stage module systems 572. Each 2nd stage module system 572 includes a module associated with a second stage of the switch fabric system 500. The module associated with the second stage can be similar to the modules 236, respectively, shown and described above with respect to
Similar to the 1st/3rd stage module systems 562, the 2nd stage module systems 572 can be application-specific integrated circuits (ASICs) or chip packages having multiple ASICs. The 2nd stage module systems 572 can be instances of the same ASIC or chip package. Said another way, the ASIC or chip package of each 2nd stage module system 572 can be substantially similar (i.e., the same kind or type) to the ASIC or chip package of other 2nd stage module systems 562. Thus, manufacturing costs can be decreased because multiple instances of a single ASIC or chip package can be produced.
In some embodiments, the switch fabric system 500 includes eight interface cards 560 each operatively coupled to eight interface cards 570 through the midplane 590. In such embodiments, each interface card 560 can include sixteen cable connector ports 584. As such, the switch fabric chassis 530 can include 128 cable connector ports 584 to which edge devices 550 can be coupled (8 interface cards (560)×16 cable connector ports (584) per interface card (560)=128 total cable connector ports 584). Accordingly, in such embodiments, 128 edge devices 550 can be coupled to the switch fabric chassis 530.
In use, the switch fabric system 500 functions similar to the switch fabric system 200, shown and described above. As such a first peripheral device (not shown in
The first peripheral device can send the data packet to the first edge device 550. The data packet can be similar to the data packet 420 shown and described in
The edge device 550 receives the data packet and parses the packet header portion of the data packet. In some embodiments, for example, the edge device 550 can use the destination MAC address in the packet header portion in conjunction with a lookup table stored at the edge device 550 to determine an identifier of the destination port to which the second peripheral device is coupled and/or an identifier of a destination edge device 550 to which the second peripheral device is coupled. The lookup table can, for example, correlate the destination MAC address with the identifier of the destination port and/or the identifier of the destination edge device 550. In some embodiments, for example, the identifier can be a port number, an address (e.g., MAC address, IP address, etc.), an internal unique identifier, an identifier of the second peripheral device itself, and/or any other suitable identifier used to identify the destination peripheral device's position within the switch fabric system 500.
A destination identifier portion (e.g., a second header) containing the identifier associated with the destination port and/or the identifier associated with a destination edge device 550 can be appended to the data packet 420. For example,
Using the information contained within the packet header portion 433 of the data packet 430, the edge device 550 can determine to which 1st/3rd stage module system 562 to send the data packet 430. While shown in
In some embodiments, for example, the edge device 550 can use a hash function using as inputs the destination MAC address, the destination IP address, the source MAC address, the source IP address, and/or the transfer protocol. Based on the inputs, the hash function can generate an identifier associated with a first stage module within a 1st/3rd stage module system 562. Because the identifier associated with the module is generated based on the information in the header portion 433 of the data packet 430, every data packet sent from the first peripheral device to the second peripheral device will be sent to the same 1st/3rd stage module system 562.
Additionally, as described above, in some embodiments, each edge device 550 can be coupled to more than one switch fabric. In such embodiments, the hash function at the edge device can be used by the edge device to determine to which switch fabric to send the data. For example, depending on the result of the hash function, the edge device 550 can send the data via a cable 580 to the first switch fabric chassis 530 or can send the data via a cable 581 to the second switch fabric chassis (not shown in
After the module associated with the first stage within the 1st/3rd stage module system 562 receives the data packet 430, it parses the destination identifier portion 432 of the data packet 430. Using the destination identifier within the destination identifier portion 432, the first stage module can determine to which second stage module system 572 to send the data packet 430. In some embodiments, for example, the first stage module can use a hash function using as an input the destination identifier. Based on the destination identifier, the hash function can generate an identifier associated with a second stage module within a second stage module system 572, and send the data packet 430 accordingly. Because the result of the hash function is based on the destination identifier, all data packets 430 within the same first stage module being sent to the same peripheral device will be sent by the first stage module to the same second stage module.
The amount of time a first stage module takes to determine to which second stage module to send the data packet 430 can be decreased by using a destination identifier portion 432 of the data packet 430 instead of the packet header portion 433. In some embodiments, for example, the destination identifier portion 433 can be smaller (e.g., fewer bytes of memory) and contain fewer fields than the packet header portion 433. Thus, parsing the destination identifier portion 432 can be faster than parsing the packet header portion 433. Additionally, using a hash function allows a first stage module to quickly determine to which second stage module to send the data packet 430. Such a hash function can be easily implemented and allows for a quick identification of the appropriate second stage module based on the destination identifier portion 432.
Additionally, because a destination MAC address is associated with a destination identifier (e.g., a destination port, a destination edge device 550, etc.) at the source edge device 550, the first stage module can forward the data packet 430 without associating a destination MAC address with a destination identifier. This reduces the amount of time used by the first stage module when determining to which second stage module to send the data packet 430. This also reduces the amount of memory used by the first stage module as the first stage module need not store associations between a destination MAC address and destination identifiers (e.g., the first stage module need not store a lookup table). Further, because the association of a destination MAC address with a destination identifier is performed at a source edge device 550 and the result stored in the destination identifier portion 432, the modules within the switch fabric chassis 530 need not perform such an association. Moreover, the first stage module can forward the data packet 430 without performing standard L2/L3 forwarding, lookup and classification functions (commonly used in Ethernet switch fabrics).
As shown in
Similar to the first stage module within the 1st/3rd stage module system 562, the second stage module within the 2nd stage module system 572 can parse the destination identifier portion 432 of the data packet 430 and use the destination identifier portion 432 (e.g., as an input to a hash function) to determine to which third stage module within a 1st/3rd stage module system 562 to send the data packet 430. The data packet 430 can be sent to the 1st/3rd stage module system 562 via the midplane 590.
Further, the third stage module within the 1st/3rd stage module system 562 can parse the destination identifier portion 432 of the data packet 430 and use the destination identifier portion 432 (e.g., as an input to a hash function) to determine to which edge device 550 to send the data packet. The data packet 430 can be sent to the edge device 550 via a cable 580.
As discussed above, because the routing decision at the source edge device 550 is based on a hash function using as inputs a destination MAC address, a destination IP address, a source MAC address, a source IP address and/or a transfer protocol, each data packet sent from a first peripheral device to a second peripheral device is sent to the same first stage module within the same 1st/3rd stage module system 562. Additionally, as discussed above, because the routing decisions in the switch fabric chassis 530 (e.g., at the 1st/3rd stage module systems 562 and the 2nd stage module systems 572) are based on the a destination identifier portion 432 appended to the data packet 430 at the source edge device 550, each data packet 430 sent from the first peripheral device to the second peripheral device traverses the same path through the switch fabric chassis 530 (e.g., passes through the same module systems 562, 572). This ensures that each data packet sent from the first peripheral device to the second peripheral device traverses the switch fabric system 500 using the same path. Accordingly, the order in which packets are sent from the first peripheral device to the second peripheral device is preserved. Said another way, if a second data packet is sent from the first peripheral device to the second peripheral device after a first data packet is sent from the first peripheral device to the second peripheral device, the first data packet will arrive at second peripheral device prior to the second data packet arriving at the second peripheral device.
In some embodiments, a control module (not shown in
In some embodiments, the first header can be similar to the packet header portion 423 shown and described with respect to
A module associated with a first stage of the switch fabric is selected based on at least one field in the first header, at 604. The module associated with the first stage can be selected using a hash function. The hash function can use as inputs at least one field in the first header. Because the module associated with the first stage is selected based on the fields in the first header, the same module associated with the first stage will be selected for other data packets having a similar first header (e.g., a second data packet's source and destination are the same as the first data packet's source and destination).
A second header is appended to the data packet, at 606. The second header includes an identifier associated with the destination port of the output module. The second header can be similar to the destination identifier portion 432 shown and described with respect to
The data packet is sent to the module associated with the first stage, at 608. The module associated with the first stage is configured to send the data packet to a module associated with a second stage of the switch fabric based on the second header. In some embodiments, the module associated with the first stage can use the destination identifier as an input to a hash function to determine to which module associated with the next stage of the switch fabric to send the data packet. Accordingly, the module associated with the first stage need not use a lookup table to associate the destination MAC address and/or the destination IP address of the destination peripheral device with the destination port to which the destination peripheral device is coupled because the second header contains the result of such an association.
Additionally, using the destination identifier to route the data packet through the switch fabric ensures that data packets sent from a same source peripheral device to the same destination peripheral device will traverse the switch fabric using the same path. This ensures that the order the data packets are received by the destination peripheral device is the same as the order in which the data packets were sent by the source peripheral device.
While various embodiments have been described above, it should be understood that they have been presented by way of example only, and not limitation. Where methods described above indicate certain events occurring in certain order, the ordering of certain events may be modified. Additionally, certain of the events may be performed concurrently in a parallel process when possible, as well as performed sequentially as described above.
While shown and described above as using hash functions to determine how to route data through a switch fabric, in other embodiments, any other suitable function can be used to route data through the switch fabric. Some embodiments can include, for example, a mapping function such as a lookup table and/or the like used to route data through the switch fabric.
Further, any suitable type of hash function can be used to route data through the switch fabric. Some embodiments can include, for example, cyclic redundancy check hash functions, checksum hash functions, secure hash algorithms (SHA) such as SHA1, SHA256, etc., message digest (MD) algorithms such as MD2, MD4, MD5, etc., Pearson hash functions, Fowler-Noll-Vo hash functions, Bloom filters and/or the like.
While shown and described as having three-stages, the switch fabric systems shown and described herein can be upgraded to switch fabrics having any number of stages greater than three stages without significantly increasing the end-to-end latency of the switch fabric system. For example, the switch fabric system 500 can be upgraded to a five-stage switch fabric system. Because the modules within the switch fabric do not parse the packet header of the data packet (e.g., packet header 433 of data packet 430 shown in
Some embodiments described herein relate to a computer storage product with a computer- or processor-readable medium (also can be referred to as a processor-readable medium) having instructions or computer code thereon for performing various computer-implemented operations. The media and computer code (also can be referred to as code) may be those designed and constructed for the specific purpose or purposes. Examples of computer-readable media include, but are not limited to: magnetic storage media such as hard disks, floppy disks, and magnetic tape; optical storage media such as Compact Disc/Digital Video Discs (CD/DVDs), Compact Disc-Read Only Memories (CD-ROMs), and holographic devices; magneto-optical storage media such as optical disks; carrier wave signal processing modules; and hardware devices that are specially configured to store and execute program code, such as general purpose microprocessors, microcontrollers, Application-Specific Integrated Circuits (ASICs), Programmable Logic Devices (PLDs), and Read-Only Memory (ROM) and Random-Access Memory (RAM) devices.
Examples of computer code include, but are not limited to, micro-code or micro-instructions, machine instructions, such as produced by a compiler, code used to produce a web service, and files containing higher-level instructions that are executed by a computer using an interpreter. For example, embodiments may be implemented using Java, C++, or other programming languages (e.g., object-oriented programming languages) and development tools. Additional examples of computer code include, but are not limited to, control signals, encrypted code, and compressed code.
Although various embodiments have been described as having particular features and/or combinations of components, other embodiments are possible having a combination of any features and/or components from any of embodiments where appropriate. For example, while shown above as being coupled to a single switch fabric, the edge devices shown and described herein can be coupled to any number of switch fabrics and/or modules associated with a first stage of a switch fabric. In some embodiments, for example, the edge devices are coupled to two switch fabrics. In other embodiments, the edge devices are coupled to more than two switch fabrics.
This application is a continuation application of U.S. patent application Ser. No. 14/610,143, filed Jan. 30, 2015, which is a continuation of U.S. patent application Ser. No. 12/607,162, now U.S. Pat. No. 8,953,603, filed Oct. 28, 2009, each entitled “Methods and Apparatus Related to a Distributed Switch Fabric,” the disclosure of each of which is hereby incorporated by reference in its entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4215402 | Mitchell et al. | Jul 1980 | A |
| 4400627 | Zola | Aug 1983 | A |
| 4813038 | Lee | Mar 1989 | A |
| 4991171 | Teraslinna | Feb 1991 | A |
| 5130984 | Cisneros | Jul 1992 | A |
| 5138615 | Lamport et al. | Aug 1992 | A |
| 5193087 | Lichtash et al. | Mar 1993 | A |
| 5801641 | Yang et al. | Sep 1998 | A |
| 5926473 | Gridley | Jul 1999 | A |
| 5987027 | Park et al. | Nov 1999 | A |
| 5987028 | Yang et al. | Nov 1999 | A |
| 5991295 | Tout et al. | Nov 1999 | A |
| 6011779 | Wills | Jan 2000 | A |
| 6073089 | Baker et al. | Jun 2000 | A |
| 6075773 | Clark et al. | Jun 2000 | A |
| 6157955 | Narad et al. | Dec 2000 | A |
| 6212184 | Venkatachary et al. | Apr 2001 | B1 |
| 6226629 | Cossock | May 2001 | B1 |
| 6266705 | Ullum et al. | Jul 2001 | B1 |
| 6452926 | Wiklund | Sep 2002 | B1 |
| 6457058 | Ullum et al. | Sep 2002 | B1 |
| 6522627 | Mauger | Feb 2003 | B1 |
| 6618397 | Huang | Sep 2003 | B1 |
| 6633548 | Bachmutsky et al. | Oct 2003 | B2 |
| 6654373 | Maher, III et al. | Nov 2003 | B1 |
| 6657962 | Barri et al. | Dec 2003 | B1 |
| 6658482 | Chen et al. | Dec 2003 | B1 |
| 6717956 | Fan et al. | Apr 2004 | B1 |
| 6721316 | Epps et al. | Apr 2004 | B1 |
| 6731644 | Epps et al. | May 2004 | B1 |
| 6735670 | Bronstein | May 2004 | B1 |
| 6754662 | Li | Jun 2004 | B1 |
| 6775230 | Watanabe et al. | Aug 2004 | B1 |
| 6778546 | Epps et al. | Aug 2004 | B1 |
| 6778984 | Lu et al. | Aug 2004 | B1 |
| 6807172 | Levenson et al. | Oct 2004 | B1 |
| 6813243 | Epps et al. | Nov 2004 | B1 |
| 6816487 | Roberts et al. | Nov 2004 | B1 |
| 6889225 | Cheng et al. | May 2005 | B2 |
| 6920135 | Lea | Jul 2005 | B1 |
| 6925085 | Krishna et al. | Aug 2005 | B1 |
| 6940862 | Goudreau | Sep 2005 | B2 |
| 6941506 | Spencer | Sep 2005 | B2 |
| 6970902 | Moon | Nov 2005 | B1 |
| 6973032 | Casley et al. | Dec 2005 | B1 |
| 6977930 | Epps et al. | Dec 2005 | B1 |
| 6985486 | Agrawal | Jan 2006 | B1 |
| 7027412 | Miyamoto et al. | Apr 2006 | B2 |
| 7046687 | Brown | May 2006 | B1 |
| 7069413 | Agesen et al. | Jun 2006 | B1 |
| 7075934 | Chiussi et al. | Jul 2006 | B2 |
| 7089240 | Basso et al. | Aug 2006 | B2 |
| 7102999 | Sindhu | Sep 2006 | B1 |
| 7113505 | Williams | Sep 2006 | B2 |
| 7190696 | Manur et al. | Mar 2007 | B1 |
| 7193997 | Van Lunteren et al. | Mar 2007 | B2 |
| 7221676 | Green et al. | May 2007 | B2 |
| 7230947 | Huber et al. | Jun 2007 | B1 |
| 7245582 | Roberts et al. | Jul 2007 | B1 |
| 7313135 | Wyatt | Dec 2007 | B2 |
| 7327680 | Kloth | Feb 2008 | B1 |
| 7356033 | Basu et al. | Apr 2008 | B2 |
| 7369561 | Wybenga et al. | May 2008 | B2 |
| 7373345 | Carpentier et al. | May 2008 | B2 |
| 7382876 | Lauter et al. | Jun 2008 | B2 |
| 7391736 | Na et al. | Jun 2008 | B2 |
| 7394809 | Kumar et al. | Jul 2008 | B2 |
| 7406038 | Oelke et al. | Jul 2008 | B1 |
| 7409487 | Chen et al. | Aug 2008 | B1 |
| 7415034 | Muller et al. | Aug 2008 | B2 |
| 7418505 | Lim et al. | Aug 2008 | B2 |
| 7428219 | Khosravi | Sep 2008 | B2 |
| 7430164 | Bare | Sep 2008 | B2 |
| 7441268 | Remedios | Oct 2008 | B2 |
| 7580415 | Hudson et al. | Aug 2009 | B2 |
| 7602787 | Cheriton | Oct 2009 | B2 |
| 7630368 | Tripathi et al. | Dec 2009 | B2 |
| 7689747 | Vega et al. | Mar 2010 | B2 |
| 7738457 | Nordmark et al. | Jun 2010 | B2 |
| 7757059 | Ofer et al. | Jul 2010 | B1 |
| 7788411 | Belgaied et al. | Aug 2010 | B2 |
| 7836212 | Tripathi et al. | Nov 2010 | B2 |
| 7978690 | Abel | Jul 2011 | B2 |
| 8045546 | Bao et al. | Oct 2011 | B1 |
| 8050256 | Bao et al. | Nov 2011 | B1 |
| 8300650 | Rojas-Cessa | Oct 2012 | B2 |
| 8953603 | Aybay | Feb 2015 | B2 |
| 9536885 | Chang | Jan 2017 | B2 |
| 20020064170 | Siu et al. | May 2002 | A1 |
| 20020118644 | Moir | Aug 2002 | A1 |
| 20020118692 | Oberman et al. | Aug 2002 | A1 |
| 20020138648 | Liu | Sep 2002 | A1 |
| 20020152209 | Merugu et al. | Oct 2002 | A1 |
| 20020159449 | Carson | Oct 2002 | A1 |
| 20020191605 | Lunteren et al. | Dec 2002 | A1 |
| 20020194294 | Blumenau et al. | Dec 2002 | A1 |
| 20030026287 | Mullendore et al. | Feb 2003 | A1 |
| 20030059045 | Ruehle | Mar 2003 | A1 |
| 20030065878 | Krishna et al. | Apr 2003 | A1 |
| 20030142668 | Wyatt | Jul 2003 | A1 |
| 20040151188 | Maveli et al. | Aug 2004 | A1 |
| 20040177157 | Mistry et al. | Sep 2004 | A1 |
| 20040205253 | Arndt et al. | Oct 2004 | A1 |
| 20040254909 | Testa | Dec 2004 | A1 |
| 20050058128 | Carson | Mar 2005 | A1 |
| 20050083935 | Kounavis et al. | Apr 2005 | A1 |
| 20050111460 | Sahita | May 2005 | A1 |
| 20050138308 | Morishita et al. | Jun 2005 | A1 |
| 20050177552 | Bass et al. | Aug 2005 | A1 |
| 20050192969 | Haga et al. | Sep 2005 | A1 |
| 20050198303 | Knauerhase et al. | Sep 2005 | A1 |
| 20050226235 | Kumar et al. | Oct 2005 | A1 |
| 20060050690 | Epps et al. | Mar 2006 | A1 |
| 20060083247 | Mehta | Apr 2006 | A1 |
| 20060221956 | Narayan et al. | Oct 2006 | A1 |
| 20060221967 | Narayan et al. | Oct 2006 | A1 |
| 20060259952 | Lok | Nov 2006 | A1 |
| 20070005685 | Chau et al. | Jan 2007 | A1 |
| 20070008949 | Balandin | Jan 2007 | A1 |
| 20070008962 | Basu et al. | Jan 2007 | A1 |
| 20070014288 | Lim et al. | Jan 2007 | A1 |
| 20070071233 | Zak | Mar 2007 | A1 |
| 20070078988 | Miloushev et al. | Apr 2007 | A1 |
| 20070079307 | Dhawan et al. | Apr 2007 | A1 |
| 20070098408 | Handelman | May 2007 | A1 |
| 20070140235 | Aysan et al. | Jun 2007 | A1 |
| 20070204265 | Oshins | Aug 2007 | A1 |
| 20070211716 | Oz et al. | Sep 2007 | A1 |
| 20070219911 | Abe | Sep 2007 | A1 |
| 20070234005 | Erlingsson et al. | Oct 2007 | A1 |
| 20070244997 | Tindal | Oct 2007 | A1 |
| 20070280243 | Wray et al. | Dec 2007 | A1 |
| 20070280253 | Rooholamini et al. | Dec 2007 | A1 |
| 20070283186 | Madnani et al. | Dec 2007 | A1 |
| 20070297428 | Bose et al. | Dec 2007 | A1 |
| 20080002663 | Tripathi et al. | Jan 2008 | A1 |
| 20080005344 | Ford et al. | Jan 2008 | A1 |
| 20080019365 | Tripathi et al. | Jan 2008 | A1 |
| 20080043756 | Droux et al. | Feb 2008 | A1 |
| 20080043765 | Belgaied et al. | Feb 2008 | A1 |
| 20080046610 | Tripathi et al. | Feb 2008 | A1 |
| 20080046735 | Nedeltchev et al. | Feb 2008 | A1 |
| 20080080548 | Mullendore et al. | Apr 2008 | A1 |
| 20080082977 | Araujo et al. | Apr 2008 | A1 |
| 20080095361 | Wifvesson et al. | Apr 2008 | A1 |
| 20080098392 | Wipfel et al. | Apr 2008 | A1 |
| 20080117909 | Johnson | May 2008 | A1 |
| 20080130517 | Lee et al. | Jun 2008 | A1 |
| 20080148341 | Ferguson | Jun 2008 | A1 |
| 20080155223 | Hiltgen et al. | Jun 2008 | A1 |
| 20080177812 | Brandle | Jul 2008 | A1 |
| 20080186875 | Kitani | Aug 2008 | A1 |
| 20080192648 | Galles | Aug 2008 | A1 |
| 20080205377 | Chao et al. | Aug 2008 | A1 |
| 20080212592 | Wybenga et al. | Sep 2008 | A1 |
| 20080219184 | Fowler et al. | Sep 2008 | A1 |
| 20080225853 | Melman et al. | Sep 2008 | A1 |
| 20080228781 | Chen et al. | Sep 2008 | A1 |
| 20080240104 | Villait et al. | Oct 2008 | A1 |
| 20080240122 | Richardson et al. | Oct 2008 | A1 |
| 20080244579 | Muller | Oct 2008 | A1 |
| 20080259934 | Matthews et al. | Oct 2008 | A1 |
| 20080270564 | Rangegowda et al. | Oct 2008 | A1 |
| 20080273465 | Gusat | Nov 2008 | A1 |
| 20090013062 | Blatherwick et al. | Jan 2009 | A1 |
| 20090025007 | Hara et al. | Jan 2009 | A1 |
| 20090037680 | Colbert et al. | Feb 2009 | A1 |
| 20090052461 | Brown et al. | Feb 2009 | A1 |
| 20090083445 | Ganga | Mar 2009 | A1 |
| 20090106409 | Murata | Apr 2009 | A1 |
| 20090106766 | Masuda | Apr 2009 | A1 |
| 20090109479 | Kato | Apr 2009 | A1 |
| 20090135816 | Nandagopal et al. | May 2009 | A1 |
| 20090150529 | Tripathi | Jun 2009 | A1 |
| 20090157701 | Lahiri et al. | Jun 2009 | A1 |
| 20090190598 | Christensen et al. | Jul 2009 | A1 |
| 20090196297 | Jabr | Aug 2009 | A1 |
| 20090198761 | Nanda et al. | Aug 2009 | A1 |
| 20090240790 | Utsunomiya et al. | Sep 2009 | A1 |
| 20090274044 | Goose et al. | Nov 2009 | A1 |
| 20090276772 | Garrett et al. | Nov 2009 | A1 |
| 20090276774 | Kinoshita | Nov 2009 | A1 |
| 20090292858 | Lambeth et al. | Nov 2009 | A1 |
| 20090307597 | Bakman | Dec 2009 | A1 |
| 20090313620 | Sedukhin et al. | Dec 2009 | A1 |
| 20100027420 | Smith | Feb 2010 | A1 |
| 20100042708 | Stamler et al. | Feb 2010 | A1 |
| 20100042719 | Kinoshita | Feb 2010 | A1 |
| 20100043068 | Varadhan et al. | Feb 2010 | A1 |
| 20100050172 | Ferris | Feb 2010 | A1 |
| 20100054129 | Kuik et al. | Mar 2010 | A1 |
| 20100077158 | Asano | Mar 2010 | A1 |
| 20100091961 | Jones et al. | Apr 2010 | A1 |
| 20100128605 | Chavan et al. | May 2010 | A1 |
| 20100131636 | Suri et al. | May 2010 | A1 |
| 20100165876 | Shukla et al. | Jul 2010 | A1 |
| 20100165877 | Shukla et al. | Jul 2010 | A1 |
| 20100169467 | Shukla et al. | Jul 2010 | A1 |
| 20110019550 | Bryers et al. | Jan 2011 | A1 |
| 20110103259 | Aybay et al. | May 2011 | A1 |
| 20120170575 | Mehra | Jul 2012 | A1 |
| Number | Date | Country |
|---|---|---|
| 101388847 | Mar 2009 | CN |
| 0 809 380 | Nov 1997 | EP |
| 2 361 139 | Oct 2001 | GB |
| Entry |
|---|
| Cisco: “Cisco VN-Link: Virtualization-Aware Networking,” Cisco Public Information, Cisco Systems, [Online] Mar. 1, 2009 (pp. 1-10), XP002593596, Retrieved from the Internet: <URL: http://www.cisco.com/en/US/solutions/collateral/ns340/ns517/ns224/ns892/ns894/white—paper—c11-525307.pdf> [retrieved on Jul. 26, 2010]. |
| U.S. Appl. No. 12/346,608, filed Dec. 30, 2008, entitled “Methods and Apparatus Related to Data Transmissions Between Virtual Resources Via a Network Device” (80 pgs). |
| Office Action dated Dec. 23, 2010 for U.S. Appl. No. 12/346,623 (19 pages). |
| Office Action dated Jun. 16, 2011 for U.S. Appl. No. 12/346,623 (13 pages). |
| Office Action dated Sep. 17, 2010 for U.S. Appl. No. 12/346,608 (28 pages). |
| Office Action dated Apr. 14, 2011 for U.S. Appl. No. 12/346,608 (24 pages). |
| Final Office Action dated Jul. 26, 2011 for U.S. Appl. No. 12/346,608 (30 pages). |
| Office Action dated Jan. 17, 2012 for U.S. Appl. No. 12/346,608 (29 pages). |
| U.S. Appl. No. 12/346,612, filed Dec. 30, 2008, entitled “Methods and Apparatus Related to Data Transmissions Between Virtual Resources at a Host Device” (78 pgs). |
| Office Action dated Apr. 1, 2011 for U.S. Appl. No. 12/346,612 (19 pages). |
| Final Office Action dated Dec. 5, 2011 for U.S. Appl. No. 12/346,612 (24 pages). |
| U.S. Appl. No. 12/346,615, filed Dec. 30, 2008, entitled “Methods and Apparatus for Routing Between Virtual Resources Based on a Routing Location Policy” (79 pgs). |
| Office Action dated Sep. 15, 2010 for U.S. Appl. No. 12/346,615 (21 pages). |
| Office Action dated Mar. 21, 2011 for U.S. Appl. No. 12/346,615 (9 pages). |
| U.S. Appl. No. 12/346,618, filed Dec. 30, 2008, entitled “Methods and Apparatus for Provisioning at a Network Device in Response to a Virtual Resource Migration Notification” (80 pgs). |
| Office Action dated Sep. 15, 2010 for U.S. Appl. No. 12/346,618 (13 pages). |
| Office Action dated Apr. 7, 2011 for U.S. Appl. No. 12/346,618 (14 pages). |
| U.S. Appl. No. 12/346,625, filed Dec. 30, 2008, entitled “Methods and Apparatus Related to Managing Communications Between Virtual Resources” (43 pgs). |
| Office Action dated Sep. 17, 2010 for U.S. Appl. No. 12/346,625 (24 pages). |
| Office Action dated Mar. 29, 2011 for U.S. Appl. No. 12/346,625 (13 pages). |
| Final Office Action dated Dec. 14, 2011 for U.S. Appl. No. 12/346,625 (14 pages). |
| Office Action dated Jul. 3, 2012 for U.S. Appl. No. 12/346,625 (14 pages). |
| Office Action dated Oct. 5, 2010 for U.S. Appl. No. 12/346,632 (16 pages). |
| Office Action dated Mar. 14, 2011 for U.S. Appl. No. 12/346,632 (21 pages). |
| Final Office Action dated Aug. 15, 2011 for U.S. Appl. No. 12/346,632 (23 pages). |
| Office Action dated Dec. 21, 2011 for U.S. Appl. No. 12/346,632 (27 pages). |
| U.S. Appl. No. 12/612,210, filed Nov. 4, 2009, entitled “Methods and Apparatus for Configuring a Virtual Network Switch” (50 pgs). |
| Partial European Search Report dated Mar. 4, 2011 for European Application No. 10187574.8 (6 pages). |
| Extended European Search Report dated Aug. 17, 2011 for European Application No. 10187574.8 (10 pages). |
| U.S. Appl. No. 12/976,508, filed Dec. 22, 2010, entitled “Methods and Apparatus for Tunnel Management Within a Data Center” (42 pgs). |
| U.S. Appl. No. 12/981,678, filed Dec. 30, 2010, entitled “Apparatus, Systems and Methods for Aggregate Routes Within a Communications Network” (38 pgs). |
| Office Action dated Oct. 28, 2010 for U.S. Appl. No. 12/346,630 (24 pages). |
| Office Action dated Apr. 27, 2011 for U.S. Appl. No. 12/346,630 (15 pages). |
| Final Office Action dated Oct. 26, 2011 for U.S. Appl. No. 12/346,630 (17 pages). |
| Partial European Search Report dated Feb. 4, 2011 for European Application No. 10188736.2 (6 pages). |
| Gunes Aybay et al., “Front-to-Back Cooling System for Modular Systems with Orthogonal Midplane Configuration” U.S. Appl. No. 12/167,604, filed Jul. 3, 2008, (25 pgs). |
| Extended European Search report for European Application No. 10188736.2, dated May 10, 2011. |
| Office Action for European Application No. 10188736.2, dated May 16, 2013. |
| Office Action for European Application No. 10188736.2, dated Nov. 18, 2013. |
| Office Action for Chinese Application No. 201010278909.5, dated Feb. 28, 2013. |
| EESR for corresponding European Application No. 15170843.5, dated Oct. 6, 2015. |
| Number | Date | Country | |
|---|---|---|---|
| 20160255019 A1 | Sep 2016 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14610143 | Jan 2015 | US |
| Child | 15151071 | US | |
| Parent | 12607162 | Oct 2009 | US |
| Child | 14610143 | US |
