Patent Grant
9813252
Some embodiments described herein relate generally to multicast group functionality within a network, and more particularly to apparatuses for efficient management of multicast groups and distribution of data packets to members thereof.
Known network fabric systems often include one or more multicast groups each including one or more member devices. Many such multicast groups are configured using the Internet Group Management Protocol (IGMP), and are configured to broadcast data packets to each member of the multicast group. Often, the process of defining and sending copies of a broadcast data packet to each member device included in a multicast group is performed at a single device within the network, resulting in a bottleneck at this replication/distribution point. Thus, a need exists for apparatus to distribute the replication and distribution tasks associated with multicast group broadcasts to multiple devices within a network fabric system.
In some embodiments, a non-transitory processor-readable medium stores code representing instructions configured to cause a processor to receive, from an access switch, a first signal including forwarding state information associated with a first peripheral processing device from a set of peripheral processing devices. The code can further represent instructions configured to cause the processor to receive, from the first peripheral processing device, a second signal including a data packet. The code can further represent instructions configured to cause the processor to send, to a replication engine associated with the set of peripheral processing devices, a third signal such that the replication engine (1) defines a copy of the data packet which is included within the third signal, and (2) sends, to a second peripheral processing device from the set of peripheral processing devices, a fourth signal including the copy of the data packet.
In some embodiments, a communications network can be operatively coupled to one or more access switches and/or compute devices. The communications network, access switches and/or compute devices can be included in a switch fabric system. The communications network can be, for example, a switch core or a multi-stage switch fabric.
In some embodiments, each access switch can be operatively coupled to one or more peripheral processing devices, and can provide connectivity, via the communications network, between the peripheral processing devices to which it is connected and to one or more other devices also coupled to the communications network (e.g., one or more other access switches, peripheral processing devices, compute devices, etc. An access switch can optionally include one or more network control entities (NCEs) configured to manage control plane information associated with one or more devices and/or entities included in the switch fabric system (e.g., forwarding state information of one or more peripheral processing devices). An access switch can also include one or more packet-forwarding engines (PFEs) configured to forward packets to one or more peripheral processing devices coupled thereto. In some embodiments, an NCE can be considered part of a control plane of the switch fabric system and a PFE can be considered part of a data plane of the switch fabric system.
Each compute device can be any combination of hardware and/or software (executing in hardware) configured to store, include, instantiate and/or host one or more logical entities associated with the switch fabric system. For example, a compute device can host one or more of: an NCE, a network management module (NMM), an L2 root module, an L3 root module, a multicast group management module (MGM), a replication engine, etc. In some embodiments, each of the above logical entities can be any combination of hardware and/or software (executing in hardware) operating at a compute device.
In some embodiments, a peripheral processing device can send a login request to an NCE hosted at an access switch. The login request can optionally have a Border Gateway Protocol (BGP) format, and can include identifier information of the peripheral processing device (e.g., Internet Protocol (IP) address, Media Access Control (MAC) address). Based at least in part on the login request, the NCE can store, at a memory, the identifier information. In some embodiments, the NCE can subsequently broadcast the identifier information and/or forwarding state information of the peripheral processing device to one or more other NCEs, NMMs, L2 root modules, L3 root modules and/or multicast group management modules.
In some embodiments, a peripheral processing device can send, to the NCE instantiated at the access switch, a request to join a multicast group. This request can optionally have an Internet Group Management Protocol (IGMP) format. Upon receipt of the request, the NCE can optionally send, based on the request, a BGP-formatted packet. The BGP-formatted packet can be configured to relay the multicast group join request of the peripheral processing device that sent the request to join the multicast group. In some embodiments, the NCE can send the BGP-formatted packet to, for example, an L2 root module, L3 root module and/or MGM. In such embodiments, the recipient module (be it an L2 root module, L3 root module or MGM) can accordingly add the requesting peripheral processing device to the specified multicast group. More specifically, the recipient module can store, at a memory, a record, file and/or association between the requesting peripheral processing device and an identifier of the multicast group (e.g., a multicast group identifier (ID), also referred to as a multicast key).
Having joined the multicast group, the peripheral processing device can subsequently send a data packet to one or more devices included in the multicast group. More specifically, the peripheral processing device can send a signal including the data packet to an access switch. The data packet can optionally include a packet header specifying a desired multicast group (via, for example, a multicast ID), a source identifier of the peripheral processing device and/or a VLAN of the peripheral processing device. The access switch can be configured to forward the data packet to an L2 root module associated with a VLAN in which the peripheral processing device and the access switch are included. In some embodiments, the L2 root module can next determine whether any devices included in the multicast group are likewise included in a different VLAN from that of the L2 root module, the access switch and the peripheral processing device.
If the L2 root module determines that all members of the specified multicast group are likewise members of the same VLAN as the L2 root module, the L2 root module can accordingly send the data packet to one or more replication engines. The one or more replication engines can each be associated with the same VLAN as the L2 root module, and can be hosted/instantiated at a compute device. In such embodiments, each replication engine can be associated with one or more member devices included in the specified multicast group, thus ensuring that each member device will receive a copy of the data packet. In some embodiments, a replication engine can be associated with multiple VLANs. In some embodiments, a single replication engine can be associated with a VLAN and/or multicast group. In some embodiments, upon receipt of the data packet, each replication engine can define a copy thereof and transmit the copy of the data packet to one or more peripheral processing devices from the multicast group. To do so, each replication engine can send the copy of the data packet via (1) the communications network and (2) one or more access switches to which a target peripheral processing device (i.e., a member of the multicast group) is connected.
If the L2 root module determines that at least one member of the specified multicast group is a member of a different VLAN from that of the L2 root module, the L2 root module can send the data packet to an L3 root module. In some embodiments, the L2 root module can also send, to the L3 root module, a separate indicator specifying the identity of the VLAN with which the L2 root module (and thus, the source peripheral processing device and access switch) is associated. In some embodiments, the L3 root module can be hosted/instantiated at a compute device operatively coupled to the communications network. In some embodiments, the compute device can be the same compute device as that at which the L2 root module is hosted. In other embodiments, the compute device can be a distinct compute device from the compute device at which the L2 root module is hosted.
Upon receipt of the data packet, the L3 root module can send, to an MGM module, the packet header included in the data packet. More specifically, the L3 root module can send, to the MGM module, the multicast ID of the specified multicast group. In some embodiments, the MGM module can be hosted at the same compute device as the L3 root module. Alternatively, the MGM module can be hosted at a distinct compute device from of the L3 root module. Based at least in part on the multicast ID, the MGM module can determine which VLANs included in the switch fabric system include member devices of the multicast group, and send a response including this information to the L3 root module.
Upon receipt of the above-described information, the L3 root module can determine which replication engines are associated with the two or more VLANs identified in the response received from the MGM module. Then, based on this determination, the L3 root module can send, to the replication engines, the data packet or a copy thereof.
Upon receipt of the data packet or data packet copy, each of the replication engines can define a copy of the data packet and send the same to one or more peripheral processing devices included in the multicast group and a VLAN with which that replication engine is associated. In some embodiments, any of the replication engines can be located at a single compute device and/or located at various compute devices in groups of one or more.
In some embodiments, the switch core 180 and the distributed control system 190 can collectively (or individually) be referred to as a “network side” of the data center 100, and the network elements outside of the switch core 180 and the data control system 190 can be referred to as a “server side” of the data center 100. In some embodiments, one or more portions of the switch core 180 and/or the distributed control system 190 can be included in the server side of the data center 100. In some embodiments, one or more network elements outside of the switch core 180 and/or the distributed control system 190 can be included in the network side of the data center 100.
The distributed control system 190 can include various network elements such as routing switches, routing engines (REs), and/or so forth. The distributed control system 190 can be a network of elements configured to manage (e.g., process, distribute, define) various types of control plane information used by the switch core 180 so that the switch core 180 can operate in a desirable fashion. In some embodiments, the control plane information can include information used to manage the switch core 180 and/or information used to manage the distributed control system 190. In some embodiments, the control plane information can include, for example, provisioning information, virtual local area network (VLAN) information, routes, forwarding states, configuration information, and/or so forth. In some embodiments, the control plane information can be defined by and/or can include information associated with (e.g., received from) the switch core 180, the distributed control system 190, and/or defined by, for example, a network administrator. In some embodiments, at least a portion of the switch core 180 and a portion of the distributed control system 190 can be included and/or located in the same physical device(s).
As represented by double-headed arrows 20 shown in
In addition, network elements within the distributed control system 190 can be configured to exchange signals with (e.g., communicate with) one another as represented by double-headed arrows 10. In some embodiments, signaling represented by arrows 10 within the distributed control system 190 (e.g., between components of the distributed control system 190) can be related to, for example, the exchange of and/or definition of control plane information. In some embodiments, one or more of the network elements (e.g., packet forwarding engine (PFE), top-of-rack (TOR) device, linecards) of the switch core 180 and/or one or more of the network elements (e.g., routing engines) of the distributed control system 190 can be referred to as intelligent network elements (INEs) (also can be referred to as an independent network elements). Mechanisms for the exchange of control plane information within the DCF (e.g., within the distributed control system 190, between the distributed control system 190 and the switch core 180) are described herein.
In some embodiments, one or more of the INEs of the switch core 180 and/or distributed control system 190 can be associated with a layer-2 (L2) domain (e.g., an L2 broadcast domain) or a layer-3 (L3) domain (e.g., an L3 broadcast domain). In some embodiments, the L2 broadcast domain can be shared by multiple INEs/virtual DCFs (VDCFs) over a single DCF fabric. More details related to a VDCF are set forth below. In some embodiments, data traffic between the INEs/VDCFs for a domain can be exchanged using the switch fabric 180. In some embodiments, one or more L2-domains can be assigned an identifier (ID) which can be common across the INEs/VDCFs that are part of that L2-domain and is used as part of the fabric notification for data packets. In some embodiments, an L2-domain identifier (ID) can also be used for exchanging control information between the member INEs/VDCFs corresponding to that L3-domain (e.g. routes and nexthops). With respect to configuration, an L2-domain can correspond to a VLAN name configured on a DCF and can be shared by one or more of INEs that are members of that VDCF. Across VDCFs, an L2-domain can correspond to a configuration used for normalizing VLAN names used in those VDCFs. In some embodiments, this configuration stanza can be referred to as equivalence-classes.
In some embodiments, an L3 routing domain can be shared by multiple INEs/VDCF over a single DCF fabric. For example, data traffic between the INEs/VDCFs for that domain can be exchanged using the DCF fabric. In some embodiments, each L3-domain can be assigned an ID which can be common across the INEs/VDCFs that are part of the L3-domain and can be used for exchanging control information corresponding to that L3-domain (e.g. routes and nexthops). For configuration purposes, an L3-domain can correspond to a routing-instance name configured on a VDCF and can be shared by one or more INEs that are members of that VDCF. Across VDCFs, an L3-domain can correspond to a configuration used for normalizing routing-instance names used in those VDCFs. In some embodiments, this configuration stanza can be referred to as equivalence-classes.
In some embodiments, one or more of the peripheral processing devices 170 can be configured to communicate via the switch core 180 of the data center 100. Specifically, the switch core 180 of the data center 100 can be configured to provide any-to-any connectivity between the peripheral processing devices 170 at relatively low latency. In some embodiments, the switch core 180 can have at least hundreds or thousands of ports (e.g., egress ports and/or ingress ports) through which peripheral processing devices 170 can transmit and/or receive data. In some embodiments, the peripheral processing devices 170 can be configured to send to and/or receive signals from the switch core 180 based on one or more protocols (e.g., an Ethernet protocol, a multi-protocol label switching (MPLS) protocol, a fibre channel protocol, a fibre-channel-over Ethernet protocol, an Infiniband-related protocol). In some embodiments, the peripheral processing devices can include one or more virtual resources such as virtual machines.
In some embodiments, the switch core 180 can be (e.g., can function as) a single consolidated switch (e.g., a single large-scale consolidated L2/L3 switch). In other words, the switch core 180 can be configured to operate as a single logical entity (e.g., a single logical network element) as opposed to, for example, a collection of distinct network elements configured to communicate with one another via Ethernet connections. The switch core 180 can be configured to connect (e.g., facilitate communication between) the peripheral processing device 170. In some embodiments, the switch core 180 can be configured to communicate via interface devices (e.g., access switches) configured to transmit data at a rate of at least 10 Gb/s. In some embodiments, the switch core 180 can be configured to communicate via interface devices (e.g., fibre-channel interface devices) configured to transmit data at a rate of, for example, 2 Gb/s, 4, Gb/s, 8 Gb/s, 10 Gb/s, 40 Gb/s, 100 Gb/s and/or faster link speeds.
Although the switch core 180 can be logically centralized, the implementation of the switch core 180 can be highly distributed, for example, for reliability. For example, portions of the switch core 180 can be physically distributed across, for example, many chassis. In some embodiments, for example, a processing stage of the switch core 180 can be included in a first chassis and another processing stage of the switch core 180 can be included in a second chassis. Both of the processing stages can logically function as part of a single consolidated switch.
In some embodiments, the switch core 180 can include an edge portion and a switch fabric portion (not shown). The edge portion can include edge devices (not shown) that can function as gateway devices between the switch fabric portion and the peripheral processing devices 170. In some embodiments, edge devices within the edge portion 185 can collectively have thousands of ports (e.g., 100,000 ports, 500,000 ports) through which data from the peripheral processing devices 170 can be transmitted (e.g., routed) into and/or out of one or more portions of the switch core 180. In some embodiments, the edge devices can be referred to as access switches, as network devices, and/or as input/output modules. In some embodiments, the edge devices can be included in, for example, a top-of-rack (TOR) of a chassis, and accordingly the edge devices can be referred to as TOR devices. In some embodiments, the INEs within the data center 100 can be configured to handle data based on different protocols.
In some embodiments, one or more of the components (e.g., a TOR device) within the data center 100 can include an application-specific integrated-circuit (ASIC). In some embodiments, the ASIC can be a packet parsing, classification, and/or forwarding ASIC. In some embodiments, the ASIC can be a buffering and fabric flow control ASIC. In some embodiments, the ASIC can be a fabric switch element ASIC.
In some embodiments, edge devices can be configured to send data to and/or receive data from the switch fabric portion of the switch core 180. In some embodiments, edge devices within the edge portion of the switch core 180 can be configured to classify, for example, data packets received at the switch core 180 from the peripheral processing devices 170. Specifically, the edge devices within the edge portion of the switch core 180 can be configured to perform Ethernet-type classification, which can include classification based on, for example, a layer-2 Ethernet address (e.g., a media access control (MAC) address) and/or a layer-4 Ethernet address (e.g., a universal datagram protocol (UDP) address). The edge devices (or other INEs of the data center 100) can include, for example, a packet forwarding engine (PFE) configured to perform, for example, a parsing function, a classifying function, a forwarding function, and/or a queuing and scheduling function. Thus, packet parsing, packet classifying, packet forwarding, and packet queuing and scheduling can occur prior to a data packet entering the switch core 180. Accordingly, these functions do not need to be performed at stages of the switch core 180. This can reduce the latency associated with the switch core 180. In some embodiments, for example, the end-to-end latency (i.e., time it takes to send data through the switch core 180 from an edge device to another edge device) can be lower than the end-to-end latency of a switch core 180 using an Ethernet protocol.
In some embodiments, one or more routing engines (REs) of the distributed control system 190 can be configured to provide control plane information to one or more PFEs of the switch core 180 so that the PFEs of the switch core 180 can appropriately process data received at the switch core 180. In some embodiments, one or more of the REs can be based on one or more virtual resources. In some embodiments, the distributed control system 190 can be defined, at least in part, by a network of REs and RE switches. In some embodiments, at least some of the signaling represented by arrows 10 shown in
Data can be processed at the peripheral processing devices 170 and/or at the switch core 180 based on different platforms. For example, communication between one or more of the peripheral processing devices 170 and an edge device at the edge portion can be a stream of data packets defined based on an Ethernet protocol or a non-Ethernet protocol. In some embodiments, various types of data processing can be performed at edge devices within the edge portion of the switch core 180 that may not be performed within the switch fabric portion of the switch core 180. For example, data packets can be parsed into cells at the edge device of edge portion of the switch core 180, and the cells can be transmitted from the edge device to the switch fabric portion of the switch core 180. The cells can be parsed into segments and transmitted within the switch fabric portion of the switch core 180 as segments (also can be referred to as flits in some embodiments). In some embodiments, the data packets can be parsed into cells at a portion of the switch fabric portion of the switch core 180. In some embodiments, a congestion resolution scheme can be implemented at and/or scheduling of transmission of data (e.g., cells) via the switch fabric portion of the switch core 180 can be performed at edge devices (e.g., access switches) within the edge portion of the switch core 180. Congestion resolution schemes and/or scheduling of transmissions of data, however, need not be performed within modules that define the switch fabric of the switch core 180.
In some embodiments, the above-described architecture can support forwarding of multi-destination frames. In some embodiments, these frames can be of one or more of the following types: L2 Broadcast, L2 Unknown Unicast, L2 Known Multicast (defined based on Generic Attribute Registration Protocol (GARP) and/or Generic Multicast Registration Protocol (GMRP)), L2 Unknown (non-IP) Multicast, L3(IP) Known Multicast (link-local and global) and L3 (IP) Unknown (i.e., sender-only) Multicast. Data frames defined according to one or more of the above-described multi-destination frame types can be collectively referred to as BUM (Broadcast, Unknown unicast and Multicast) traffic.
The physical ports 231 and 232 can be configured to communicate with Ethernet and/or Fibre Channel peripheral processing devices, optionally via an Ethernet network. Additionally or alternatively, the physical ports 231 and 232 can be configured to communicate with Fibre Channel devices, such as Fibre Channel switches. For example, the physical ports 231 and 232 can implement a physical layer using twisted-pair electrical signaling via electrical cables or fiber-optic signaling via fiber-optic cables. In some embodiments, one of the physical ports 231 and 232 can implement one physical layer such as twisted-pair electrical signaling, and the other of the physical ports 231 and 232 can implement a different physical layer, such as fiber-optic signaling. Furthermore, the physical ports 231 and 232 can be configured to allow the compute device 200 to communicate with other peripheral processing devices, switching devices and/or edge devices (e.g., other compute devices (or “compute nodes”)) via a common protocol such as Ethernet, Fibre Channel and/or Fibre Channel over Ethernet (FCoE). In some embodiments, one of the physical ports 231 and 232 can implement one protocol such as Ethernet/FCoE and the other of the physical ports 231 and 232 can implement a different protocol such as Fibre Channel. Thus, the compute device 200 can be in communication with multiple peripheral processing and/or switching devices using homogeneous or heterogeneous physical layers and/or protocols via the physical ports 231 and 232.
The L2 switching module 221 can be any hardware-based module and/or software-based module (executing in hardware) configured to receive and process information from one or more devices or modules capable of communicating on a layer-2 basis, i.e., communicating based at least in part on physical address (e.g., an Ethernet MAC address) of a sender and/or a recipient device or module. As shown in
In some embodiments, the L2 switching module 221 can receive, from a peripheral processing device (e.g., a member device included in a specified multicast group), a data packet to be transmitted to one or more member devices included in a specified multicast group. The data packet can optionally include, in a packet header, an identifier of the specified multicast group (e.g., a multicast group ID). In such embodiments, the L2 switching module 221 can accordingly forward the received data packet to one or more other compute devices for copying and/or transmission of the data packet to the member devices. In some embodiments, one or more of the other compute devices can be and/or can include at least one replication engine module configured to: (1) define one or more copies of a data packet and (2) send the copies of the data packet to one or more devices (e.g., peripheral processing devices) included in a multicast group.
The L3 switching module 222 can be any hardware-based module and/or software-based module (executing in hardware) configured to receive and/or process information from and/or associated with one or more devices or modules capable of communicating on a layer-3 basis, i.e., communicating based at least in part on a network layer address (e.g., an Internet Protocol (IP) address) of a sender and/or recipient device or module. As shown in
Based at least in part on this information, the L3 switching module 222 can determine which VLANs are associated with the various member peripheral processing devices of a multicast group specified by a received packet. Then, based at least in part on this VLAN information, the L3 switching module 222 can further determine to which of a set of replication engines associated with each such VLAN to send the data packet for replication and subsequent transmission.
Finally, the L3 switching module 222 can optionally send the data packet to at least a first replication engine (e.g., a replication engine module instantiated/hosted at a compute device) for copying and transmission to one or more peripheral processing devices included in the specified multicast group.
In some embodiments, the L3 switching module 222 can send the data packet to a replication engine along with information associated with one or more other replication engines. The one or more other replication engines can optionally be associated with at least one VLAN, the VLAN including at least one peripheral processing device from the specified multicast group. Then, based at least in part on the replication engine information, the first replication engine can send the data packet to the one or more other replication engines for copying and transmission thereof to the remaining peripheral processing devices from the multicast group. In this manner, the L3 switching module 223 can send a single signal to a single replication engine such that multiple replication engines define copies of a packet included in the signal and then send the copies to multiple peripheral processing devices.
The multicast management module 223 can be any hardware-based module and/or software-based module (executing in hardware) configured to store and/or provide information associated with one or more multicast groups, peripheral processing devices and/or VLANs. As shown in
In some embodiments, the multicast management module 223 can be configured to store, at the memory 220, information associated with one or more multicast groups, including, for example, multicast group information (e.g., multicast group identifier (ID), multicast group name), multicast group member device information (e.g., device MAC addresses, device IP addresses), VLAN device membership information (e.g., association between a given device and a VLAN), etc. In such embodiments, the compute device 200 can be configured to reply to one or more queries for any or all of the above information.
The replication engine module 224 can be any hardware-based module and/or software-based module (executing in hardware) configured to define and/or transmit one or more data packets to one or more member devices (e.g., devices included in a multicast group). As shown in
The communications network 310 can be any combination of hardware and/or software (executing on hardware) configured to transmit data between any of the peripheral processing devices 341-344, the compute device 320, the compute device 350, and/or any of the access switches 331-332. In some embodiments, the communications network 310 can be a switch fabric or switch core, such as a multi-stage switch fabric. The communications network 310 can optionally transmit data based at least in part on the Ethernet, Fibre Channel, FCoE, and/or another network protocol (such as cell-based network transmission). Additional details related to communications networks such as switch fabrics and multi-stage switch fabrics using cell-based network transmission are disclosed in U.S. patent application Ser. No. 12/495,337 entitled “Methods and Apparatus Related to Any-to-Any Connectivity within a Data Center” filed Jun. 30, 2009, which is incorporated herein by reference in its entirety. In some embodiments, the communications network 310 can include one or more hardware devices configured to exchange data according to one or more of the above-enumerated network protocols. Additional details related to communications networks such as switch fabrics and multi-stage switch fabrics are disclosed in U.S. patent application Ser. No. 12/558,130 entitled “Methods and Apparatus Related to a Low Cost Data Center Architecture,” filed Sep. 11, 2009, which is incorporated herein by reference in its entirety.
Each of the access switches 331-332 can be any combination of hardware and/or software (executing in hardware) situated at the edges of the communications network 310. As shown in
In some embodiments, each of the access switches 331-332 can be physically located within a chassis of the switch fabric system 300. In some embodiments, for example, each access switch 331-332 can be located within the same chassis. In other embodiments, each access switch 331-332 can be located within a different chassis. Structurally, the access switches 331-332 can function as both source access switches and destination access switches. Accordingly, the access switches 331-332 can send signals including data (e.g., a data stream of data frames, packets and/or data cells) to and receive signals including data from a data plane portion of the communications network 310, and to and/or from the peripheral processing devices 341-344. Each of the access switches 331-332 can optionally be referred to as an edge device and/or a top-of-the-rack “TOR” device.
As shown in
As discussed in further detail herein, the access switches 331 and the access switch 332 can be configured to host one or more network control entities (NCEs) to manage, for example, the peripheral processing devices 341-342 and 343-344, respectively. As shown in
In some embodiments, each of the NCEs 372-373 can be defined and/or spawned by a controlling entity or module, such as a network management module (not shown in
Each of the access switches 331 and 332 can be further configured to host one or more packet-forwarding engines (PFEs), such as the PFE 374 hosted at the access switch 331 and the PFE 375 hosted at the access switch 332. In some embodiments, each of the PFE 374 and the PFE 375 can be a hardware module and/or software-based module (executing in hardware) instantiated and/or hosted at a physical device (e.g., an access switch) and configured to transmit traffic between two or more devices. More specifically, each of the PFE 374 and the PFE 375 can receive one or more packets and forward the same to one or more peripheral processing devices operatively coupled to the access switch at which that PFE is hosted. For example, as shown in
The compute devices 320 and 350 can each be any combination of hardware and/or software (executing on hardware) configured to perform one or more network management tasks. In some embodiments, the compute devices 320 and 350 can be server devices. The compute devices 320 and 350 can be physically and/or operatively coupled to the communications network 310 via, for example, a wired and/or wireless Ethernet, Fibre Channel or other physical and/or logical connection.
As shown in
As shown in
Each of the peripheral processing devices 341-344 can be any combination of hardware and/or software (executing in hardware) capable of transmitting and/or receiving information across the communications network 310 via an access switch. In some embodiments, one or more of the above-enumerated peripheral processing devices can optionally be, for example, a compute node, a service node, a router, or a storage node. In some embodiments, one or more of the peripheral processing devices 341-344 can perform one or more computing tasks, such as one or more data storage, Software as a Service (SAS), web service, content request, or other computing tasks.
The peripheral processing devices 341-344 can be in communication with and/or operatively coupled to one or more physical ports of the access switches 331-332 (not shown in
In some embodiments, a peripheral processing device can send a request to join a multicast group included in the switch fabric system 300. For example, as shown in
The NCE 372 can next send, via the communications network 310, a signal 381 to the L2 root module 322. The signal 381 can be based at least in part on the signal 380, and can include a request to join the specified multicast group. In some embodiments, the signal 381 can have a BGP format configured to be processed by the L2 root module 322. Upon receipt of the signal 381 including the multicast join request, the L2 root module 322 can store, (e.g., at the memory 220 included in the compute device 200 shown in
The switch fabric system 300 can also be configured to transmit (e.g., multicast) one or more data packets to one or more members of a multicast group. For example, as shown in
Upon receipt of the signal 390, the NCE 372 can define and send, via the access switch 331, a signal 391. As shown in
Upon receipt of the signal 391, the L2 root module 322 of the compute device 320 can perform a lookup and/or query on the multicast ID of the packet header included in the data packet. For example, the L2 root module 322 can send a first query to a database (not shown in
In some embodiments, the L2 root module 322 can send a second query configured to determine which replication engines from the replication engines 352-356 are associated with the various members of the multicast group (e.g., the peripheral processing devices 341-344), and thus to which replication engines the data packet should be sent by the L2 root module 322 for replication and transmission. (Alternatively, this second query can be included in the first query, such that the L2 root module 322 sends only a single query sufficient to retrieve/receive the multicast group and replication engine information described above.) Based at least in part on the second query, the L2 root module 322 can receive a second response including identifier information of at least one replication engine from the replication engines 352-356 associated with the multicast group. In some embodiments, the L2 root module 322 can receive forwarding state, login and/or other information associated with the replication engines 352-356 via a login or other signal received from the compute device 350 (not shown in
Having determined identifier information of each member device or entity included in the multicast group, the L2 root module 322 can send a signal 392 including the data packet to the compute device 350 (via the communications network 310). More specifically, the L2 root module 322 can send the signal 392 to one or more replication engines 352-356 associated with the multicast group (as indicated by the second response described above). Alternatively, the L2 root module 322 can send the signal 392 to a single replication engine instantiated at the compute device 350. In such embodiments, the single replication engine can be configured to determine which replication engines from the replication engines 352-356 to employ in defining and transmitting copies of the data packet. Having made this determination, the single replication engine can next propagate the signal 392 and/or a copy of the data packet to one or more additional recipient replication engines from the replication engines 352-356.
Upon receipt of the data packet via the signal 392 and/or another replication engine, each of the selected replication engines from the replication engines 352-356 can define a copy of the data packet. As shown in
Upon receipt of the signal 393, the PFE 374 can define and send a signal 396 to the peripheral processing device 342. The signal 396 can include the copy of the data packet. Upon receipt of the signals 394-395, the PFE 375 can send signals 397 and 398 to the peripheral processing devices 343 and 344, respectively. As with the signal 396, the signals 397 and 398 can include a copy of the data packet for receipt and processing by the peripheral processing devices 343 and 344, respectively.
The communications network 410 can be any combination of hardware and/or software (executing on hardware) configured to transmit data between any of the peripheral processing devices 441-443 and 445-446, the compute devices 420, 422, 424 and 426, and/or any of the access switches 431-433. In some embodiments, the communications network 410 can be a switch fabric or switch core, such as a multi-stage switch fabric. In some embodiments, the communications network 410 can be similar to the communications network 310 discussed in connection with
Each of the access switches 431-433 can be any combination of hardware and/or software (executing in hardware) situated at the edges of the communications network 410. As shown in
As shown in
As discussed in further detail herein, the access switches 431, 432 and 433 can be configured to host one or more network control entities (NCEs) to manage, for example, the peripheral processing devices 441-442, 443 and 445-446, respectively. As shown in
The compute devices 420, 422, 424 and 426 can each be any combination of hardware and/or software (executing on/in hardware) configured to perform one or more network management tasks (e.g., control plane tasks). In some embodiments, the compute devices 420, 422, 424 and 426 can be physically and/or operatively coupled to the communications network 410 and can be similar to the compute device 320 and/or the compute device 350 discussed in connection with
As shown in
As also shown in
The L3 root module 423 can also be configured to receive one or more multicast group join requests and/or one or more multicast data packets for transmission to a multicast group. In some embodiments, the L3 root module 423 can exchange information with the MGM module 425 hosted at the compute device 424. The exchanged information can include and/or can be based on, for example, multicast group, VLAN and/or member peripheral processing device information. Said differently, the L3 root module 423 can exchange information with the MGM module 425 regarding which multicast groups within the switch fabric system 400 include which peripheral processing devices and/or which VLANs include which peripheral processing devices. In this manner, the L3 root module 423 can also determine and/or exchange information with the MGM module 425 regarding which VLANs include one or more peripheral processing devices from a given multicast group.
The compute device 424 includes and/or hosts the MGM module 425. Although not shown in
As shown in
Each of the peripheral processing devices 441-443 and 445-446 can be any combination of hardware and/or software (executing in hardware) capable of transmitting and/or receiving information across the communications network 410 via an access switch. The peripheral processing devices 441-443 and 445-446 can be configured to send data (e.g., data frames, data packets, data cells, etc.) to and receive data from the access switches 431-433. In some embodiments, each of the peripheral processing devices 441-443 and 445-446 can be similar to one or more of the peripheral processing devices 341-344 of
As shown in
Upon receipt of the signal 490, the NCE 472 can optionally store and broadcast forwarding state information of the peripheral processing device 441. The forwarding state information can include, for example an IP address, a MAC address and/or other identifying information of the peripheral processing device 441. In such embodiments, the NCE 472 can optionally broadcast the forwarding state information of the peripheral processing device 441 to one or more other control plane entities of the switch fabric system 400. For example, the NCE 472 can send signals 491-495 to the L2 root module 421, the L3 root module 423, the MGM module 425, and the NCEs 474 and 476, respectively.
In some embodiments, each of the signals 491-495 can have a BGP format. Upon receipt of a signal including forwarding state information of the peripheral processing device 441, each control plane entity (e.g., the L2 root module 421, the L3 root module 423, the MGM module 425, the NCE 474 and/or the NCE 476) can store, at a memory, the updated forwarding state information. For example, a control plane entity can update a forwarding state table, file, record or database based at least in part on the forwarding state information. In this manner, the NCE 472 can ensure that subsequent signals and/or data packets sent to the peripheral processing device 441 can be properly routed through the communications network 410 to arrive at the peripheral processing device 441 via the access switch 431.
Although not shown in
Based at least in part on the received signal, the L3 root module 423 and/or the MGM module 425 can accordingly update, at a memory, membership information of the multicast group. The updated information can include, for example, a MAC address, IP address, VLAN and/or other information of the peripheral processing device 441. In this manner, the peripheral processing device 441 can be added to a specified multicast group and can thus be configured to receive subsequent multicast broadcasts and/or packets directed to the multicast group. In some embodiments, the MGM module can send, in response to the signal including the request, a second signal indicating that the requesting peripheral processing device(s) has been associated with the multicast ID, i.e., included in/added to the multicast group. In some embodiments, the second signal can be sent to the L3 root module 423 and can have a Protocol Independent Multicast (PIM) format.
The communications network 510 can be similar to the communications network 410 described in connection with
Each of the access switches 531, 532 and 533 can be configured to host one or more network control entities (NCEs) to manage, for example, the peripheral processing devices 541-542, 543 and 545-546, respectively. As shown in
The compute devices 520, 522, 524 and 526 can each be any combination of hardware and/or software (executing on/in hardware) configured to perform one or more network management tasks (e.g., control plane tasks). In some embodiments, the compute devices 520, 522, 524 and 526 can be physically and/or operatively coupled to the communications network 510 and can be similar to the compute device 420 and/or the compute device 450 discussed in connection with
As shown in
As also shown in
The compute device 524 includes and/or hosts the MGM module 525. Although not shown in
As shown in
Each of the peripheral processing devices 541-543 and 545-546 can be any combination of hardware and/or software (executing in hardware) capable of transmitting and/or receiving information across the communications network 510 via an access switch. The peripheral processing devices 541-543 and 545-546 can be configured to send data (e.g., data frames, data packets, data cells, etc.) to and receive data from the access switches 531-533. In some embodiments, each of the peripheral processing devices 541-543 and 545-546 can be similar to one or more of the peripheral processing devices 441-443 and 445-446 of
In some embodiments, any of the peripheral processing devices 541-543 and 545-546 can be configured to send a signal to a multicast group spanning multiple VLANs. For example, as shown in
The NCE 572 can next define and send a signal 591 to the L2 root module 521 hosted at the compute device 520. The signal 591 can include the data packet (and thus the packet header). In some embodiments, the signal can have an Ethernet and/or Internet Protocol format.
Upon receipt of the signal 591, the L2 root module 521 can determine, based on the packet header and/or multicast group ID, a multicast group to which the data packet is directed. Then, based at least in part on the multicast group ID, the L2 root module 521 can determine which devices are included in the multicast group. To do so, the L2 root module 521 can query a memory, database, or other data store, record or file (not shown in
Based at least in part on the multicast group membership and VLAN information described above, the L2 root module 521 can determine that one or more multicast group member devices is not included in the same VLAN as the peripheral processing device that sent the data packet. More specifically, the L2 root module 521 can determine that the peripheral processing device 541 (included in the VLAN 580) is in a different VLAN from, for example, the peripheral processing device 545 (which is also a member of the multicast group, but is included in the VLAN 585). In this instance, inasmuch as the L2 root module 521 is in direct communication with and/or authorized to administer over only devices included in the VLAN 580, the L2 root module 521 can determine that it is incapable of sending the data packet to the peripheral processing devices 545-546. Having made this determination, the L2 root module 521 can send a signal 592 to the L3 root module 523. The signal 592 can include the data packet. In some embodiments, rather than perform the determining step described above, the L2 root module 521 can alternatively forward the data packet (included in the signal 592) to the L3 root module 523 immediately upon receipt from the NCE 572.
Having received the signal 592 including the data packet, the L3 root module 523 can send a signal 593 to the multicast group management (MGM) module 525. The signal 593 can include, for example, the multicast group ID, and can be configured to retrieve, from the MGM module 525, information associated with each multicast group member device and VLAN. In some embodiments, the L3 root module 523 can send a signal 594 to the L3 root module 523, the signal 594 including information describing the VLAN membership of each device included in the multicast group. In some embodiments the signal 594 can further include information describing associations between one or more of the replication engines 554-556 and one or more of the VLANs 580 and 585.
Upon receipt of the signal 594, the L3 root module 523 can determine to which replication engines from the replication engines 554-556 it should send the data packet such that the data packet is copied and transmitted to each multicast group member device included in each of the VLANs 580 and 585. More specifically, the L3 root module 523 can determine that the replication engines 554 and 555 are associated with the VLAN 580 (and thus the peripheral processing devices 541-543), and that the replication engine 556 is associated with the VLAN 585 (and thus the peripheral processing devices 545-546).
Having made the above-described determinations, in some embodiments the L3 root module 523 can next send a signal 595 to the compute device 526. More specifically, the L3 root module 523 can send, via the communications network 510, the signal 595 to at least one of the replication engines 554-556 hosted at the compute device 526. As described in connection with
Upon receipt of the data packet (be it directly from the L3 root module 523 or another of the replication engines 354-356), each replication engine can define at least one copy of the data packet and transmit the same, via the communications network 510, to the peripheral processing devices included in the VLAN with which that replication engine is associated. More specifically, the replication engine 554 can define a copy of the data packet and send a signal 596 including the same. As shown in
A layer-2 root device can receive a data packet from a peripheral processing device via an access switch, 600. More specifically, a layer-2 root device or module (“L2 device”) can receive the data packet via an access switch operatively coupled to a source peripheral processing device and to the L2 device. In some embodiments, each of the access switch and the L2 device can exchange information via a switch core (e.g., a multi-stage switch fabric) of a switch fabric system. In some embodiments, the L2 device can be a hardware-based module or a software-based module (executing in hardware) hosted at a compute device (or “compute node”) coupled to the switch core.
The L2 device can determine that the data packet is associated with one or more peripheral processing devices included in a VLAN other than the VLAN (“VLAN A”) with which the L2 device is associated, 610. For example, the L2 device can examine the data packet (e.g., a header of the data packet) to determine a multicast group ID included therein. Based at least in part on the multicast group ID, the L2 device can query a database or other data store to determine a set of member devices included in the multicast group and which (if any) VLANs of the switch fabric system include a member device from the set of devices. Based on this information, the L2 device can determine whether any of the multicast group member devices is included in a VLAN other than VLAN A. If the L2 device determines that all member devices included in the multicast group are included in VLAN A, the L2 device can send the data packet to one or more replication engines for copying and transmission of the data packet thereto (see steps 660-670 below).
Alternatively, if the L2 device determines that one or more member devices included in the multicast group is not included in VLAN A, the L2 device can send the data packet to a layer-3 root device, 620. The layer-3 root device can be, for example, a device and/or module (“L3 device”) configured to store information regarding and/or manage one or more modules and/or devices based at least in part on the network layer of those modules and/or devices. In some embodiments, the L3 device can be operatively coupled to the L2 device via the switch core and/or directly.
The L3 device can receive the data packet and send a packet header of the data packet to a multicast group manager module (MGM), 630. In some embodiments, the packet header can include a source address of the sending peripheral processing device (e.g., an IP address, a MAC address) and a multicast group ID. The MGM module can be operatively coupled to the switch core and can be configured to exchange information with the L2 device, the L3 device and/or one or more replication engines also coupled to the switch core.
The MGM module can receive the packet header from the L3 device, 640. Based at least in part on the multicast group ID, the MGM module can determine the existence of one or more multicast group member devices (e.g., peripheral processing devices) and one or more VLANs included in the switch fabric system (e.g., the VLAN A). Having determined which multicast group devices are associated with which VLANs, the MGM module can send the association information to the L3 device.
Upon receipt of the association information described above, the L3 device can send one or more signals including the data packet to a set of replication engines, 650. More specifically, the L3 device can send, via the switch core, a signal including the data packet to a first replication engine from the set of replication engines (e.g., a “root node” of a replication engine tree structure). In this manner, the L3 device can send the data packet to a first replication engine, which can subsequently send the data packet to one or more other replication engines associated with one or more of the VLANs associated with one or more of the multicast group member devices. The replication engines can each be a hardware-based module and/or a software-based module (executing in hardware) hosted and/or instantiated at a device, such as a compute device operatively coupled to the switch core. In some embodiments, one or more of the replication engines can be hosted at one or more devices or servers positioned throughout the switch fabric system.
Each replication engine can define one or more copies of the data packet, 660. More specifically, each replication engine associated with a VLAN that includes at least one multicast group member device can define a copy of the data packet and include the same in one or more signals.
Having defined the one or more signals including the copies of the data packet, each replication engine can send its copy or copies of the data packet to the multicast group member devices with which it is associated, 670. More specifically, each replication engine can send at least one copy of the data packet to a peripheral processing device via the switch core and/or one or more access switches. In some embodiments, each replication engine can send a copy of the data packet to each multicast group member device included in the VLAN with which that replication engine is associated. In some embodiments, one or more replication engines can send a copy of the data packet to at least one—but not necessarily all—multicast group member devices included in the VLAN with which that replication engine is associated.
Some embodiments described herein relate to a computer storage product with a computer-readable medium (also can be referred to as a processor-readable medium) having instructions or computer code thereon for performing various computer-implemented operations. The media and computer code (also can be referred to as code) may be those designed and constructed for the specific purpose or purposes. Examples of computer-readable media include, but are not limited to: magnetic storage media such as hard disks, floppy disks, and magnetic tape; optical storage media such as Compact Disc/Digital Video Discs (CD/DVDs), Compact Disc-Read Only Memories (CD-ROMs), and holographic devices; magneto-optical storage media such as optical disks; carrier wave signal processing modules; and hardware devices that are specially configured to store and execute program code, such as Application-Specific Integrated Circuits (ASICs), Programmable Logic Devices (PLDs), and read-only memory (ROM) and RAM devices.
Examples of computer code include, but are not limited to, micro-code or micro-instructions, machine instructions, such as produced by a compiler, code used to produce a web service, and files containing higher-level instructions that are executed by a computer using an interpreter. For example, embodiments may be implemented using Java, C++, or other programming languages (e.g., object-oriented programming languages) and development tools. Additional examples of computer code include, but are not limited to, control signals, encrypted code, and compressed code.
While various embodiments have been described above, it should be understood that they have been presented by way of example only, not limitation, and various changes in form and details may be made. Any portion of the apparatus and/or methods described herein may be combined in any combination, except mutually exclusive combinations. The embodiments described herein can include various combinations and/or sub-combinations of the functions, components and/or features of the different embodiments described. For example, multiple L2 root modules can be hosted at multiple compute devices operatively coupled to a common switch core.
The present application claims priority to U.S. provisional patent application No. 61/316,719 entitled “Multicasting within a Distributed Control Plane of a Switch,” filed on Mar. 23, 2010, and to U.S. provisional patent application No. 61/316,720 entitled “Methods and Apparatus Related To Distributed Control Plane Switch Management,” filed on Mar. 23, 2010, both of which are hereby incorporated by reference in their entireties. This patent application is also related to co-pending U.S. patent application Ser. No. 12/495,337, entitled “Methods and Apparatus Related to Any-to-Any Connectivity within a Data Center” and filed on Jun. 30, 2009; to U.S. patent application Ser. No. 12/495,344, entitled “Methods and Apparatus Related to Lossless Operation within a Data Center” and filed on Jun. 30, 2009; to U.S. patent application Ser. No. 12/495,358, entitled “Methods and Apparatus Related to Low Latency within a Data Center” and filed on Jun. 30, 2009; to U.S. patent application Ser. No. 12/495,361, entitled “Methods and Apparatus Related to Flow Control within a Data Center Switch Fabric” and filed on Jun. 30, 2009; to U.S. patent application Ser. No. 12/495,364, entitled “Methods and Apparatus Related to Virtualization of Data Center Resources” and filed on Jun. 30, 2009; to U.S. patent application Ser. No. 12/558,130, entitled “Methods and Apparatus Related to a Low Cost Data Center Architecture” and filed on Sep. 11, 2009; and to U.S. patent application Ser. No. 12/558,126, entitled “Methods and Apparatus Related to a Flexible Data Center Security Architecture” and filed on Sep. 11, 2009. Each of the above-identified applications is incorporated herein by reference in its entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 599518 | Charles Runge | Feb 1898 | A |
| 4942574 | Zelle | Jul 1990 | A |
| 5367520 | Cordell | Nov 1994 | A |
| 5457682 | Haag et al. | Oct 1995 | A |
| 5689508 | Lyles | Nov 1997 | A |
| 5801641 | Yang et al. | Sep 1998 | A |
| 5926473 | Gridley | Jul 1999 | A |
| 5945922 | Gao et al. | Aug 1999 | A |
| 5991295 | Tout et al. | Nov 1999 | A |
| 5991297 | Palnati | Nov 1999 | A |
| 6049542 | Prasad | Apr 2000 | A |
| 6049546 | Ramakrishnan | Apr 2000 | A |
| 6067286 | Jones et al. | May 2000 | A |
| 6078503 | Gallagher et al. | Jun 2000 | A |
| 6246692 | Dai et al. | Jun 2001 | B1 |
| 6335930 | Lee | Jan 2002 | B1 |
| 6473428 | Nichols et al. | Oct 2002 | B1 |
| 6553028 | Tang et al. | Apr 2003 | B1 |
| 6587470 | Elliot et al. | Jul 2003 | B1 |
| 6597689 | Chiu et al. | Jul 2003 | B1 |
| 6665495 | Miles et al. | Dec 2003 | B1 |
| 6751238 | Lipp et al. | Jun 2004 | B1 |
| 6760339 | Noel et al. | Jul 2004 | B1 |
| 6807175 | Jennings et al. | Oct 2004 | B1 |
| 6829237 | Carson et al. | Dec 2004 | B2 |
| 6850704 | Dave | Feb 2005 | B1 |
| 6868082 | Allen, Jr. et al. | Mar 2005 | B1 |
| 6876652 | Bell et al. | Apr 2005 | B1 |
| 6981078 | Paul | Dec 2005 | B2 |
| 6990097 | Norman et al. | Jan 2006 | B2 |
| 7046661 | Oki et al. | May 2006 | B2 |
| 7082134 | Lim et al. | Jul 2006 | B1 |
| 7088710 | Johnson et al. | Aug 2006 | B1 |
| 7173931 | Chao et al. | Feb 2007 | B2 |
| 7177919 | Truong et al. | Feb 2007 | B1 |
| 7178052 | Hebber et al. | Feb 2007 | B2 |
| 7180862 | Peebles et al. | Feb 2007 | B2 |
| 7230947 | Huber et al. | Jun 2007 | B1 |
| 7248760 | Corbalis et al. | Jul 2007 | B1 |
| 7295566 | Chiu et al. | Nov 2007 | B1 |
| 7406038 | Oelke et al. | Jul 2008 | B1 |
| 7408927 | George | Aug 2008 | B2 |
| 7420972 | Woo | Sep 2008 | B1 |
| 7428219 | Khosravi | Sep 2008 | B2 |
| 7430171 | Black et al. | Sep 2008 | B2 |
| 7489625 | Varma | Feb 2009 | B2 |
| 7496252 | Corbalis et al. | Feb 2009 | B1 |
| 7505458 | Menon et al. | Mar 2009 | B2 |
| 7519054 | Varma | Apr 2009 | B2 |
| 7564869 | Cafiero et al. | Jul 2009 | B2 |
| 7586909 | Walrand et al. | Sep 2009 | B1 |
| 7590102 | Varma | Sep 2009 | B2 |
| 7606262 | Beshai et al. | Oct 2009 | B1 |
| 7688816 | Park et al. | Mar 2010 | B2 |
| 7720064 | Rohde | May 2010 | B1 |
| 7746799 | Kokot et al. | Jun 2010 | B2 |
| 7751416 | Smith et al. | Jul 2010 | B2 |
| 7792017 | Lu et al. | Sep 2010 | B2 |
| 7830905 | Scott et al. | Nov 2010 | B2 |
| 7843836 | Elliott et al. | Nov 2010 | B2 |
| 7873693 | Mehrotra et al. | Jan 2011 | B1 |
| 7899930 | Turner et al. | Mar 2011 | B1 |
| 7936776 | Deng | May 2011 | B2 |
| 7961734 | Panwar et al. | Jun 2011 | B2 |
| 8054832 | Shukla | Nov 2011 | B1 |
| 8265071 | Sindhu et al. | Sep 2012 | B2 |
| 20020019958 | Cantwell et al. | Feb 2002 | A1 |
| 20020034183 | Prabhakar et al. | Mar 2002 | A1 |
| 20020061020 | Chao et al. | May 2002 | A1 |
| 20020064170 | Siu et al. | May 2002 | A1 |
| 20020118692 | Oberman et al. | Aug 2002 | A1 |
| 20020136484 | MacDonald | Sep 2002 | A1 |
| 20020141397 | Piekarski et al. | Oct 2002 | A1 |
| 20020145974 | Saidi et al. | Oct 2002 | A1 |
| 20020159449 | Carson et al. | Oct 2002 | A1 |
| 20020168012 | Ramaswamy | Nov 2002 | A1 |
| 20030026287 | Mullendore et al. | Feb 2003 | A1 |
| 20030081540 | Jones et al. | May 2003 | A1 |
| 20030084219 | Yao et al. | May 2003 | A1 |
| 20030123453 | Ooghe | Jul 2003 | A1 |
| 20030165140 | Tang et al. | Sep 2003 | A1 |
| 20030200330 | Oelke et al. | Oct 2003 | A1 |
| 20030200473 | Fung | Oct 2003 | A1 |
| 20030223420 | Ferolito | Dec 2003 | A1 |
| 20040030766 | Witkowski | Feb 2004 | A1 |
| 20040034864 | Barrett et al. | Feb 2004 | A1 |
| 20040039986 | Solomon et al. | Feb 2004 | A1 |
| 20040062202 | Storr | Apr 2004 | A1 |
| 20040117438 | Considine et al. | Jun 2004 | A1 |
| 20040165598 | Shrimali et al. | Aug 2004 | A1 |
| 20040258003 | Kokot et al. | Dec 2004 | A1 |
| 20050002334 | Chao et al. | Jan 2005 | A1 |
| 20050025141 | Chao et al. | Feb 2005 | A1 |
| 20050055428 | Terai et al. | Mar 2005 | A1 |
| 20050102549 | Davies et al. | May 2005 | A1 |
| 20050114656 | Liu | May 2005 | A1 |
| 20050120160 | Plouffe | Jun 2005 | A1 |
| 20050175017 | Christensen et al. | Aug 2005 | A1 |
| 20060018379 | Cooper | Jan 2006 | A1 |
| 20060029072 | Perera et al. | Feb 2006 | A1 |
| 20060092940 | Ansari | May 2006 | A1 |
| 20060165070 | Hall et al. | Jul 2006 | A1 |
| 20060165085 | Konda | Jul 2006 | A1 |
| 20060165098 | Varma | Jul 2006 | A1 |
| 20060165111 | Varma | Jul 2006 | A1 |
| 20060165112 | Varma | Jul 2006 | A1 |
| 20060269187 | Lin et al. | Nov 2006 | A1 |
| 20070002883 | Edsall et al. | Jan 2007 | A1 |
| 20070006056 | Lehner et al. | Jan 2007 | A1 |
| 20070091891 | Zwiebel | Apr 2007 | A1 |
| 20070121499 | Pal et al. | May 2007 | A1 |
| 20070189283 | Agarwal et al. | Aug 2007 | A1 |
| 20070280253 | Rooholamini et al. | Dec 2007 | A1 |
| 20070291535 | Eberle et al. | Dec 2007 | A1 |
| 20080044181 | Sindhu | Feb 2008 | A1 |
| 20080065749 | Kucukyavuz et al. | Mar 2008 | A1 |
| 20080075071 | Beshai | Mar 2008 | A1 |
| 20080080548 | Mullendore et al. | Apr 2008 | A1 |
| 20080095160 | Yadav | Apr 2008 | A1 |
| 20080151863 | Lawrence et al. | Jun 2008 | A1 |
| 20080159277 | Vobbilisetty et al. | Jul 2008 | A1 |
| 20080175239 | Sistanizadeh et al. | Jul 2008 | A1 |
| 20080212472 | Musacchio et al. | Sep 2008 | A1 |
| 20080219260 | Du | Sep 2008 | A1 |
| 20080259555 | Bechtolsheim et al. | Oct 2008 | A1 |
| 20080275975 | Pandey et al. | Nov 2008 | A1 |
| 20080285449 | Larsson et al. | Nov 2008 | A1 |
| 20080315985 | Johnsen et al. | Dec 2008 | A1 |
| 20080317025 | Manula et al. | Dec 2008 | A1 |
| 20080320117 | Johnsen et al. | Dec 2008 | A1 |
| 20090037585 | Miloushev et al. | Feb 2009 | A1 |
| 20090037607 | Farinacci | Feb 2009 | A1 |
| 20090052345 | Brown et al. | Feb 2009 | A1 |
| 20090070775 | Riley | Mar 2009 | A1 |
| 20090074414 | Miles et al. | Mar 2009 | A1 |
| 20090129775 | Handelman | May 2009 | A1 |
| 20090161692 | Hirata et al. | Jun 2009 | A1 |
| 20090214208 | Beshai | Aug 2009 | A1 |
| 20090279701 | Moisand | Nov 2009 | A1 |
| 20090300608 | Ferris et al. | Dec 2009 | A1 |
| 20090323706 | Germain | Dec 2009 | A1 |
| 20100017497 | Brown et al. | Jan 2010 | A1 |
| 20100020806 | Vahdat et al. | Jan 2010 | A1 |
| 20100061240 | Sindhu et al. | Mar 2010 | A1 |
| 20100061241 | Sindhu et al. | Mar 2010 | A1 |
| 20100061242 | Sindhu et al. | Mar 2010 | A1 |
| 20100061367 | Sindhu et al. | Mar 2010 | A1 |
| 20100061389 | Sindhu et al. | Mar 2010 | A1 |
| 20100061391 | Sindhu et al. | Mar 2010 | A1 |
| 20100061394 | Sindhu et al. | Mar 2010 | A1 |
| 20100165876 | Shukla et al. | Jul 2010 | A1 |
| 20100165877 | Shukla et al. | Jul 2010 | A1 |
| 20100169467 | Shukla et al. | Jul 2010 | A1 |
| 20100189121 | Beshai | Jul 2010 | A1 |
| 20100192202 | Ker | Jul 2010 | A1 |
| 20100306408 | Greenberg et al. | Dec 2010 | A1 |
| 20110052191 | Beshai | Mar 2011 | A1 |
| 20120320795 | Shukla et al. | Dec 2012 | A1 |
| 20130003726 | Sindhu et al. | Jan 2013 | A1 |
| Number | Date | Country |
|---|---|---|
| 1098236 | Feb 1995 | CN |
| 1167417 | Dec 1997 | CN |
| 101132286 | Feb 2008 | CN |
| 1128585 | Aug 2001 | EP |
| 1 318 628 | Jun 2003 | EP |
| 1 892 905 | Feb 2008 | EP |
| 1 924 030 | May 2008 | EP |
| 2 369 782 | Sep 2011 | EP |
| 2 362 289 | Nov 2001 | GB |
| Entry |
|---|
| Extended Search Report for European Application No. 11158837.2, mailed Jun. 21, 2011. |
| Search Report for European Application No. 09170037.7, mailed Dec. 10, 2009, 7 pages. |
| Search Report for European Application No. 12185735.3, mailed Oct. 29, 2012. |
| Chinese Office Action dated Nov. 30, 2012 issued in corresponding Chinese Application No. 200910246898.X. |
| F.K. Liotopoulos et al., “A Modular, 160 Gbps ATM Switch Architecture for Multimedia Networking Support, based on a 3-Stage Clos Network” Proceedings of the International Teletraffic Congress. ITC-16. Teletraffic Engineering in a Competitive World. Edinburgh, UK, Jun. 7, 1999, Amsterdam: Elsevier, NL, vol. 3A, XP000877657 ISBN: 978-0-444-50268-1, pp. 529-538. |
| H. Jonathan Chao et al. “Matching Algorithms for Three-Stage Bufferless Clos Network Switches” IEEE Communications Magazine, Oct. 2003, pp. 46-54. |
| Jonathan S. Turner et al. “Multirate Clos Networks” IEEE Communications Magazine, Oct. 2003, pp. 1-11. |
| Office Action mailed Apr. 1, 2011 for U.S. Appl. No. 12/495,337 (17 pages). |
| Office Action mailed Dec. 13, 2011 for U.S. Appl. No. 12/495,337 (17 pages). |
| Office Action mailed May 11, 2011 for U.S. Appl. No. 12/495,344 (11 pages). |
| Office Action mailed Jan. 4, 2012 for U.S. Appl. No. 12/495,344 (13 pages). |
| Final Office Action mailed Sep. 13, 2012 for U.S. Appl. No. 12/495,344. |
| Office Action mailed Dec. 21, 2011 for U.S. Appl. No. 12/495,358 (12 pages). |
| Office Action mailed Jun. 15, 2011 for U.S. Appl. No. 12/495,361. |
| Office Action mailed Jan. 5, 2012 for U.S. Appl. No. 12/495,361. |
| Final Office Action mailed Sep. 13, 2012 for U.S. Appl. No. 12/495,361. |
| Office Action mailed Apr. 20, 2011 for U.S. Appl. No. 12/495,364. |
| Final Office Action mailed Oct. 21, 2011 for U.S. Appl. No. 12/495,364. |
| Office Action mailed Mar. 26, 2012 for U.S. Appl. No. 12/558,130. |
| Office Action mailed Jul. 17, 2012 for U.S. Appl. No. 12/495,337. |
| Office Action mailed Apr. 10, 2013 for U.S. Appl. No. 12/495,337. |
| Chinese Office Action mailed May 29, 2013 for Chinese Application No. 201110069908.4. |
| Chinese Office Action mailed Oct. 22, 2013 for Chinese Application No. 201110069908.4. |
| Chinese Office Action mailed Feb. 17, 2014 for Chinese Application No. 201110069908.4. |
| European Office Action mailed Oct. 4, 2013 for European Application No. 11158837.2. |
| Final Office Action dated Aug. 26, 2013 for U.S. Appl. No. 12/495,337. |
| Office Action dated Aug. 15, 2013 for U.S. Appl. No. 12/495,344. |
| Final Office Action dated Jan. 17, 2014 for U.S. Appl. No. 12/495,344. |
| Final Office Action dated Aug. 16, 2013 for U.S. Appl. No. 12/495,361. |
| Office Action dated Jan. 4, 2016 for U.S. Appl. No. 12/495,364. |
| Office Action dated Aug. 30, 2016 for U.S. Appl. No. 12/495,364. |
| Office Action dated Nov. 21, 2013 for U.S. Appl. No. 13/608,799. |
| Office Action dated May 11, 2015 for U.S. Appl. No. 14/621,892. |
| Final Office Action dated Jan. 21, 2016 for U.S. Appl. No. 14/621,892. |
| Office Action dated May 6, 2016 for U.S. Appl. No. 14/621,892. |
| Second Chinese Office Action dated Jul. 3, 2013 issued in corresponding Chinese Application No. 200910246898.X. |
| Final Office Action dated Jul. 24, 2014 for U.S. Appl. No. 13/608,799. |
| Office Action dated Dec. 9, 2016 for European Application No. 09170037.7. |
| Office Action dated Dec. 9, 2016 for European Application No. 12185735.3. |
| Office Action for European Application No. 09170037.7, dated Dec. 9, 2016. |
| Office Action for European Application No. 12185735.3, dated Dec. 9, 2016. |
| Number | Date | Country | |
|---|---|---|---|
| 20120069842 A1 | Mar 2012 | US |
| Number | Date | Country | |
|---|---|---|---|
| 61316719 | Mar 2010 | US | |
| 61316720 | Mar 2010 | US |
