Global VLANs for fabric switches

Information

  • Patent Grant
  • 9602430
  • Patent Number
    9,602,430
  • Date Filed
    Tuesday, August 20, 2013
    10 years ago
  • Date Issued
    Tuesday, March 21, 2017
    7 years ago
Abstract
One embodiment of the present invention provides a switch. The switch includes a virtual local area network (VLAN) configuration module. During operation, the VLAN configuration module maps local resources of the switch and/or locally coupled end device information to a global VLAN identifier, wherein the global VLAN is persistent across a fabric switch. The fabric switch is operable to accommodate a plurality of switches and operates as a single logical switch.
Description
BACKGROUND

Field


This disclosure relates to computer networking. More specifically, this disclosure relates to systems and techniques for determining a virtualized network across multiple switches.


Related Art


The exponential growth of the Internet has made it a popular delivery medium for a variety of applications running on physical and virtual devices. Such applications have brought with them an increasing demand for bandwidth. As a result, equipment vendors race to build larger and faster switches with versatile capabilities, such as network virtualization and multi-tenancy, to accommodate diverse network demands efficiently. However, the size of a switch cannot grow infinitely. It is limited by physical space, power consumption, and design complexity, to name a few factors. Furthermore, switches with higher capability are usually more complex and expensive. More importantly, because an overly large and complex system often does not provide economy of scale, simply increasing the size and capability of a switch may prove economically unviable due to the increased per-port cost.


A flexible way to improve the scalability of a switch system is to build a fabric switch. A fabric switch is a collection of individual member switches. These member switches form a single, logical switch that can have an arbitrary number of ports and an arbitrary topology. As demands grow, customers can adopt a “pay as you grow” approach to scale up the capacity of the fabric switch.


Meanwhile, layer-2 (e.g., Ethernet) switching technologies continue to evolve. More routing-like functionalities, which have traditionally been the characteristics of layer-3 (e.g., Internet Protocol or IP) networks, are migrating into layer-2. Notably, the recent development of the Transparent Interconnection of Lots of Links (TRILL) protocol allows Ethernet switches to function more like routing devices. TRILL overcomes the inherent inefficiency of the conventional spanning tree protocol, which forces layer-2 switches to be coupled in a logical spanning-tree topology to avoid looping. TRILL allows routing bridges (RBridges) to be coupled in an arbitrary topology without the risk of looping by implementing routing functions in switches and including a hop count in the TRILL header.


As Internet traffic is becoming more diverse, network virtualization is becoming progressively more important as a value proposition for network architects. In addition, the evolution of virtual computing has make multi-tenancy attractive and, consequently, placed additional requirements on the network. For example, virtual servers are being allocated to a large number of tenants while a respective tenant operating multiple virtualized networks. It is often desirable that the network infrastructure can provide a large number virtualized network to support multi-tenancy and ensure network separation among the tenants.


While a fabric switch brings many desirable features to a network, some issues remain unsolved in facilitating a large number of virtualized networks across the fabric switch.


SUMMARY

One embodiment of the present invention provides a switch. The switch includes a virtual local area network (VLAN) configuration module. During operation, the VLAN configuration module maps local resources of the switch and/or locally coupled end device information to a global VLAN identifier, wherein the global VLAN identifier is persistent across a fabric switch. The fabric switch is operable to accommodate a plurality of switches and operates as a single logical switch.


In a variation on this embodiment, the mapping between a global VLAN and the local resources and/or locally coupled end device information is local to the switch.


In a variation on this embodiment, the global VLAN is mapped to one or more of: a local port, a media access control (MAC) address, an Institute of Electrical and Electronics Engineers (IEEE) 802.1Q Service VLAN (S-VLAN) identifier, an IEEE Customer VLAN (C-VLAN) identifier, and a Virtual Private Network (VPN) identifier.


In a variation on this embodiment, the switch also includes an internal identifier module which determines an internal identifier based on the local resources and/or locally coupled end device information. The VLAN configuration module then maps the internal identifier to the global VLAN identifier.


In a variation on this embodiment, the VLAN configuration module identifies the global VLAN identifier based on one or more fields in a packet.


In a variation on this embodiment, the VLAN configuration module maps a tenant of the switch to the global VLAN identifier.


In a variation on this embodiment, the global VLAN identifier is associated with an Internet Protocol (IP) sub-network (subnet) with layer-3 routing support. The global VLAN identifier is then associated with a logical layer-3 interface.


In a further variation, the logical layer-3 interface is operable as a default gateway for locally coupled virtual machines. The logical layer-3 interface is associated with a virtual IP address and a virtual MAC address. The virtual IP address and the virtual MAC address are associated with a respective switch in the fabric switch.


In a further variation, the switch includes a routing module which creates a route between two subnets associated with two global VLAN identifiers without requiring a routing protocol. The switch also includes a forwarding module which determines an output port for a packet from a first of the two subnets to a second of the two subnets based on the route.


In a further variation, the switch also includes a separation module which maintains a virtual routing and forwarding separation comprising a subset of local routes. This virtual routing and forwarding separation is persistent across the fabric switch.


In a variation on this embodiment, the switch also includes a packet processor which adds a Transparent Interconnection of Lots of Links (TRILL) header to a packet. This TRILL header includes the global VLAN identifier.


In a variation on this embodiment, the global VLAN identifier is persistent across a plurality of fabric switches.


In a variation on this embodiment, the switch also includes a control module operable, which runs a control plane with automatic configuration capabilities based on a protocol associated with the fabric switch and forms a logical Ethernet switch based on the automatic configuration capabilities of the control plane. The control module also receives an automatically assigned identifier corresponding to the logical Ethernet switch without requiring manual configuration of the identifier and joins the fabric switch via the control plane.





BRIEF DESCRIPTION OF THE FIGURES


FIG. 1A illustrates an exemplary fabric switch with global VLAN support, in accordance with an embodiment of the present invention.



FIG. 1B illustrates an exemplary a member switch of a fabric switch with global VLAN support, in accordance with an embodiment of the present invention.



FIG. 2 presents a flowchart illustrating the process of a member switch of a fabric switch forwarding a packet based on an internal identifier and/or an internal policy identifier, in accordance with an embodiment of the present invention.



FIG. 3A presents a flowchart illustrating the process of a distributed service manager in a member switch of a fabric switch determining a global VLAN, in accordance with an embodiment of the present invention.



FIG. 3B presents a flowchart illustrating the process of a distributed service manager in a member switch of a fabric switch determining a global VLAN based on information from a virtualization manager, in accordance with an embodiment of the present invention.



FIG. 4 illustrates exemplary global VLANs spanning a plurality of fabric switches, in accordance with an embodiment of the present invention.



FIG. 5A illustrates an exemplary fabric switch with Internet Protocol (IP) support over global VLANs, in accordance with an embodiment of the present invention.



FIG. 5B illustrates an exemplary a member switch in a fabric switch with IP support over global VLANs, in accordance with an embodiment of the present invention.



FIG. 6 presents a flowchart illustrating the process a member switch of a fabric switch forwarding a packet across subnets, in accordance with an embodiment of the present invention.



FIG. 7 illustrates an exemplary switch with global VLAN support, in accordance with an embodiment of the present invention.





In the figures, like reference numerals refer to the same figure elements.


DETAILED DESCRIPTION

The following description is presented to enable any person skilled in the art to make and use the invention, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present invention. Thus, the present invention is not limited to the embodiments shown, but is to be accorded the widest scope consistent with the claims.


Overview


In embodiments of the present invention, the problem of providing large-scale network virtualization in a fabric switch is solved by creating a global virtual local area network (VLAN) across the fabric switch while localizing Institute of Electrical and Electronics Engineers (IEEE) 802.1Q VLANs in a respective member switch. In network virtualization, a large and complex network infrastructure can be carved up into multiple virtual networks (e.g., a layer-2 VLAN) to facilitate manageability. Furthermore, the network infrastructure can serve a plurality of tenants, a respective of which may require a number of VLANs. Especially in a fabric switch (e.g., an Ethernet fabric switch), a respective member switch can serve a plurality of tenants while a plurality of member switches can serve the same tenant. As a result, a fabric switch requires a large number of VLANs which are consistent across the member switches of fabric switch. However, with existing technologies, the total number of VLANs is limited and can bound the number of VLANs the fabric switch can support for a respective tenant.


To solve this problem, member switches in a fabric switch, in conjunction with each other, facilitates a large number global virtualized layer-2 networks (e.g., global VLANs) across the fabric switch. A respective member switch can use local resources (e.g., a local port) and/or information regarding a respective end device (e.g., local server or virtual machine) to map the end device to a global VLAN. Examples of such end device information include, but are not limited to, a physical or virtual device identifier (e.g., a media access control (MAC) address), an IEEE 802.1Q Service VLAN (S-VLAN) identifier and/or Customer VLAN (C-VLAN) identifier, and a Virtual Private Network (VPN) identifier. For example, a member switch can locally associate the limited number of IEEE 802.1Q VLANs with local end devices based on the availability. The member switch further associates the end devices with global VLANs, which are consistent across the fabric switch, based on the local resources and/or end device information.


In other words, a global VLAN provides fabric-wide VLAN for end devices coupled to one or more member switches. However, the association with the global VLAN is local to a member switch. As a result, the same global VLAN can be mapped to two end devices based on two different sets information in two member switches. In this way, a fabric switch can associate end devices coupled to different member switches and associated with different 802.1Q VLANs with the same global VLAN, thereby facilitating a large number of fabric-wide virtualized layer-2 networks with localized association.


In some embodiments, a global VLAN can support Internet Protocol (IP) routing. A global VLAN then can be associated with an IP sub-network (subnet) and can operate as a logical layer-3 interface assigned with an IP address from the subnet in a respective member switch. A respective member switch can maintain a mapping between the global VLAN and the corresponding subnet. In some embodiments, the layer-3 interface operates as a default gateway for the corresponding global VLAN and is assigned a virtual IP address, which is consistent in a respective member switch. Because the layer-3 interface is associated with the same virtual IP address in a respective member switch, the layer-3 interface operates as a distributed layer-3 gateway.


In some embodiments, the fabric switch is an Ethernet fabric switch. In an Ethernet fabric switch, any number of switches coupled in an arbitrary topology may logically operate as a single switch. Any new switch may join or leave the fabric switch in “plug-and-play” mode without any manual configuration. A fabric switch appears as a single logical switch to an external device. In some further embodiments, the fabric switch is a Transparent Interconnection of Lots of Links (TRILL) network and a respective member switch of the fabric switch is a TRILL routing bridge (RBridge).


Although the present disclosure is presented using examples based on the TRILL protocol, embodiments of the present invention are not limited to networks defined using TRILL, or a particular Open System Interconnection Reference Model (OSI reference model) layer. For example, embodiments of the present invention can also be applied to a multi-protocol label switching (MPLS) network. In this disclosure, the term “fabric switch” is used in a generic sense, and can refer to a network operating in any networking layer, sub-layer, or a combination of networking layers.


In this disclosure, the term “end device” can refer to a physical or virtual device coupled to a fabric switch. An end device can be a host, a server, a conventional layer-2 switch, a layer-3 router, or any other type of device. Additionally, an end device can be coupled to other switches or hosts further away from a network. An end device can also be an aggregation point for a number of network devices to enter the network. The terms “device” and “machine” are used interchangeably.


The term “hypervisor” is used in a generic sense, and can refer to any virtual machine manager. Any software, firmware, or hardware that creates and runs virtual machines can be a “hypervisor.” The term “virtual machine” also used in a generic sense and can refer to software implementation of a machine or device. Any virtual device which can execute a software program similar to a physical device can be a “virtual machine.” A host external device on which a hypervisor runs one or more virtual machines can be referred to as a “host machine.”


The term “VLAN” is used in a generic sense, and can refer to any virtualized network. Any virtualized network comprising a segment of physical networking devices, software network resources, and network functionality can be can be referred to as a “VLAN.” “VLAN” should not be interpreted as limiting embodiments of the present invention to layer-2 networks. “VLAN” can be replaced by other terminologies referring to a virtualized network or network segment, such as “Virtual Private Network (VPN),” “Virtual Private LAN Service (VPLS),” or “Easy Virtual Network (EVN).”


The term “packet” refers to a group of bits that can be transported together across a network. “Packet” should not be interpreted as limiting embodiments of the present invention to layer-3 networks. “Packet” can be replaced by other terminologies referring to a group of bits, such as “frame,” “cell,” or “datagram.”


The term “switch” is used in a generic sense, and can refer to any standalone or fabric switch operating in any network layer. “Switch” can be a physical device or software running on a computing device. “Switch” should not be interpreted as limiting embodiments of the present invention to layer-2 networks. Any device that can forward traffic to an external device or another switch can be referred to as a “switch.” Examples of a “switch” include, but are not limited to, a layer-2 switch, a layer-3 router, a TRILL RBridge, or a fabric switch comprising a plurality of similar or heterogeneous smaller physical switches.


The term “RBridge” refers to routing bridges, which are bridges implementing the TRILL protocol as described in Internet Engineering Task Force (IETF) Request for Comments (RFC) “Routing Bridges (RBridges): Base Protocol Specification,” available at http://tools.ietf.org/html/rfc6325, which is incorporated by reference herein. Embodiments of the present invention are not limited to application among RBridges. Other types of switches, routers, and forwarders can also be used.


The term “edge port” refers to a port in a fabric switch which exchanges data frames with an external device outside of the fabric switch. The term “inter-switch port” refers to a port which couples a member switch of a fabric switch with another member switch and is used for exchanging data frames between the member switches.


Network Architecture



FIG. 1A illustrates an exemplary fabric switch with global VLAN support, in accordance with an embodiment of the present invention. As illustrated in FIG. 1A, a fabric switch 100 includes member switches 101, 102, 103, 104, and 105. In some embodiments, one or more switches in fabric switch 100 can be virtual switches (e.g., a software switch running on a computing device). Switches 103 and 105 are coupled to host machines 120 and 130, respectively. Member switches in fabric switch 100 use edge ports to communicate to end devices and inter-switch ports to communicate to other member switches. For example, switch 103 is coupled to end devices, such as host machine 120, via edge ports and to switches 101, 102, and 104 via inter-switch ports. Host machines 120 and 130 include hypervisors 122 and 132, respectively. Virtual machines (VMs) 124, 126, and 128 run on hypervisor 122, and virtual machines 134, 136, and 138 run on hypervisor 132.


During operation, fabric switch 100 receives a mapping between local resources (e.g., a local port) and/or end device information, and a global VLAN. Examples of such information regarding an end device include, but are not limited to, a physical or virtual device identifier (e.g., a media access control (MAC) address), an IEEE 802.1Q Service VLAN (S-VLAN) identifier and/or Customer VLAN (C-VLAN) identifier, and a Virtual Private Network (VPN) identifier. In some embodiments, a network administrator provides the mapping to one of the member switches of fabric switch 100, which in turn, distributes the mapping to all other member switches based on an internal information distribution service of fabric switch 100.


In some embodiments, switch 102 receives end device information, such as the MAC address, of a respective virtual machine from a virtualization manager 110 coupled to switch 102. Examples of a virtualization manager include, but are not limited to, VMWare vCenter, Citrix XenCenter, and Microsoft Virtual Machine Manager. Upon receiving the end device information, switch 102 distributes the information to all other member switches based on an internal information distribution service of fabric switch 100. A respective member switch of fabric switch 100 includes a distributed service manager which manages global VLANs across fabric switch 100. Because allocation of IEEE 802.1Q VLAN is local to a member switch while global VLAN is fabric-wide, the service manager in a respective member switch determines IEEE 802.1Q VLAN allocation for local virtual machines based on local availability and policy. Fabric switch 100 then notifies virtualization manager 110 regarding the local VLAN allocation, which in turn associates the VLAN to the corresponding virtual machine.


For example, based on local availability and policy, the service manager in switch 105 determines that virtual machines 134 and 136 should be associated with VLAN 112 (denoted with dashed lines), and virtual machine 138 should be associated with VLAN 114 (denoted with dotted lines). Switch 105 notifies virtualization manager 110 via fabric switch 100 regarding this VLAN assignment information. Upon receiving this information, virtualization manager 110 associates VLANs 112 and 114 to the corresponding virtual machines. Similarly, based on information from switch 103, virtualization manager 110 associates virtual machine 124 with VLAN 112 and virtual machines 126 and 128 with VLAN 114.


Based on the received information and VLAN allocation, the service manager of a respective member switch identifies the virtual machines belonging to a layer-2 domain. For example, the service manager running on switches 103 and 105 individually determine that virtual machines 124, 126, and 136 belong to a layer-2 domain even though their local VLANs are different. Hence, the service manager associates virtual machines 124, 126, and 136 to global VLAN 142 and represents global VLAN 142 with an identifier. Similarly, the service manager associates virtual machines 128, 134, and 138 to global VLAN 144 and represents global VLAN 144 with an identifier. The distributed service manager in fabric switch 100 can generate a respective global VLAN identifier such a way that the same global VLAN identifier is generated in a respective member switch. In some embodiments, a respective member switch uses at least 20 bits to represent the identifiers of global VLANs 142 and 144. In this way, fabric switch 100 supports a large number of fabric-wide virtualized networks and overcomes the limitations of IEEE 802.1Q VLANs.


It should be noted that fabric switch 100 is not the same as conventional switch stacking. In switch stacking, multiple switches are interconnected at a common location (often within the same rack), based on a particular topology (e.g., ring or linear topology). These stacked switches typically share a common address, e.g., IP address, so they can be addressed as a single switch externally. However, the switches are manually configured to join the switch stack. Furthermore, switch stacking requires a significant amount of manual configuration of the ports and inter-switch links. The need for manual configuration prohibits switch stacking from being a viable option in building a large-scale switching system. The topology restriction imposed by switch stacking also limits the number of switches that can be stacked. This is because it is very difficult, if not impossible, to design a stack topology that allows the overall switch bandwidth to scale adequately with the number of switch units.


In contrast, fabric switch 100 can include an arbitrary number of switches with individual addresses, can be based on an arbitrary topology (e.g., a mesh topology), and does not require extensive manual configuration. The switches can reside in the same location, or be distributed over different locations. Furthermore, a respective switch operates in conjunction with each other, without requiring any master controller. These features overcome the inherent limitations of switch stacking and make it possible to build a large “switch farm” which can be treated as a single, logical switch. Due to the automatic configuration capabilities of fabric switch 100, an individual physical switch (e.g., switch 103) can dynamically join or leave fabric switch 100 without disrupting services to the rest of the network. The automatic and dynamic configurability of fabric switch 100 allows a network operator to build its switching system in a distributed and “pay-as-you-grow” fashion without sacrificing scalability. The ability to respond to changing network conditions makes fabric switch 100 an ideal solution in a virtual computing environment, where network loads often change with time.


In some embodiments, fabric switch 100 is a Transparent Interconnection of Lots of Links (TRILL) network and a respective member switch of fabric switch 100, such as switch 103, is a TRILL routing bridge (RBridge). During operation, virtual machine 124 sends a packet to virtual machine 136. Because virtual machines 124 and 136 belong to global VLAN 142, switch 103 considers this packet to be forwarded within the same layer-2 domain. Switch 103 identifies virtual machine 136 to be coupled to switch 105 (i.e., learns the MAC address of virtual machine 136 via switch 105). Switch 103 encapsulates the packet in a TRILL header, specifies switch 105 as the egress switch, and forwards the packet to switch 105. MAC address learning and Packet forwarding in a fabric switch is specified in U.S. Patent Publication No. 2011/0268125, titled “Virtual Cluster Switching,” the disclosure of which is incorporated herein in its entirety. In some embodiments, switch 103 includes the identifier of global VLAN 142 as a Virtual Private Network (VPN) identifier in the option fields of the TRILL header. Upon receiving the packet, switch 105 determines that the packet is for the local switch (i.e., switch 105) and should be forwarded in global VLAN 142, as specified in the TRILL header. Switch 105 removes the TRILL encapsulation and forwards the packet via the edge port which couples host machine 130. Hypervisor 132 in host machine 130 receives the packet and provides the packet to virtual machine 136.


Suppose that virtual machine 124, which is coupled to switch 103, migrates to host machine 130, which is coupled to switch 105, and starts running on hypervisor 132. Then the mapping for global VLAN 142 in switch 105 should support virtual machine 124. For example, switch 105's edge port 174 and VLAN 112 should be mapped to the identifier of global VLAN 142 in switch 105. If not, the IEEE 802.1Q VLAN tag mapped to the identifier of global VLAN 142 in switch 105 can be allocated to virtual machine 124 in host machine 130. In some embodiments, the service manager of switch 105 notifies hypervisor 132 to associate virtual machine 124 with the new VLAN. An error can occur if hypervisor 132 cannot configure the new VLAN for virtual machine 124. The network administrator can be notified of such potential error before virtual machine 124 migrates.


In some embodiments, a respective member switch of fabric switch 100 (e.g., switch 103) runs a control plane with automatic configuration capabilities based on Fibre Channel (FC) protocol and forms a logical Ethernet switch based on the automatic configuration capabilities of the control plane. To an external end device, such as host machine 120, fabric switch 100 appears as one, single Ethernet switch. Upon joining fabric switch 100 via the control plane, a respective member switch receives an automatically assigned identifier corresponding to the logical Ethernet switch without requiring manual configuration. However, unlike an FC fabric, the data packets in fabric switch 100 can be encapsulated and forwarded based on another forwarding protocol. Examples of this forwarding protocol include, but are not limited to, Ethernet, TRILL, and IP. These features allow switch 103 to operate in conjunction with other member switches of fabric switch 100 in a distributed way, without requiring a central controller.


In some embodiments, fabric switch 100 maintains a port profile for a respective virtual machine. A port profile represents Fibre Channel over Ethernet (FCoE) configuration, VLAN configuration, data center bridging (DCB) configuration, quality of service (QoS) configuration, and/or security configuration of one or more virtual machines. The MAC address of a virtual machine associates with the corresponding port profile to the virtual machine. The VLAN configuration in a port profile can indicate the global VLAN configuration for the virtual machine. Port profile management in a switch is specified in U.S. Patent Publication No. 2011/0299413, titled “Port profile management for virtual cluster switching,” the disclosure of which is incorporated herein in its entirety.


A respective member switch, such as switch 103, locally maintains global VLAN information to facilitate its fabric-wide deployment. FIG. 1B illustrates an exemplary a member switch of a fabric switch with global VLAN support, in accordance with an embodiment of the present invention. In this example, switch 103 is also coupled to a host machine 160, which includes virtual machines 164, 166, and 168 running on hypervisor 162. During operation, based on information from switch 103, virtualization manager 110 associates virtual machines 164 and 166 with VLAN 112 and virtual machine 168 with VLAN 114. Service manager 180 running on switch 103 determines that virtual machines 164 and 168 belong to the same layer-2 domain of virtual machine 128 even though their local VLANs are different. Hence, service manager 180 associates virtual machines 164 and 168 with global VLAN 144. Similarly, service manager 180 associates virtual machine 166 with global VLAN 142.


The association between a virtual machine and a global VLAN is maintained at switch 103. It should be noted that any end device outside of fabric switch 100 can be agnostic to global VLANs 142 and 144. For example, hypervisor 122 and virtual machine 128 can be agnostic to the association between global VLAN 144 and virtual machine 128. To maintain the association, switch 103 maintains a mapping 150 between global VLAN 142, and corresponding local resources and end device information of virtual machines 124 and 126. For example, switch 103 can map the MAC addresses of virtual machines 124 and 126, and port 172 to the identifier of global VLAN 142. Similarly, switch 103 can map the port and MAC address of virtual machine 128 to the identifier of global VLAN 144 in mapping 150. In this way, switch 103 localizes IEEE 802.1Q VLANs to switch 103 while maintaining a fabric-wide layer-2 virtualized network in conjunction with other member switches of fabric switch 100.


In some embodiments, global VLANs 142 and 144 can represent tenant separation. For example, virtual machines 124, 126, and 166 can belong to one tenant while virtual machines 128, 164, and 168 can belong to another tenant. Because global VLAN 142 and 144 provides the fabric-wide virtual separation for the tenants, and IEEE 802.1Q VLAN 112 and 114 are local, the full set of IEEE 802.1Q VLANs is available for deployment to a respective tenant. For example, VLANs 112 and 114 are available for deployment to both tenants. In some embodiments, a respective member switch can maintain a mapping between a tenant and one or more global VLANs associated with the tenant.


In some embodiments, switch 103 maintains two configuration tables that describe its instance: a fabric switch configuration database and a default switch configuration table. The fabric switch configuration database describes the configuration of fabric switch 100 when switch 103 is part of fabric switch 100. The default switch configuration table describes switch 103's default configuration. In some embodiments, the fabric switch configuration database includes an identifier of fabric switch 100. In one embodiment, switch 103 also maintains a switch index within fabric switch 100. This switch index is unique and persistent within fabric switch 100. That is, when switch 103 joins fabric switch 100 for the first time, fabric switch 100 assigns the switch index to switch 103. This switch index persists with switch 103, even if switch 103 leaves fabric switch 100. When switch 103 joins fabric switch 100 again at a later time, the same switch index is used by fabric switch 100 to retrieve previous configuration information for switch 103.


Internal Identifier


In the example in FIG. 1B, switch 103 maps local resources and/or end device information to a global VLAN identifier. In some embodiments, switch 103 maps the local resources and/or end device information to a local internal identifier, and maps the internal identifier to a global VLAN. In this way, switch 103 can separate the local mapping and can use this local mapping for more efficient forwarding. Mapping 150 in switch 103 can include a mapping between internal identifiers and local resources and/or end device information. For example, switch 103 can map the MAC addresses of locally coupled virtual machines 124 and 126, and port 172 to internal identifier 152. Similarly, switch 103 can maintain a mapping between the port and MAC address of virtual machine 128 and internal identifier 154. Switch 103 maps internal identifiers 152 and 154 to the identifier of global VLANs 142 and 144, respectively. Local internal identifier management and its operations in a switch are specified in U.S. Patent Publication No. 2011/0299533, titled “Internal virtual network identifier and internal policy identifier,” the disclosure of which is incorporated herein in its entirety.


In some embodiments, fabric switch 100 includes one or more overlay VLANs. A gateway device can facilitate bridging between a virtual network instance (VNI) of an overlay VLAN and a global VLAN. For example, switch 103 can operate as a gateway and provide the bridging between global VLANs 142 and 144, and an overlay VNI. Switch 103 can maintain a mapping between internal identifiers 152 and 154, and corresponding overlay VNIs. Because switch 103 maps the identifiers of global VLANs 142 and 144 to internal identifiers 152 and 154, respectively, switch 103 can, in turn, obtain the association between global VLANs 142 and 144, and the corresponding overlay VNIs. It should be noted that, unlike of regular overlay VLANs, fabric switch 100 is aware of a respective source MAC address of a respective VLAN.


In some embodiments, switch 103 uses internal identifier to efficiently forward traffic. Switch 103 can maintain a forwarding table which indicates an output port for a corresponding internal identifier and forward a respective packet based on the forwarding table. FIG. 2 presents a flowchart illustrating the process of a member switch of a fabric switch forwarding a packet based on an internal identifier and/or an internal policy identifier, in accordance with an embodiment of the present invention. Upon receiving a packet via a local port (operation 204), the switch determines an internal identifier for the packet based on the local port and/or one or more fields in the packet's header (operation 204). Examples of the fields in the packet's header include, but are not limited to, a customer VLAN identifier, a service provider VLAN identifier, and a source MAC address.


In some embodiments, the switch also generates an internal policy identifier for the packet. This policy identifier indicates forwarding and quality of service policies for the packet. The switch determines the internal identifier for the packet based on the local port and/or one or more fields in the packet's header (operation 206). The switch can maintain a mapping between the internal identifier and/or internal policy identifier, and the port and the header fields. Upon receiving the packet, the switch can consult the mapping to determine the internal identifier and/or internal policy identifier. The switch then obtains the global VLAN identifier corresponding to the internal identifier (operation 208). The switch can consult the mapping to determine the global VLAN identifier corresponding to the internal identifier.


In some embodiments, the switch is a TRILL RBridge. The switch encapsulates the packet in a TRILL header (operation 210), as described in conjunction with FIG. 1A. It should be noted that the TRILL header includes an identifier to the egress switch, which can be an RBridge identifier. The switch includes the obtained global VLAN identifier in the TRILL header (operation 212) and determines an output port for the packet based on the internal identifier and/or internal policy identifier (operation 214). In some embodiments, the switch consults a forwarding table to determine the output port. The forwarding table can include a mapping between the internal identifier and/or internal policy identifier, and the corresponding output port.


Global VLAN


In the example in FIG. 1A, a distributed service manager in a respective member switch determines the global VLANs for fabric switch 100. FIG. 3A presents a flowchart illustrating the process of a distributed service manager in a member switch of a fabric switch determining a global VLAN, in accordance with an embodiment of the present invention. During operation, the service manager determines a global VLAN identifier associated with the fabric switch (operation 302). In some embodiments, the service manager determines the global VLAN identifier by receiving a mapping between a global VLAN identifier and corresponding local resources and/or end device information. In some embodiments, the service manager uses at least 20 bits to represent the global VLAN identifier. In some embodiments, the service manager identifies an internal identifier associated with the global VLAN (operation 304), as described in conjunction with FIG. 1B. The service manager maps the global VLAN identifier to the internal identifier (operation 306). Because the internal identifier corresponds to the local resources and/or end device information, mapping the global VLAN identifier to the internal identifier allows the service manager to associate the global VLAN identifier to the corresponding local resources and/or end device information.


In some further embodiments, a virtualization manager provides end device information, such as the MAC address, for a respective virtual machine associated with a fabric switch. FIG. 3B presents a flowchart illustrating the process of a distributed service manager in a member switch of a fabric switch determining a global VLAN based on information from a virtualization manager, in accordance with an embodiment of the present invention. During operation, the service manager obtains the information of virtual machines associated with the fabric switch (operation 352). The service manager then identifies one or more virtual machines operable to be in the same layer-2 domain (operation 354).


In some embodiments, the service manager determines IEEE 802.1Q VLAN allocation for the local virtual machines based on local availability and policy (operation 356), and notifies the local VLAN allocation to the virtualization manager via the fabric switch (operation 358). The service manager determines a local internal identifier which corresponds to the identified virtual machines (operation 360), as described in conjunction with FIG. 2. The service manager also determines a global VLAN identifier for the identified virtual machines (operation 362), associating the identified virtual machines with a layer-2 domain. In some embodiments, the service manager generates the corresponding global VLAN identifier based on the end device information received from the virtualization manager. The service manager then maps the global VLAN identifier to the internal identifier (operation 364), as described in conjunction with FIG. 1B.


Global VLAN Across Multiple Fabric Switches


In some embodiments, a global VLAN can be deployed across a plurality of fabric switches. A network operator can deploy multiple fabric switches, often within the same date center, and serve the same tenants via these multiple switches. These tenants may require virtualized layer-2 domains across a plurality of fabric switches. FIG. 4 illustrates exemplary global VLANs spanning a plurality of fabric switches, in accordance with an embodiment of the present invention. In this example, fabric switch 100 is coupled to another fabric switch 400, which includes member switches 401, 402, 403, 404, and 405. In some embodiments, one or more switches in fabric switch 400 can be a virtual switch (e.g., a software switch operating in a computing device). Switch 405 is coupled to host machine 430. Virtual machines 434, 436, and 438 run on hypervisor 432 in host machine 430. Based on IEEE 802.1Q VLAN allocation from fabric switch 400, virtual machines 434 and 436 are associated with VLAN 112 and virtual machine 438 is associated with VLAN 114, as described in conjunction with FIG. 1A.


In some embodiments, fabrics switches 100 and 400 are coupled to each other via multi-homed connections between edge ports of switches 102 and 105 in fabrics switch 100 and edge ports of switches 401 and 403 in fabric switch 400. Fabric switches 100 and 400 can also be coupled to each other via one or more tunnels. Examples of such a tunnel include, but are not limited to, Virtual Extensible Local Area Network (VXLAN), Generic Routing Encapsulation (GRE), and its variations, such as Network Virtualization using GRE (NVGRE) and openvSwitch GRE.


A respective member switch of fabric switch 400 includes a distributed service manager which manages global VLANs across fabric switch 400. During operation, the service manager of a respective member switch identifies the virtual machines belonging to a layer-2 domain based on received end device information from virtualization manager 110. For example, the service manager running on switches 103 and 405 individually determine that virtual machines 124, 126, and 436 belong to a layer-2 domain even though their local VLANs are different. Because the service manager runs in a distributed way in fabric switches 100 and 400, the service manager in both determines the same global VLAN for the virtual machines belonging to the same layer-2 domain. The distributed service manager in fabric switches 100 and 400 can generate a global VLAN identifier such a way that the same identifier is generated in a respective member switch in a respective fabric switch.


For example, the service managers in fabric switches 100 and 400 associate virtual machines 124, 126, and 436 to global VLAN 142 and represents global VLAN 142 with an identifier. Similarly, the service managers associate virtual machines 128, 134, and 438 to global VLAN 144 and represents global VLAN 144 with an identifier. It should be noted that the same identifier is allocated to global VLAN 142 (or global VLAN 144) in both fabric switches 100 and 400. In some embodiments, a respective distributed service manager in a fabric switch uses a formula to determine the global VLAN identifier.


In some embodiments, fabric switches 100 and 400 are TRILL networks and a respective member switch of fabric switches 100 and 400, such as switches 103 and 405, are TRILL RBridges. Suppose that virtual machine 124 sends a packet to virtual machine 436. Because these virtual machines belong to global VLAN 142, switch 103 considers this packet to be forwarded within the same layer-2 domain. Because virtual machine 436 is in the same layer-2 domain, switch 103 broadcasts an Address Resolution Protocol (ARP) request in global VLAN 142 to obtain the MAC address of virtual 436 and determines that virtual machine 436 is coupled via an edge port of switch 105. Switch 103 encapsulates the packet in a TRILL header and forwards the frame to switch 105, which couples fabric switch 400. In some embodiments, switch 103 includes the identifier of global VLAN 142 as a VPN identifier in the option fields of the TRILL header. Upon receiving the packet, switch 105 determines that destination virtual machine 436 is coupled via an edge port and is in global VLAN 442, as specified in the TRILL header. Switch 105 removes the TRILL encapsulation and forwards the packet via the edge port which couples switch 403 of fabric switch 400.


Because fabric switch 100 allows a tenant to use the internal IEEE 802.1Q VLAN tag, switch 105 uses double-tagging (i.e., IEEE 802.1Q in IEEE 802.1Q VLAN) for the packet while forwarding the packet to fabric switch 400. The outer service tag (S-TAG) represents the service provider's network while the inner customer tag (C-tag) represents a client's network. To represent global VLAN 142 to fabric switch 400, switch 105 uses a formula on the identifier of global VLAN 142 to determine the C-tag and S-tag for the packet. In some embodiments, switch 105 uses the formula ((identifier of global VLAN 142)/4092+1) to determine the S-tag and ((identifier of global VLAN 142) % 4092+1) to determine the C-tag.


Upon receiving the packet, switch 403 determines the identifier of global VLAN 142 from the S-tag and C-tag of the received packet and identifies virtual machine 436 to be locally coupled to switch 405 (i.e., learns the MAC address of virtual machine 436 via switch 405). Switch 403 encapsulates the packet in a TRILL header, includes an identifier of global VLAN 142 in the TRILL header, and forwards the frame to switch 405. In some embodiments, switch 403 includes the identifier of global VLAN 142 as a VPN identifier in the option fields of the TRILL header. Upon receiving the packet, switch 405 determines that the packet is for global VLAN 442, as specified in the TRILL header. Switch 405 removes the TRILL encapsulation and forwards the packet to host machine 430. Hypervisor 432 receives the packet and provides the packet to virtual machine 436.


IP Over Global VLAN


In some embodiments, a global VLAN can support IP routing. FIG. 5A illustrates an exemplary fabric switch with IP support over global VLANs, in accordance with an embodiment of the present invention. In this example, global VLANs 142 and 144 can support IP routing and are associated with IP subnets 542 and 544, respectively. Consequently, global VLANs 142 and 144 can operate as logical layer-3 interfaces assigned with an IP address from the corresponding subnet in a respective member switch in fabric switch 100. A respective member switch in fabric switch 100 can maintain a mapping between global VLANs 142 and 144, and corresponding subnets 542 and 544, respectively. Furthermore, virtual machines 124, 126, and 136 are assigned IP addresses from subnet 542, and virtual machines 128, 134, and 136 are assigned IP addresses from subnet 544.


In some embodiments, the corresponding layer-3 interfaces operate as default gateways for global VLANs 142 and 144. To operate as a default gateway, a respective layer-3 interface is assigned the same virtual IP address and a virtual MAC address in a respective member switch. For example, the same virtual IP address from subnet 542 is assigned to the layer-3 interface of global VLAN 142 in a respective member switch. As a result, the layer-3 interface operates as a distributed layer-3 gateway for global VLAN 142. In this way, the same virtual IP address of subnet 542 is configured as the default gateway address of virtual machines 124, 126, and 136 even though they are coupled to different member switches of fabric switch 100. Similarly, the same virtual IP address of subnet 544 is configured as the default gateway address of virtual machines 128, 134, and 136.


Because a respective member switch in fabric switch 100 maintains a mapping between the identifier of global VLAN 142 and subnet 542, the logical layer-3 interface of subnet 542 corresponds to global VLAN 142. Similarly, the logical layer-3 interface of subnet 544 corresponds to global VLAN 144. Consequently, a member switch, such as switch 103, can have a connected route between subnets 542 and 544 (i.e., switch 103 can route between subnets 542 and 544 without requiring a routing protocol).


During operation, virtual machine 124 sends a packet to virtual machine 134. Because these virtual machines belong to separate subnets (i.e., subnets 542 and 544, respectively), this packet requires routing between subnets 542 and 544. Virtual machine 124 is configured with the virtual IP address of the layer-3 interface of global VLAN 142 as the default gateway. If virtual machine 124 has not learned the corresponding virtual MAC address, virtual machine 124 sends an ARP query using the virtual IP address. Because a respective member switch in fabric switch 100 is associated with the virtual IP address, upon receiving the query, switch 103 responds with the corresponding virtual MAC address. Virtual machine 124 then forwards the packet using the virtual MAC address. Furthermore, a respective member switch in fabric switch 100 is associated with the virtual MAC address. As a result, switch 103 receives the packet, considers the layer-2 destination of the packet to be the local switch (i.e., switch 103), and promotes the packet to layer-3.


Switch 103 has logical layer-3 interfaces to both subnets 542 and 544. Consequently, switch 103 can perform connected routing between these subnets. Switch 103 determines that virtual machine 134 belongs to subnet 544, and hence, is associated with global VLAN 144. Because global VLAN 144 is local to switch 103, switch 103 can forward the packet to virtual machine 134 via global VLAN 144, as described in conjunction with FIG. 1A. If switch 103 has not learned the MAC address of virtual machine 134, switch 103 can use ARP to send a layer-2 broadcast message in global VLAN 144 for the IP address of virtual machine and obtain the MAC address of virtual machine 134.


A respective member switch in fabric switch 100 can have separate virtual routing and forwarding (VRFs) for a respective service entity (e.g., tenants, departments of the same corporation, etc.). A VRF manages routes only for a corresponding service entity. This allows the member switches to have routing separation among the service entities. In this way, a VRF provides layer-3 virtualization in the member switch. In some embodiments, such a VRF can be global and persistent in fabric switch 100. These global VRFs provide the same routing separation among the service entities in a respective member switch.



FIG. 5B illustrates an exemplary a member switch in a fabric switch with IP support over global VLANs, in accordance with an embodiment of the present invention. In this example, switch 103 serves two service entities requiring logical separation and maintains two global VRFs (GVRFs) 522 and 524. Suppose that global VLANs 142 and 144 belong to the two service entities, respectively. Global VRFs 522 and 524 are then associated with global VLANs 142 and 144, respectively. Global VRFs 522 and 524 allows subnets 542 and 544 to have overlapping IP addresses. Switch 103 can include respective identifiers of global VRFs 522 and 524 in corresponding packets to distinguish between subnets 542 and 544. For example, if subnets 542 and 544 both include an IP address, the identifier of global VRF 522 or 524 along with the IP address indicates to which service entity a packet with the IP address belongs. This provides a respective service entity flexibility and layer-3 virtualization across fabric switch 100.



FIG. 6 presents a flowchart illustrating the process a member switch of a fabric switch forwarding a packet across subnets, in accordance with an embodiment of the present invention. Upon receiving a packet (operation 602), the switch checks whether the destination subnet is local (operation 604). In some embodiments, the switch checks whether the switch has an interface to the subnet to determine whether the destination subnet is local. If not, the switch forwards the packet to the next-hop subnet toward the destination subnet based on a local routing table (operation 620). If the destination subnet is local, the switch identifies the global VLAN identifier corresponding to the local destination subnet (operation 606).


The switch then checks whether the destination MAC address of the packet is known (operation 608). If the destination MAC address of the packet is not known, the switch sends a broadcast ARP query message to global VLAN corresponding to the destination subnet (operation 614) and obtains the destination MAC address of the packet via the ARP response of the query message (operation 616). If the destination MAC address of the packet is known (operation 608) or the switch has obtained the destination MAC address of the packet (operation 616), the switch encapsulates the packet in a TRILL header and includes the global VLAN identifier in the TRILL header (operation 610). The switch then forwards the packet toward the destination MAC address (operation 612), as described in conjunction with FIG. 1A. It should be noted that such forwarding in a global VLAN can be within or across fabric switch boundary.


Exemplary Switch



FIG. 7 illustrates an exemplary switch with global VLAN support, in accordance with an embodiment of the present invention. In this example, a switch 700 includes a number of communication ports 702, a packet processor 710, a VLAN configuration module 740, and a storage 750. One or more of these modules can be included in a distributed service manager in switch 700. In some embodiments, packet processor 710 adds a TRILL header to a packet. In some embodiments, switch 700 includes a fabric switch management module 724, which maintains a membership in a fabric switch. Switch 700 maintains a configuration database in storage 750 that maintains the configuration state of a respective switch within the fabric switch. Switch 700 maintains the state of the fabric switch, which is used to join other switches. Under such a scenario, communication ports 702 can include inter-switch communication channels for communication within a fabric switch. This inter-switch communication channel can be implemented via a regular communication port and based on any open or proprietary format (e.g., TRILL protocol).


During operation, VLAN configuration module 740 maps local resources of the switch (e.g., one of communication ports 702) and/or locally coupled end device information to a global VLAN identifier. This mapping between the global VLAN and the local resources and/or locally coupled end device information is local to switch 700, as described in conjunction with FIG. 1A. In some embodiments, switch 700 also includes an internal identifier module 722 which determines an internal identifier based on the local resources and/or locally coupled end device information. VLAN configuration 740 module then maps the internal identifier to the global VLAN identifier, as described in conjunction with FIG. 1B.


When switch 700 receives a packet from another member switch of the fabric switch via one of communication ports 702, packet processor 710 examines the TRILL header and identifies a global VLAN identifier of the packet. However, if the packet if received from another fabric switch via one of communication ports 702, packet processor 710 examines the packet headers. Based on this examination, VLAN configuration module identifies the global VLAN identifier based on one or more fields in the packet.


In some embodiments, the global VLAN identifier in switch 700 is associated with an IP subnet with layer-3 routing support and with a logical layer-3 interface. This logical layer-3 interface is operable as a default gateway for the virtual machines locally coupled to switch 700. Under such a scenario, the logical layer-3 interface is associated with a virtual IP address and a virtual MAC address. The virtual IP address and the virtual MAC address are associated with switch 700 and other member switches of the fabric switch.


In some embodiments, switch 700 also includes a routing and forwarding module 730, which creates a route between two subnets associated with two global VLAN identifiers without requiring a routing protocol, as described in conjunction with FIGS. 5A and 5B. Routing and forwarding module 730 determines an output port for a packet from a first of the two subnets to a second of the two subnets based on the route. In some embodiments, switch 700 also includes a separation module 732, which maintains one or more virtual routing and forwarding separations, each comprising a subset of local routes. This virtual routing and forwarding separation is persistent across the fabric switch.


Note that the above-mentioned modules can be implemented in hardware as well as in software. In one embodiment, these modules can be embodied in computer-executable instructions stored in a memory which is coupled to one or more processors in switch 700. When executed, these instructions cause the processor(s) to perform the aforementioned functions.


In summary, embodiments of the present invention provide a switch and a method for providing a global VLAN across a plurality of switches. In one embodiment, the switch includes a VLAN configuration module. During operation, the VLAN configuration module maps local resources of the switch and/or locally coupled end device information to a global VLAN identifier, wherein the global VLAN is persistent across a fabric switch. The fabric switch is operable to accommodate a plurality of switches and operates as a single logical switch.


The methods and processes described herein can be embodied as code and/or data, which can be stored in a computer-readable non-transitory storage medium. When a computer system reads and executes the code and/or data stored on the computer-readable non-transitory storage medium, the computer system performs the methods and processes embodied as data structures and code and stored within the medium.


The methods and processes described herein can be executed by and/or included in hardware modules or apparatus. These modules or apparatus may include, but are not limited to, an application-specific integrated circuit (ASIC) chip, a field-programmable gate array (FPGA), a dedicated or shared processor that executes a particular software module or a piece of code at a particular time, and/or other programmable-logic devices now known or later developed. When the hardware modules or apparatus are activated, they perform the methods and processes included within them.


The foregoing descriptions of embodiments of the present invention have been presented only for purposes of illustration and description. They are not intended to be exhaustive or to limit this disclosure. Accordingly, many modifications and variations will be apparent to practitioners skilled in the art. The scope of the present invention is defined by the appended claims.

Claims
  • 1. A switch configured to be a member of a network of interconnected switches, the switch comprising: virtual local area network (VLAN) configuration circuitry configured to: in response to identifying a new locally reachable end device, determine a customer VLAN for the end device based on currently allocated customer VLANs at the switch and local resources of the switch, wherein the determination of the customer VLAN is local to the switch;determine a layer-2 forwarding domain within the network of interconnected switches for the end device, wherein the layer-2 forwarding domain includes a plurality of customer VLANs; andmap the customer VLAN to a global VLAN identifier identifying the layer 2 forwarding domain in the network of interconnected switches;wherein the global VLAN identifier is persistent across the network of interconnected switches; andwherein the network of interconnected switches is identified based on a fabric identifier.
  • 2. The switch of claim 1, wherein scope of the mapping between the global VLAN and the customer VLAN is local within the switch.
  • 3. The switch of claim 1, wherein the global VLAN is further mapped to one or more of: a local port associated with the end device;a media access control (MAC) address of the end device; anda Virtual Private Network (VPN) identifier; and
  • 4. The switch of claim 1, further comprising: internal identifier circuitry configured to determine an internal identifier based on the local resources and/or information associated with the end device, wherein scope of the internal identifier is internal and local within the switch; andwherein the VLAN configuration circuitry is further configured to map the internal identifier to the global VLAN identifier.
  • 5. The switch of claim 1, wherein the VLAN configuration circuitry is further configured to identify the global VLAN identifier from the mapping between the global VLAN and the customer VLAN based on one or more fields in a packet.
  • 6. The switch of claim 1, wherein the VLAN configuration circuitry is further configured to map the global VLAN identifier further to a tenant of the switch.
  • 7. The switch of claim 1, wherein the global VLAN identifier is associated with an Internet Protocol (IP) sub-network (subnet) with layer-3 routing support; and wherein the global VLAN identifier is associated with a logical layer-3 interface.
  • 8. The switch of claim 7, wherein the logical layer-3 interface is associated with a virtual IP address and a virtual MAC address; andwherein the virtual IP address and the virtual MAC address are assigned to a respective switch in the network of interconnected switches.
  • 9. The switch of claim 7, further comprising: routing circuitry configured to determine a route between a first subnet associated with a first global VLAN identifier and a second subnet associated with a second global VLAN identifier without requiring a routing protocol; andforwarding circuitry configured to determine an output port for a packet from the first subnet destined to the second subnet based on the route.
  • 10. The switch of claim 7, further comprising separation circuitry configured to maintain a virtual routing and forwarding (VRF) instance comprising a subset of routes maintained by the switch, wherein the VRF instance is persistent across the network of interconnected switches.
  • 11. The switch of claim 1, further comprising a packet processor configured to encapsulate a layer-2 frame with one or more of: a Transparent Interconnection of Lots of Links (TRILL) header, wherein the TRILL header includes the global VLAN identifier; andan IP-based encapsulation header, wherein the encapsulation header includes the global VLAN identifier.
  • 12. The switch of claim 1, wherein the global VLAN identifier is persistent across a plurality of networks of interconnected switches.
  • 13. A computer-executable method, comprising: in response to identifying a locally reachable end device from a switch, determining a customer VLAN for the end device based on currently allocated customer VLANs at the switch and local resources of the switch, wherein the determination of the customer VLAN is local to the switch, and wherein the switch is configured to be a member of a network of interconnected switches;determining a layer-2 forwarding domain for the end device within the network of interconnected switches for the end device, wherein the layer-2 forwarding domain includes a plurality of customer VLANs; andmapping the customer VLAN to a global VLAN identifier identifying the layer-2 forwarding domain in the network of interconnected switches, wherein the global VLAN identifier is persistent across the network of interconnected switches; andwherein the network of interconnected switches is identified based on a fabric identifier.
  • 14. The method of claim 13, wherein scope of the mapping between the global VLAN and the customer VLAN is local within the switch.
  • 15. The method of claim 13, wherein the global VLAN is further mapped to one or more of: a local port, which is associated with the end device, of the switch;a media access control (MAC) address of the end device; anda Virtual Private Network (VPN) identifier; and
  • 16. The method of claim 13, further comprising: determining an internal identifier based on the local resources and/or information associated with the end device, wherein scope of the internal identifier is internal and local within the switch; andmapping the internal identifier to the global VLAN identifier.
  • 17. The method of claim 13, further comprising identifying the global VLAN identifier from the mapping between the global VLAN and the customer VLAN based on one or more fields in a packet.
  • 18. The method of claim 13, further comprising mapping the global VLAN identifier further to a tenant of the switch.
  • 19. The method of claim 13, wherein the global VLAN identifier is associated with an Internet Protocol (IP) sub-network (subnet) with layer-3 routing support; and wherein the global VLAN identifier is associated with a logical layer-3 interface.
  • 20. The method of claim 19, wherein the logical layer-3 interface is associated with a virtual IP address and a virtual MAC address; andwherein the virtual IP address and the virtual MAC address are assigned to a respective switch in the network of interconnected switches.
  • 21. The method of claim 19, further comprising: determining a route between a first subnet associated with a first global VLAN identifier and a second subnet associated with a second global VLAN identifier without requiring a routing protocol; anddetermining an output port for a packet from the first subnet destined to the second subnet based on the route.
  • 22. The method of claim 19, further comprising maintaining a virtual routing and forwarding (VRF) instance comprising a subset of routes maintained by the switch, wherein the VRF instance is persistent across the network of interconnected switches.
  • 23. The method of claim 13, further comprising encapsulating a layer-2 frame with one or more of: a Transparent Interconnection of Lots of Links (TRILL) header, wherein the TRILL header includes the global VLAN identifier; andan IP-based encapsulation header, wherein the encapsulation header includes the global VLAN identifier.
  • 24. The method of claim 13, wherein the global VLAN identifier is persistent across a plurality of networks of interconnected switches.
  • 25. A computing system, comprising: a processor; anda memory storing instructions that when executed by the processor cause the system to perform a method, the method comprising: in response to identifying a locally reachable end device from a switch, determining a customer VLAN for the end device based on currently allocated customer VLANs at the switch and local resources of the switch, wherein the determination of the customer VLAN is local to the switch, and wherein the switch is configured to be a member of a network of interconnected switches;determining a layer-2 forwarding domain for the end device within the network of interconnected switches for the end device, wherein the layer-2 forwarding domain includes a plurality of customer VLANs; andmapping the customer VLAN to a global VLAN identifier identifying the layer-2 forwarding domain in the network of interconnected switches, wherein the global VLAN identifier is persistent across the network of interconnected switches; andwherein the network of interconnected switches is identified based on a fabric identifier.
  • 26. The computing system of claim 25, wherein scope of the mapping between the global VLAN and the customer VLAN is local within the switch.
  • 27. The computing system of claim 25, wherein the global VLAN is further mapped to one or more of: a local port, which is associated with the end device, of the switch;a media access control (MAC) address of the end device; anda Virtual Private Network (VPN) identifier; and
  • 28. The computing system of claim 25, wherein the method further comprises: determining an internal identifier based on the local resources and/or information associated with the end device, wherein scope of the internal identifier is internal and local within the switch; andmapping the internal identifier to the global VLAN identifier.
  • 29. The computing system of claim 25, wherein the method further comprises identifying the global VLAN identifier from the mapping between the global VLAN and the customer VLAN based on one or more fields in a packet.
  • 30. The computing system of claim 25, wherein the method further comprises mapping the global VLAN identifier further to a tenant of the switch.
  • 31. The computing system of claim 25, wherein the global VLAN identifier is associated with an Internet Protocol (IP) sub-network (subnet) with layer-3 routing support; and wherein the global VLAN identifier is associated with a logical layer-3 interface.
  • 32. The computing system of claim 31, wherein the logical layer-3 interface is associated with a virtual IP address and a virtual MAC address; andwherein the virtual IP address and the virtual MAC address are associated with a respective switch in the network of interconnected switches.
  • 33. The computing system of claim 31, wherein the method further comprises: determining a route between a first subnet associated with a first global VLAN identifier and a second subnet associated with a second global VLAN identifier without requiring a routing protocol; anddetermining an output port for a packet from the first subnet destined to the second subnet based on the route.
  • 34. The computing system of claim 31, wherein the method further comprises maintaining a virtual routing and forwarding (VRF) instance comprising a subset of routes maintained by the switch, wherein the VRF instance is persistent across the network of interconnected switches.
  • 35. The computing system of claim 25, wherein the method further comprises encapsulating a layer-2 frame with one or more of: a Transparent Interconnection of Lots of Links (TRILL) header, wherein the TRILL header includes the global VLAN identifier; andan IP-based encapsulation header, wherein the encapsulation header includes the global VLAN identifier.
  • 36. The computing system of claim 25, wherein the global VLAN identifier is persistent across a plurality of networks of interconnected switches.
RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No. 61/691,723, titled “Global VLANs for Fabric Switch,” by inventors Suresh Vobbilisetty, Phanidhar Koganti, and Chi Chong, filed 21 Aug. 2012, the disclosure of which is incorporated by reference herein. The present disclosure is related to U.S. patent application Ser. No. 13/087,239, titled “Virtual Cluster Switching,” by inventors Suresh Vobbilisetty and Dilip Chatwani, filed 14 Apr. 2011, and to U.S. patent application Ser. No. 13/092,752, titled “Name Services for Virtual Cluster Switching,” by inventors Suresh Vobbilisetty, Phanidhar Koganti, and Jesse B. Willeke, filed 22 Apr. 2011, the disclosures of which are incorporated by reference herein.

US Referenced Citations (411)
Number Name Date Kind
829529 Keathley Aug 1906 A
5390173 Spinney Feb 1995 A
5802278 Isfeld Sep 1998 A
5878232 Marimuthu Mar 1999 A
5959968 Chin Sep 1999 A
5973278 Wehrill, III Oct 1999 A
5983278 Chong Nov 1999 A
6041042 Bussiere Mar 2000 A
6085238 Yuasa Jul 2000 A
6104696 Kadambi Aug 2000 A
6185214 Schwartz Feb 2001 B1
6185241 Sun Feb 2001 B1
6331983 Haggerty Dec 2001 B1
6438106 Pillar Aug 2002 B1
6498781 Bass Dec 2002 B1
6542266 Phillips Apr 2003 B1
6633761 Singhal Oct 2003 B1
6771610 Seaman Aug 2004 B1
6873602 Ambe Mar 2005 B1
6937576 DiBenedetto Aug 2005 B1
6956824 Mark Oct 2005 B2
6957269 Williams Oct 2005 B2
6975581 Medina Dec 2005 B1
6975864 Singhal Dec 2005 B2
7016352 Chow Mar 2006 B1
7061877 Gummalla Jun 2006 B1
7173934 Lapuh Feb 2007 B2
7197308 Singhal Mar 2007 B2
7206288 Cometto Apr 2007 B2
7310664 Merchant Dec 2007 B1
7313637 Tanaka Dec 2007 B2
7315545 Chowdhury et al. Jan 2008 B1
7316031 Griffith Jan 2008 B2
7330897 Baldwin Feb 2008 B2
7380025 Riggins May 2008 B1
7397794 Lacroute Jul 2008 B1
7430164 Bare Sep 2008 B2
7453888 Zabihi Nov 2008 B2
7477894 Sinha Jan 2009 B1
7480258 Shuen Jan 2009 B1
7508757 Ge Mar 2009 B2
7558195 Kuo Jul 2009 B1
7558273 Grosser, Jr. Jul 2009 B1
7571447 Ally Aug 2009 B2
7599901 Mital Oct 2009 B2
7688736 Walsh Mar 2010 B1
7688960 Aubuchon Mar 2010 B1
7690040 Frattura Mar 2010 B2
7706255 Kondrat et al. Apr 2010 B1
7716370 Devarapalli May 2010 B1
7720076 Dobbins May 2010 B2
7729296 Choudhary Jun 2010 B1
7787480 Mehta Aug 2010 B1
7792920 Istvan Sep 2010 B2
7796593 Ghosh Sep 2010 B1
7808992 Homchaudhuri Oct 2010 B2
7836332 Hara Nov 2010 B2
7843906 Chidambaram et al. Nov 2010 B1
7843907 Abou-Emara Nov 2010 B1
7860097 Lovett Dec 2010 B1
7898959 Arad Mar 2011 B1
7912091 Krishnan Mar 2011 B1
7924837 Shabtay Apr 2011 B1
7937756 Kay May 2011 B2
7945941 Sinha May 2011 B2
7949638 Goodson May 2011 B1
7957386 Aggarwal Jun 2011 B1
8018938 Fromm Sep 2011 B1
8027354 Portolani Sep 2011 B1
8054832 Shukla Nov 2011 B1
8068442 Kompella Nov 2011 B1
8078704 Lee Dec 2011 B2
8102781 Smith Jan 2012 B2
8102791 Tang Jan 2012 B2
8116307 Thesayi Feb 2012 B1
8125928 Mehta Feb 2012 B2
8134922 Elangovan Mar 2012 B2
8155150 Chung Apr 2012 B1
8160063 Maltz Apr 2012 B2
8160080 Arad Apr 2012 B1
8170038 Belanger May 2012 B2
8175107 Yalagandula May 2012 B1
8194674 Pagel Jun 2012 B1
8195774 Lambeth Jun 2012 B2
8204061 Sane Jun 2012 B1
8213313 Doiron Jul 2012 B1
8213336 Smith Jul 2012 B2
8230069 Korupolu Jul 2012 B2
8239960 Frattura Aug 2012 B2
8249069 Raman Aug 2012 B2
8270401 Barnes Sep 2012 B1
8295291 Ramanathan Oct 2012 B1
8295921 Wang Oct 2012 B2
8301686 Appajodu Oct 2012 B1
8339994 Gnanasekaran Dec 2012 B2
8351352 Eastlake, III Jan 2013 B1
8369335 Jha Feb 2013 B2
8369347 Xiong Feb 2013 B2
8392496 Linden Mar 2013 B2
8462774 Page Jun 2013 B2
8467375 Blair Jun 2013 B2
8520595 Yadav Aug 2013 B2
8599850 Jha Dec 2013 B2
8599864 Chung Dec 2013 B2
8615008 Natarajan Dec 2013 B2
8706905 McGlaughlin Apr 2014 B1
8724456 Hong May 2014 B1
8806031 Kondur Aug 2014 B1
8826385 Congdon Sep 2014 B2
8918631 Kumar Dec 2014 B1
8937865 Kumar Jan 2015 B1
8995272 Agarwal Mar 2015 B2
20010005527 Vaeth Jun 2001 A1
20010055274 Hegge Dec 2001 A1
20020019904 Katz Feb 2002 A1
20020021701 Lavian Feb 2002 A1
20020039350 Wang Apr 2002 A1
20020054593 Morohashi May 2002 A1
20020091795 Yip Jul 2002 A1
20030041085 Sato Feb 2003 A1
20030123393 Feuerstraeter Jul 2003 A1
20030147385 Montalvo Aug 2003 A1
20030174706 Shankar Sep 2003 A1
20030189905 Lee Oct 2003 A1
20030208616 Laing Nov 2003 A1
20030216143 Roese Nov 2003 A1
20040001433 Gram Jan 2004 A1
20040003094 See Jan 2004 A1
20040010600 Baldwin Jan 2004 A1
20040049699 Griffith Mar 2004 A1
20040057430 Paavolainen Mar 2004 A1
20040081171 Finn Apr 2004 A1
20040117508 Shimizu Jun 2004 A1
20040120326 Yoon Jun 2004 A1
20040156313 Hofmeister et al. Aug 2004 A1
20040165595 Holmgren Aug 2004 A1
20040165596 Garcia Aug 2004 A1
20040205234 Barrack Oct 2004 A1
20040213232 Regan Oct 2004 A1
20050007951 Lapuh Jan 2005 A1
20050044199 Shiga Feb 2005 A1
20050074001 Mattes Apr 2005 A1
20050094568 Judd May 2005 A1
20050094630 Valdevit May 2005 A1
20050122979 Gross Jun 2005 A1
20050157645 Rabie et al. Jul 2005 A1
20050157751 Rabie Jul 2005 A1
20050169188 Cometto Aug 2005 A1
20050195813 Ambe Sep 2005 A1
20050207423 Herbst Sep 2005 A1
20050213561 Yao Sep 2005 A1
20050220096 Friskney Oct 2005 A1
20050265356 Kawarai Dec 2005 A1
20050278565 Frattura Dec 2005 A1
20060007869 Hirota Jan 2006 A1
20060018302 Ivaldi Jan 2006 A1
20060023707 Makishima et al. Feb 2006 A1
20060029055 Perera Feb 2006 A1
20060034292 Wakayama Feb 2006 A1
20060036765 Weyman Feb 2006 A1
20060059163 Frattura Mar 2006 A1
20060062187 Rune Mar 2006 A1
20060072550 Davis Apr 2006 A1
20060083254 Ge Apr 2006 A1
20060098589 Kreeger May 2006 A1
20060140130 Kalkunte Jun 2006 A1
20060168109 Warmenhoven Jul 2006 A1
20060184937 Abels Aug 2006 A1
20060221960 Borgione Oct 2006 A1
20060227776 Chandrasekaran Oct 2006 A1
20060235995 Bhatia Oct 2006 A1
20060242311 Mai Oct 2006 A1
20060245439 Sajassi Nov 2006 A1
20060251067 DeSanti Nov 2006 A1
20060256767 Suzuki Nov 2006 A1
20060265515 Shiga Nov 2006 A1
20060285499 Tzeng Dec 2006 A1
20060291388 Amdahl Dec 2006 A1
20060291480 Cho Dec 2006 A1
20070036178 Hares Feb 2007 A1
20070053294 Ho Mar 2007 A1
20070083625 Chamdani Apr 2007 A1
20070086362 Kato Apr 2007 A1
20070094464 Sharma Apr 2007 A1
20070097968 Du May 2007 A1
20070098006 Parry May 2007 A1
20070116224 Burke May 2007 A1
20070116422 Reynolds May 2007 A1
20070156659 Lim Jul 2007 A1
20070177525 Wijnands Aug 2007 A1
20070177597 Ju Aug 2007 A1
20070183313 Narayanan Aug 2007 A1
20070211712 Fitch Sep 2007 A1
20070258449 Bennett Nov 2007 A1
20070274234 Kubota Nov 2007 A1
20070289017 Copeland, III Dec 2007 A1
20080052487 Akahane Feb 2008 A1
20080056135 Lee Mar 2008 A1
20080065760 Damm Mar 2008 A1
20080080517 Roy Apr 2008 A1
20080095160 Yadav Apr 2008 A1
20080101386 Gray May 2008 A1
20080112400 Dunbar et al. May 2008 A1
20080133760 Berkvens et al. Jun 2008 A1
20080159277 Vobbilisetty Jul 2008 A1
20080172492 Raghunath Jul 2008 A1
20080181196 Regan Jul 2008 A1
20080181243 Vobbilisetty Jul 2008 A1
20080186981 Seto Aug 2008 A1
20080205377 Chao Aug 2008 A1
20080219172 Mohan Sep 2008 A1
20080225852 Raszuk Sep 2008 A1
20080225853 Melman Sep 2008 A1
20080228897 Ko Sep 2008 A1
20080240129 Elmeleegy Oct 2008 A1
20080267179 LaVigne Oct 2008 A1
20080285458 Lysne Nov 2008 A1
20080285555 Ogasahara Nov 2008 A1
20080298248 Roeck Dec 2008 A1
20080304498 Jorgensen Dec 2008 A1
20080310342 Kruys Dec 2008 A1
20090022069 Khan Jan 2009 A1
20090037607 Farinacci Feb 2009 A1
20090042270 Dolly Feb 2009 A1
20090044270 Shelly Feb 2009 A1
20090067422 Poppe Mar 2009 A1
20090067442 Killian Mar 2009 A1
20090079560 Fries Mar 2009 A1
20090080345 Gray Mar 2009 A1
20090083445 Ganga Mar 2009 A1
20090092042 Yuhara Apr 2009 A1
20090092043 Lapuh Apr 2009 A1
20090106405 Mazarick Apr 2009 A1
20090116381 Kanda May 2009 A1
20090129384 Regan May 2009 A1
20090138577 Casado May 2009 A1
20090138752 Graham May 2009 A1
20090161584 Guan Jun 2009 A1
20090161670 Shepherd Jun 2009 A1
20090168647 Holness Jul 2009 A1
20090199177 Edwards Aug 2009 A1
20090204965 Tanaka Aug 2009 A1
20090213783 Moreton Aug 2009 A1
20090222879 Kostal Sep 2009 A1
20090232031 Vasseur Sep 2009 A1
20090245137 Hares Oct 2009 A1
20090245242 Carlson Oct 2009 A1
20090246137 Hadida Oct 2009 A1
20090252049 Ludwig Oct 2009 A1
20090252061 Small Oct 2009 A1
20090260083 Szeto Oct 2009 A1
20090279558 Davis Nov 2009 A1
20090292858 Lambeth Nov 2009 A1
20090316721 Kanda Dec 2009 A1
20090323698 LeFaucheur Dec 2009 A1
20090323708 Ihle Dec 2009 A1
20090327392 Tripathi Dec 2009 A1
20090327462 Adams Dec 2009 A1
20100027420 Smith Feb 2010 A1
20100046471 Hattori Feb 2010 A1
20100054260 Pandey Mar 2010 A1
20100061269 Banerjee Mar 2010 A1
20100074175 Banks Mar 2010 A1
20100097941 Carlson Apr 2010 A1
20100103813 Allan Apr 2010 A1
20100103939 Carlson Apr 2010 A1
20100131636 Suri May 2010 A1
20100158024 Sajassi Jun 2010 A1
20100165877 Shukla Jul 2010 A1
20100165995 Mehta Jul 2010 A1
20100168467 Johnston Jul 2010 A1
20100169467 Shukla Jul 2010 A1
20100169948 Budko Jul 2010 A1
20100182920 Matsuoka Jul 2010 A1
20100195489 Zhou Aug 2010 A1
20100215042 Sato Aug 2010 A1
20100220724 Rabie Sep 2010 A1
20100226381 Mehta Sep 2010 A1
20100246388 Gupta Sep 2010 A1
20100257263 Casado Oct 2010 A1
20100265849 Harel Oct 2010 A1
20100271960 Krygowski Oct 2010 A1
20100272107 Papp Oct 2010 A1
20100281106 Ashwood-Smith Nov 2010 A1
20100284414 Agarwal Nov 2010 A1
20100284418 Gray Nov 2010 A1
20100287262 Elzur Nov 2010 A1
20100287548 Zhou Nov 2010 A1
20100290473 Enduri Nov 2010 A1
20100299527 Arunan Nov 2010 A1
20100303071 Kotalwar Dec 2010 A1
20100303075 Tripathi Dec 2010 A1
20100303083 Belanger Dec 2010 A1
20100309820 Rajagopalan Dec 2010 A1
20100309912 Mehta Dec 2010 A1
20100329110 Rose Dec 2010 A1
20110019678 Mehta Jan 2011 A1
20110035489 McDaniel Feb 2011 A1
20110035498 Shah Feb 2011 A1
20110044339 Kotalwar Feb 2011 A1
20110044352 Chaitou Feb 2011 A1
20110064086 Xiong Mar 2011 A1
20110064089 Hidaka Mar 2011 A1
20110072208 Gulati Mar 2011 A1
20110085560 Chawla Apr 2011 A1
20110085563 Kotha Apr 2011 A1
20110110266 Li May 2011 A1
20110134802 Rajagopalan Jun 2011 A1
20110134803 Dalvi Jun 2011 A1
20110134925 Safrai Jun 2011 A1
20110142053 Van Der Merwe Jun 2011 A1
20110142062 Wang Jun 2011 A1
20110161494 McDysan Jun 2011 A1
20110161695 Okita Jun 2011 A1
20110176412 Stine Jul 2011 A1
20110188373 Saito Aug 2011 A1
20110194403 Sajassi Aug 2011 A1
20110194563 Shen Aug 2011 A1
20110228780 Ashwood-Smith Sep 2011 A1
20110231570 Altekar Sep 2011 A1
20110231574 Saunderson Sep 2011 A1
20110235523 Jha Sep 2011 A1
20110243133 Villait Oct 2011 A9
20110243136 Raman Oct 2011 A1
20110246669 Kanada Oct 2011 A1
20110255538 Srinivasan Oct 2011 A1
20110255540 Mizrahi Oct 2011 A1
20110261828 Smith Oct 2011 A1
20110268120 Vobbilisetty Nov 2011 A1
20110268125 Vobbilisetty Nov 2011 A1
20110273988 Tourrilhes Nov 2011 A1
20110274114 Dhar Nov 2011 A1
20110280572 Vobbilisetty Nov 2011 A1
20110286457 En Nov 2011 A1
20110296052 Guo Dec 2011 A1
20110299391 Vobbilisetty Dec 2011 A1
20110299413 Chatwani Dec 2011 A1
20110299414 Yu Dec 2011 A1
20110299527 Yu Dec 2011 A1
20110299528 Yu Dec 2011 A1
20110299531 Yu Dec 2011 A1
20110299532 Yu Dec 2011 A1
20110299533 Yu Dec 2011 A1
20110299534 Koganti Dec 2011 A1
20110299535 Vobbilisetty Dec 2011 A1
20110299536 Cheng Dec 2011 A1
20110317559 Kern Dec 2011 A1
20110317703 Dunbar et al. Dec 2011 A1
20120011240 Hara Jan 2012 A1
20120014261 Salam Jan 2012 A1
20120014387 Dunbar Jan 2012 A1
20120020220 Sugita Jan 2012 A1
20120027017 Rai Feb 2012 A1
20120033663 Guichard Feb 2012 A1
20120033665 Da Silva Feb 2012 A1
20120033668 Humphries Feb 2012 A1
20120033669 Mohandas Feb 2012 A1
20120033672 Page Feb 2012 A1
20120063363 Li Mar 2012 A1
20120075991 Sugita Mar 2012 A1
20120099567 Hart Apr 2012 A1
20120099602 Nagapudi Apr 2012 A1
20120106339 Mishra May 2012 A1
20120117438 Shaffer May 2012 A1
20120131097 Baykal May 2012 A1
20120131289 Taguchi May 2012 A1
20120134266 Roitshtein May 2012 A1
20120147740 Nakash Jun 2012 A1
20120158997 Hsu Jun 2012 A1
20120163164 Terry Jun 2012 A1
20120177039 Berman Jul 2012 A1
20120210416 Mihelich Aug 2012 A1
20120243539 Keesara Sep 2012 A1
20120275297 Subramanian Nov 2012 A1
20120275347 Banerjee Nov 2012 A1
20120278804 Narayanasamy Nov 2012 A1
20120294192 Masood Nov 2012 A1
20120294194 Balasubramanian Nov 2012 A1
20120320800 Kamble Dec 2012 A1
20120320926 Kamath et al. Dec 2012 A1
20120327766 Tsai et al. Dec 2012 A1
20120327937 Melman et al. Dec 2012 A1
20130003535 Sarwar Jan 2013 A1
20130003737 Sinicrope Jan 2013 A1
20130003738 Koganti Jan 2013 A1
20130028072 Addanki Jan 2013 A1
20130034015 Jaiswal Feb 2013 A1
20130034021 Jaiswal Feb 2013 A1
20130067466 Combs Mar 2013 A1
20130070762 Adams Mar 2013 A1
20130114595 Mack-Crane et al. May 2013 A1
20130124707 Ananthapadmanabha May 2013 A1
20130127848 Joshi May 2013 A1
20130136123 Ge May 2013 A1
20130148546 Eisenhauer Jun 2013 A1
20130194914 Agarwal Aug 2013 A1
20130219473 Schaefer Aug 2013 A1
20130250951 Koganti Sep 2013 A1
20130259037 Natarajan Oct 2013 A1
20130294451 Li Nov 2013 A1
20130301642 Radhakrishnan Nov 2013 A1
20130308647 Rosset Nov 2013 A1
20130346583 Low Dec 2013 A1
20140013324 Zhang Jan 2014 A1
20140025736 Wang Jan 2014 A1
20140044126 Sabhanatarajan Feb 2014 A1
20140056298 Vobbilisetty Feb 2014 A1
20140105034 Sun Apr 2014 A1
20150010007 Matsuhira Jan 2015 A1
20150030031 Zhou Jan 2015 A1
20150143369 Zheng May 2015 A1
Foreign Referenced Citations (11)
Number Date Country
102801599 Nov 2012 CN
0579567 May 1993 EP
0579567 Jan 1994 EP
0993156 Dec 2000 EP
1398920 Mar 2004 EP
1916807 Apr 2008 EP
2001167 Dec 2008 EP
2008056838 May 2008 WO
2009042919 Apr 2009 WO
2010111142 Sep 2010 WO
2014031781 Feb 2014 WO
Non-Patent Literature Citations (206)
Entry
Touch, J. et al., ‘Transparent Interconnection of Lots of Links (TRILL): Problem and Applicability Statement’, May 2009, Network Working Group, pp. 1-17.
Zhai F. Hu et al. ‘RBridge: Pseudo-Nickname; draft-hu-trill-pseudonode-nickname-02.txt’, May 15, 2012.
Office Action dated Sep. 17. 2015, U.S. Appl. No. 14/577,785, filed Dec. 19, 2014.
Narten, T. et al. “Problem Statement: Overlays for Network Virtualization draft-narten-nvo3-overlay-problem statement-01”, Oct. 31, 2011.
Knight, Paul et al. “Layer 2 and 3 Virtual Private Networks: Taxonomy, Technology, and Standardization Efforts”, 2004.
An Introduction to Brocade VCS Fabric Technology, Dec. 3, 2012.
Kreeger, L. et al. “Network Virtualization Overlay Control Protocol Requirements draft-kreeger-nvo3-overlay-cp-00”, Aug. 2, 2012.
Knight, Paul et al., “Network based IP VPN Architecture using Virtual Routers”, May 2003.
Louati, Wajdi et al., “Network-Based Virtual Personal Overlay Networks Using Programmable Virtual Routers”, 2005.
Brocade Unveils “The Effortless Network”, 2009.
The Effortless Network: HyperEdge Technology for the Campus LAN, 2012.
Foundary FastIron Configuration Guide, Software Release FSX 04.2.00b, Software Release FWS 04.3.00, Software Release FGS 05.0.00a, 2008.
FastIron and TurbuIron 24x Configuration Guide, 2010.
FastIron Configuration Guide, Supporting IronWare Software Release 07.0.00, 2009.
Christensen, M. et al., Considerations for Internet Group Management Protocol (IGMP) and Multicast Listener Discovery (MLD) Snooping Switches, 2006.
Perlman, Radia et al. “RBridges: Base Protocol Specification”, <draft-ietf-trill-rbridge-protocol-16.txt>, 2010.
Brocade Fabric OS (FOS) 6.2 Virtual Fabrics Feature Frequently Asked Questions, 2009.
Eastlake III, Donald et al., “RBridges: TRILL Header Options”, 2009.
Perlman, Radia “Challenges and Opportunities in the Design of TRILL: a Routed layer 2 Technology”, 2009.
Perlman, Radia et al., “RBridge VLAN Mapping”, <draft-ietf-trill-rbridge-vlan-mapping-01.txt>, 2009.
Knight, S. et al., “Virtual Router Redundancy Protocol”, 1998.
“Switched Virtual Internetworking moves beyond bridges and routers”, 8178 Data Communications Sep. 23, 1994, No. 12.
Touch, J. et al., “Transparent Interconnection of Lots of Links (TRILL): Problem and Applicability Statement”, 2009.
Lapuh, Roger et al., “Split Multi-link Trunking (SMLT)”, 2002.
Lapuh, Roger et al., “Split Multi-Link Trunking (SMLT) draft-Lapuh-network-smlt-08”, 2009.
Nadas, S. et al., “Virtual Router Redundancy Protocol (VRRP) Version 3 for IPv4 and IPv6”, 2010.
Office Action for U.S. Appl. No. 12/725,249, filed Mar. 16, 2010, dated Sep. 12, 2012.
Office Action for U.S. Appl. No. 12/725,249, filed Mar. 16, 2010, dated Apr. 26, 2013.
Office Action for U.S. Appl. No. 13/087,239, filed Apr. 14, 2011, dated Dec. 5, 2012.
Office Action for U.S. Appl. No. 13/087,239, filed Apr. 14, 2011, dated May 22, 2013.
Office Action for U.S. Appl. No. 13/098,490, filed May 2, 2011, dated Dec. 21, 2012.
Office Action for U.S. Appl. No. 13/098,490, filed May 2, 2011, dated Jul. 9, 2013.
Office Action for U.S. Appl. No. 13/092,724, filed Apr. 22, 2011, dated Feb. 5, 2013.
Office Action for U.S. Appl. No. 13/092,724, filed Apr. 22, 2011, dated Jul. 16, 2013.
Office Action for U.S. Appl. No. 13/092,580, filed Apr. 22, 2011, dated Jun. 10, 2013.
Office Action for U.S. Appl. No. 13/042,259, filed Mar. 7, 2011, dated Mar. 18, 2013.
Office Action for U.S. Appl. No. 13/092,460, filed Apr. 22, 2011, dated Jun. 21, 2013.
Office Action for U.S. Appl. No. 13/042,259, filed Mar. 7, 2011, dated Jul. 31, 2013.
Office Action for U.S. Appl. No. 13/092,701, filed Apr. 22, 2011, dated Jan. 28, 2013.
Office Action for U.S. Appl. No. 13/092,701, filed Apr. 22, 2011, dated Jul. 3, 2013.
Office Action for U.S. Appl. No. 13/092,752, filed Apr. 22, 2011, dated Feb. 5, 2013.
Office Action for U.S. Appl. No. 13/092,752, filed Apr. 22, 2011, dated Jul. 18, 2013.
Office Action for U.S. Appl. No. 13/950,974, filed Nov. 19, 2010, dated Dec. 20, 2012.
Office Action for U.S. Appl. No. 12/950,974, filed Nov. 19, 2010, dated May 24, 2012.
Office Action for U.S. Appl. No. 13/092,877, filed Apr. 22, 2011, dated Mar. 4, 2013.
Office Action for U.S. Appl. No. 13/902,877, filed Apr. 22, 2011, dated Sep. 5, 2013.
Office Action for U.S. Appl. No. 12/950,968, filed Nov. 19, 2010, dated Jun. 7, 2012.
Office Action for U.S. Appl. No. 12/950,968, filed Nov. 19, 2010, dated Jan. 4, 2013.
Office Action for U.S. Appl. No. 13/092,864, filed Apr. 22, 2011, dated Sep. 19, 2012.
Office Action for U.S. Appl. No. 13/098,360, filed Apr. 29, 2011, dated May 31, 2013.
Office Action for U.S. Appl. No. 13/044,326, filed Mar. 9, 2011, dated Oct. 2, 2013.
Office Action for U.S. Appl. No. 13/030,806, filed Feb. 18, 2011, dated Dec. 3, 2012.
Office Action for U.S. Appl. No. 13/030,806, filed Feb. 18, 2011, dated Jun. 11, 2013.
Office Action for U.S. Appl. No. 13/030,688, filed Feb. 18, 2011, dated Apr. 25, 2013.
Office Action for U.S. Appl. No. 13/044,301, filed Mar. 9, 2011, dated Jun. 11, 2013.
Office Action for U.S. Appl. No. 13/044,301, filed Mar. 9, 2011, dated Feb. 22, 2013.
Office Action for U.S. Appl. No. 13/050,102, filed Oct. 26, 2012, dated Oct. 26, 2012.
Office Action for U.S. Appl. No. 13/050,102, filed May 16, 2013, date May 16, 2013.
Office Action for U.S. Appl. No. 13/184,526, filed Jul. 16, 2011, dated Jan. 28, 2013.
Office Action for U.S. Appl. No. 13/184,526, filed May 22, 2013, dated May 22, 2013.
Office Action for U.S. Appl. No. 13/092,873, filed Apr. 22, 2011, dated Jun. 19, 2013.
Office Action for U.S. Appl. No. 13/365,993, filed Feb. 3, 2012, dated Jul. 23, 2013.
Office Action for U.S. Appl. No. 13/365,808, filed Jul. 18, 2013, dated Jul. 18, 2013.
Office Action for U.S. Appl. No. 13/312,903, filed Dec. 6, 2011, dated Jun. 13, 2013.
Office action dated Apr. 26, 2012, U.S. Appl. No. 12/725,249, filed Mar. 16, 2010.
Office action dated Sep. 12, 2012, U.S. Appl. No. 12/725,249, filed Mar. 16, 2010.
Office action dated Dec. 21, 2012, U.S. Appl. No. 13/098,490, filed May 2, 2011.
Office action dated Mar. 27, 2014, U.S. Appl. No. 13/098,490, filed May 2, 2011.
Office action dated Jul. 9, 2013, U.S. Appl. No. 13/098,490, filed May 2, 2011.
Office action dated May 22, 2013, U.S. Appl. No. 13/087,239, filed Apr. 14, 2011.
Office action dated Dec. 5, 2012, U.S. Appl. No. 13/087,239, filed Apr. 14, 2011.
Office action dated Apr. 9, 2014, U.S. Appl. No. 13/092,724, filed Apr. 22, 2011.
Office action dated Feb. 5, 2013, U.S. Appl. No. 13/092,724, filed Apr. 22, 2011.
Office action dated Jan. 10, 2014, U.S. Appl. No. 13/092,580, filed Apr. 22, 2011.
Office action dated Jun. 10, 2013, U.S. Appl. No. 13/092,580, filed Apr. 22, 2011
Office action dated Jan. 16, 2014, U.S. Appl. No. 13/042,259, filed Mar. 7, 2011.
Office action dated Mar. 18, 2013, U.S. Appl. No. 13/042,259, filed Mar. 7, 2011.
Office action dated Jul. 31, 2013, U.S. Appl. No. 13/042,259, filed Mar. 7, 2011.
Office action dated Aug. 29, 2014, U.S. Appl. No. 13/042,259, filed Mar. 7, 2011.
Office action dated Mar. 14, 2014, U.S. Appl. No. 13/092,460, filed Apr. 22, 2011.
Office action dated Jun. 21, 2013, U.S. Appl. No. 13/092,460, filed Apr. 22, 2011.
Office action dated Jan. 28, 2013, U.S. Appl. No. 13/092,701, filed Apr. 22, 2011.
Office action dated Mar. 26, 2014, U.S. Appl. No. 13/092,701, filed Apr. 22, 2011.
Office action dated Jul. 3, 2013, U.S. Appl. No. 13/092,701, filed Apr. 22, 2011.
Office action dated Jul. 18, 2013, U.S. Appl. No. 13/092,752, filed Apr. 22, 2011.
Office action dated Dec. 20, 2012, U.S. Appl. No. 12/950,974, filed Nov. 19, 2010.
Office action dated May 24, 2012, U.S. Appl. No. 12/950,974, filed Nov. 19, 2010.
Office action dated Jan. 6, 2014, U.S. Appl. No. 13/092,877, filed Apr. 22, 2011.
Office action dated Sep. 5, 2013, U.S. Appl. No. 13/092,877, filed Apr. 22, 2011.
Office action dated Mar. 4, 2013, U.S. Appl. No. 13/092,877, filed Apr. 22, 2011.
Office action dated Jan. 4, 2013, U.S. Appl. No. 12/950,968, filed Nov. 19, 2010.
Office action dated Jun. 7, 2012, U.S. Appl. No. 12/950,968, filed Nov. 19, 2010.
Office action dated Sep. 19, 2012, U.S. Appl. No. 13/092,864, filed Apr. 22, 2011.
Office action dated May 31, 2013, U.S. Appl. No. 13/098,360, filed Apr. 29, 2011.
Office action dated Oct. 2, 2013, U.S. Appl. No. 13/044,326, filed Mar. 9, 2011.
Office action dated Dec. 3, 2012, U.S. Appl. No. 13/030,806, filed Feb. 18, 2011.
Office action dated Apr. 22, 2014, U.S. Appl. No. 13/030,806, filed Feb. 18, 2011.
Office action dated Jun. 11, 2013, U.S. Appl. No. 13/030,806, filed Feb. 18, 2011.
Office action dated Apr. 25, 2013, U.S. Appl. No. 13/030,688, filed Feb. 18, 2011.
Office action dated Feb. 22, 2013, U.S. Appl. No. 13/044,301, filed Mar. 9, 2011.
Office action dated Jun. 11, 2013, U.S. Appl. No. 13/044,301, filed Mar. 9, 2011.
Office action dated Oct. 26, 2012, U.S. Appl. No. 13/050,102, filed Mar. 17, 2011.
Office action dated May 16, 2013, U.S. Appl. No. 13/050,102, filed Mar. 17, 2011.
Office action dated Aug. 4, 2014, U.S. Appl. No. 13/050,102, filed Mar. 17, 2011.
Office action dated Jan. 28, 2013, U.S. Appl. No. 13/148,526, filed Jul. 16, 2011.
Office action dated Dec. 2, 2013, U.S. Appl. No. 13/184,526, filed Jul. 16, 2011.
Office action dated May 22, 2013, U.S. Appl. No. 13/148,526, filed Jul. 16, 2011.
Office action dated Aug. 21, 2014, U.S. Appl. No. 13/184,526, filed Jul. 16, 2011.
Office action dated Nov. 29, 2013, U.S. Appl. No. 13/092,873, filed Apr. 22, 2011.
Office action dated Jun. 19, 2013, U.S. Appl. No. 13/092,873, filed Apr. 22, 2011.
Office action dated Jul. 18, 2013, U.S. Appl. No. 13/365,808, filed Feb. 3, 2012.
Office Action dated Mar. 6, 2014, U.S. Appl. No. 13/425,238, filed Mar. 20, 2012.
Office action dated Nov. 12, 2013, U.S. Appl. No. 13/312,903, filed Dec. 6, 2011.
Office action dated Jun. 13, 2013, U.S. Appl. No. 13/312,903, filed Dec. 6, 2011.
Office Action dated Jun. 18, 2014, U.S. Appl. No. 13/440,861, filed Apr. 5, 2012.
Office Action dated Feb. 28, 2014, U.S. Appl. No. 13/351,513, filed Jan. 17, 2012.
Office Action dated May 9, 2014, U.S. Appl. No. 13/484,072, filed May 30, 2012.
Office Action dated May 14, 2014, U.S. Appl. No. 13/533,843, filed Jun. 26, 2012.
Office Action dated Feb. 20, 2014, U.S. Appl. No. 13/598,204, filed Aug. 29, 2012.
Office Action dated Jun. 6, 2014, U.S. Appl. No. 13/669,357, filed Nov. 5, 2012.
Huang, Nen-Fu et al., ‘An Effective Spanning Tree Algorithm for a Bridged LAN’, Mar. 16, 1992.
Office Action for U.S. Appl. No. 13/351,513, filed Jan. 17, 2012, dated Feb. 28, 2014.
Office Action for U.S. Appl. No. 13/533,843, filed Jun. 26, 2012, dated Oct. 21, 2013.
Perlman, Radia et al., ‘RBridges: Base Protocol Specification; Draft-ietf-trill-rbridge-protocol-16.txt’, Mar. 3, 2010, pp. 1-117.
‘An Introduction to Brocade VCS Fabric Technology’, Brocade white paper, http://community.brocade.com/docs/DOC-2954, Dec. 3, 2012.
U.S. Appl. No. 13/030,806 Office Action dated Dec. 3, 2012.
Office Action dated Apr. 9, 2014, U.S. Appl. No. 13/092,752, filed Apr. 22, 2011.
Brocade ‘Brocade Unveils ‘The Effortless Network’’, http://newsroom.brocade.com/press-releases/brocade-unveils-the-effortless-network-nasdaq-brcd-0859535, 2012.
Kreeger, L. et al., ‘Network Virtualization Overlay Control Protocol Requirements draft-kreeger-nvo3-overlay-cp-00’, Jan. 30, 2012.
Lapuh, Roger et al., ‘Split Multi-link Trunking (SMLT)’, draft-lapuh-network-smlt-08, Jul. 2008.
Office Action for U.S. Appl. No. 13/365,993, filed Feb. 3, 2012, from Cho, Hong Sol., dated Jul. 23, 2013.
Office Action for U.S. Appl. No. 12/950,974, filed Nov. 19, 2010, dated Dec. 20, 2012.
Office Action for U.S. Appl. No. 13/092,877, filed Apr. 22, 2011, dated Sep. 5, 2013.
Office Action for U.S. Appl. No. 13/092,887 dated Jan. 6, 2014.
Office Action for U.S. Appl. No. 13/098,490, filed May 2, 2011, dated Mar. 27, 2014.
Office Action for U.S. Appl. No. 13/092,873, filed Apr. 22, 2011, dated Nov. 29, 2013.
Office Action for U.S. Appl. No. 13/184,526, filed Jul. 16, 2011, dated Dec. 2, 2013.
Office Action for U.S. Appl. No. 13/598,204, filed Aug. 29, 2012, dated Feb. 20, 2014.
Office Action for U.S. Appl. No. 13/044,326, filed Mar. 9, 2011, dated Jul. 7, 2014.
Office Action for U.S. Appl. No. 13/092,752, filed Apr. 22, 2011, dated Apr. 9, 2014.
Office Action for U.S. Appl. No. 13/092,873, filed Apr. 22, 2011, dated Jul. 25, 2014.
Office Action for U.S. Appl. No. 13/092,877, filed Apr. 22, 2011, dated Jun. 20, 2014.
Office Action for U.S. Appl. No. 13/312,903, filed Dec. 6, 2011, dated Aug. 7, 2014.
Office Action for U.S. Appl. No. 13/351,513, filed Jan. 17, 2012, dated Jul. 24, 2014.
Office Action for U.S. Appl. No. 13/425,238, filed Mar. 20, 2012, dated Mar. 6, 2014.
Office Action for U.S. Appl. No. 13/556,061, filed Jul. 23, 2012, dated Jun. 6, 2014.
Office Action for U.S. Appl. No. 13/742,207 dated Jul. 24, 2014, filed Jan. 15, 2013.
Office Action for U.S. Appl. No. 13/950,974, filed Nov. 19, 2010, from Haile, Awet A., dated Dec. 2, 2012.
Perlman R: ‘Challenges and opportunities in the design of TRILL: a routed layer 2 technology’, 2009 IEEE Globecom Workshops, Honolulu, HI, USA, Piscataway, NJ, USA, Nov. 30, 2009 (Nov. 30, 2009), pp. 1-6, XP002649647, DOI: 10.1109/GLOBECOM.2009.5360776 ISBN: 1-4244-5626-0 [retrieved on Jul. 19, 2011].
TRILL Working Group Internet-Draft Intended status: Proposed Standard RBridges: Base Protocol Specificaiton Mar. 3, 2010.
Office action dated Aug. 14, 2014, U.S. Appl. No. 13/092,460, filed Apr. 22, 2011.
Office action dated Jul. 7, 2014, for U.S. Appl. No. 13/044,326, filed Mar. 9, 2011.
Office Action dated Dec. 19, 2014, for U.S. Appl. No. 13/044,326, filed Mar. 9, 2011.
Office Action for U.S. Appl. No. 13/092,873, filed Apr. 22, 2011, dated Nov. 7, 2014.
Office Action for U.S. Appl. No. 13/092,877, filed Apr. 22, 2011, dated Nov. 10, 2014.
Office Action for U.S. Appl. No. 13/157,942, filed Jun. 10, 2011.
Mckeown, Nick et al. “OpenFlow: Enabling Innovation in Campus Networks”, Mar. 14, 2008, www.openflow.org/documents/openflow-wp-latest.pdf.
Office Action for U.S. Appl. No. 13/044,301, dated Mar. 9, 2011.
Office Action for U.S. Appl. No. 13/184,526, filed Jul. 16, 2011, dated Jan. 5, 2015.
Office Action for U.S. Appl. No. 13/598,204, filed Aug. 29, 2012, dated Jan. 5, 2015.
Office Action for U.S. Appl. No. 13/669,357, filed Nov. 5, 2012, dated Jan. 30, 2015.
Office Action for U.S. Appl. No. 13/851,026, filed Mar. 26, 2013, dated Jan. 30, 2015.
Office Action for U.S. Appl. No. 13/786,328, filed Mar. 5, 2013, dated Mar. 13, 2015.
Office Action for U.S. Appl. No. 13/092,460, filed Apr. 22, 2011, dated Mar. 13, 2015.
Office Action for U.S. Appl. No. 13/425,238, dated Mar. 12, 2015.
Office Action for U.S. Appl. No. 13/092,752, filed Apr. 22, 2011, dated Feb. 27, 2015.
Office Action for U.S. Appl. No. 13/042,259, filed Mar. 7, 2011, dated Feb. 23, 2015.
Office Action for U.S. Appl. No. 13/044,301, filed Mar. 9, 2011, dated Jan. 29, 2015.
Office Action for U.S. Appl. No. 13/050,102, filed Mar. 17, 2011, dated Jan. 26, 2015.
Office action dated Oct. 2, 2014, for U.S. Appl. No. 13/092,752, filed Apr. 22, 2011.
Kompella, Ed K. et al., ‘Virtual Private LAN Service (VPLS) Using BGP for Auto-Discovery and Signaling’ Jan. 2007.
Abawajy J. “An Approach to Support a Single Service Provider Address Image for Wide Area Networks Environment” Centre for Parallel and Distributed Computing, School of Computer Science Carleton University, Ottawa, Ontario, K1S 5B6, Canada.
Office Action for U.S. Appl. No. 13/425,238, filed Mar. 20, 2012, dated Mar. 12, 2015.
Office Action for U.S. Appl. No. 14/577,785, filed Dec. 19, 2014, dated Apr. 13, 2015.
Mahalingam “VXLAN: A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks” Oct. 17, 2013 pp. 1-22, Sections 1, 4 and 4.1.
Office action dated Apr. 30, 2015, U.S. Appl. No. 13/351,513, filed Jan. 17, 2012.
Office Action dated Apr. 1, 2015, U.S. Appl. No. 13/656,438, filed Oct. 19, 2012.
Office Action dated May 21, 2015, U.S. Appl. No. 13/288,822, filed Nov. 3, 2011.
Siamak Azodolmolky et al. “Cloud computing networking: Challenges and opportunities for innovations”, IEEE Communications Magazine, vol. 51, No. 7, Jul. 1, 2013.
Office Action dated Apr. 1, 2015 U.S. Appl. No. 13/656,438, filed Oct. 19, 2012.
Office action dated Jun. 8, 2015, U.S. Appl. No. 14/178,042, filed Feb. 11, 2014.
Office Action Dated Jun. 10, 2015, U.S. Appl. No. 13/890,150, filed May 8, 2013.
Office Action dated Jun. 18, 215, U.S. Appl. No. 13/098,490, filed May 2, 2011.
Office Action dated Jun. 16, 2015, U.S. Appl. No. 13/048,817, filed Mar. 15, 2011.
Office Action dated Jul. 31, 2015, U.S. Appl. No. 13/598,204, filed Aug. 29, 2014.
Office Action dated Jul. 31, 2015, U.S. Appl. No. 14/473,941, filed Aug. 29, 2014.
Office Action dated Jul. 31, 2015, U.S. Appl. No. 14/488,173, filed Sep. 16, 2014.
Office Action dated Aug. 21, 2015, U.S. Appl. No. 13/776,217, filed Feb. 25, 2013.
Office Action dated Aug. 19, 2015, U.S. Appl. No. 14/156,374, filed Jan. 15, 2014.
Office Action dated Sep. 2, 2015, U.S. Appl. No. 14/151,693, filed Jan. 9, 2014.
Office Action dated Sep. 17, 2015, U.S. Appl. No. 14/577,785, filed Dec. 19, 2014.
Office Action dated Sep. 22, 2015 U.S. Appl. No. 13/656,438, filed Oct. 19, 2012.
Office Action dated Nov. 5, 2015, U.S. Appl. No. 14/178,042, filed Feb. 11, 2014.
Office Action dated Oct. 19, 2015, U.S. Appl. No. 14/215,996, filed Mar. 17, 2014.
Office Action dated Sep. 18, 2015, U.S. Appl. No. 13/345,566, filed Jan. 6, 2012.
Open Flow Switch Specification Version 1.1.0, Feb. 28, 2011.
Open Flow Switch Specification Version 1.0.0, Dec. 31, 2009.
Open Flow Configuration and Management Protocol 1.0 (OF-Config 1.0) Dec. 23, 2011.
Open Flow Switch Specification Version 1.2 Dec. 5, 2011.
Office action dated Feb. 2, 2016, U.S. Appl. No. 13/092,460, filed Apr. 22, 2011.
Office Action dated Feb. 2, 2016. U.S. Appl. No. 14/154,106, filed Jan. 13, 2014.
Office Action dated Feb. 3, 2016, U.S. Appl. No. 13/098,490, filed May 2, 2011.
Office Action dated Feb. 4, 2016, U.S. Appl. No. 13/557,105, filed Jul. 24, 2012.
Office Action dated Feb. 11, 2016, U.S. Appl. No. 14/488,173, filed Sep. 16, 2014.
Office Action dated Feb. 24, 2016, U.S. Appl. No. 13/971,397, filed Aug. 20, 2013.
Office Action dated Feb. 24, 2016, U.S. Appl. No. 12/705,508, filed Feb. 12, 2010.
Related Publications (1)
Number Date Country
20140056298 A1 Feb 2014 US
Provisional Applications (1)
Number Date Country
61691723 Aug 2012 US