Patent Grant
9628407
The present disclosure is related to U.S. Pat. No. 8,867,552, titled “Virtual Cluster Switching,” by inventors Suresh Vobbilisetty and Dilip Chatwani, issued 21 Oct. 2014; and U.S. patent application Ser. No. 14/512,268, titled “Distributed Configuration Management in a Switch Group,” by inventors Vidyasagara R. Guntaka, Suresh Vobbilisetty, Manjunath A. G. Gowda, and Himanshu Varshney, filed 10 Oct. 2014, the disclosures of which are incorporated by reference herein.
Field
The present disclosure relates to communication networks. More specifically, the present disclosure relates to a method for a constructing a scalable, multi-version system.
Related Art
The exponential growth of the Internet has made it a popular delivery medium for a variety of applications running on physical and virtual devices. Such applications have brought with them an increasing demand for bandwidth. As a result, equipment vendors race to build larger and faster switches with versatile capabilities. However, the size of a switch cannot grow infinitely. It is limited by physical space, power consumption, and design complexity, to name a few factors. Furthermore, switches with higher capability are usually more complex and expensive. More importantly, because an overly large and complex system often does not provide economy of scale, simply increasing the size and capability of a switch may prove economically unviable due to the increased per-port cost.
A flexible way to improve the scalability of a switch system is to build a fabric switch. A fabric switch is a collection of individual member switches. These member switches form a single, logical switch that can have an arbitrary number of ports and an arbitrary topology. As demands grow, customers can adopt a “pay as you grow” approach to scale up the capacity of the fabric switch. However, these member switches typically have the same software version to operate as a single switch. As a result, a switch with an updated software version may not operate as a member switch with switches with an older software version. Hence, a software version update may not be performed gradually while keeping a respective member switch operational in the fabric switch.
Meanwhile, a switch, an individual or a member switch of a fabric switch, continues to store more configuration information as the switch participates in network virtualizations, partitions, and switch groups, and operates on a plurality of network protocols of different network layers. This configuration needs to be applied to the switch when the switch powers up, and thus, should be persistent. A switch typically stores such configuration information in a local storage in an unstructured format. The switch reads the information during booting up (i.e., powering up), and loads the information into memory. Managing persistent storage in unstructured format is inefficient and requires runtime structuring.
While persistent storage brings many desirable features to a switch, some issues remain unsolved in efficiently facilitating interoperability of multiple software versions in a switch group.
One embodiment of the present invention provides a first switch. The first switch includes one or more ports, a switch group module, a persistent storage module, and a synchronization module. The switch group module maintains a membership in a switch group. The switch group includes a plurality of switches and operates as a single switch. The persistent storage module stores configuration information associated with the switch group in a data structure, which includes one or more columns for attribute values of the configuration information, in a local persistent storage. The synchronization module obtains update information from a portable representation. This update information comprises a first set of attribute values for a first data structure in the persistent storage. The synchronization module identifies a software version of a second switch in the switch group and generates an update instruction to update the first data structure in a persistent storage in the second switch with target update information. The target update information comprises a second set of attribute values updated from the first set of attribute values.
In a variation on this embodiment, the software version of the second switch is the same as the first switch, and wherein the first and second sets of attribute values are the same.
In a variation on this embodiment, the synchronization module identifies the software version of the second switch from a mapping between an identifier of the second switch and the software version.
In a variation on this embodiment, the synchronization module verifies whether the target update information is prepared for storage in a persistent storage of a respective switch of the switch group. If the update information is prepared, the synchronization module generates an instruction message instructing to commit the target update information to the persistent storage and switch modules of a respective switch of the switch group.
In a further variation, if the update information is not prepared, the synchronization module rolls back the target update information from a persistent storage of a switch, which has succeeded in storage preparation, of the switch group.
In a variation on this embodiment, the first switch also includes a conversion module which converts a first attribute value to a second attribute value. The first attribute value is of a first attribute type and is in the first set of attribute values, and the second attribute value is of a second attribute type and is in the second set of attribute values.
In a variation on this embodiment, the first switch also includes a conversion module which assigns a default attribute value to an attribute in the target update information.
In a variation on this embodiment, the persistent storage is an object relational database, and a data structure in the persistent storage is a table in the database.
In a variation on this embodiment, the portable representation is based on one or more of: Extensible Markup Language (XML) and JavaScript Object Notation (JSON).
In a variation on this embodiment, the switch group is an Ethernet fabric switch, wherein the plurality of switches included the switch group operate as a single Ethernet switch.
One embodiment of the present invention provides a switch. The switch includes one or more ports, a switch group module, a persistent storage module, and a synchronization module. The switch group module maintains a membership in a switch group. The switch group includes a plurality of switches and operates as a single switch. The persistent storage module stores configuration information associated with the switch group in a table, which includes one or more columns for attribute values of the configuration information, in a local persistent storage. The synchronization module identifies update information from a received user command. The update information comprises a first set of attribute values for a first data structure in the persistent storage. The synchronization module generates a portable representation of the received update information and generates a notification message comprising the portable representation. This notification message is destined for a second switch in the switch group.
In a variation on this embodiment, the synchronization module verifies whether the update information is prepared for storage in the persistent storage. If the update information is prepared, the synchronization module generates an acknowledgement message comprising a transaction identifier associated with the preparation.
In a variation on this embodiment, the portable representation is based on one or more of: Extensible Markup Language (XML) and JavaScript Object Notation (JSON).
In the figures, like reference numerals refer to the same figure elements.
The following description is presented to enable any person skilled in the art to make and use the invention, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present invention. Thus, the present invention is not limited to the embodiments shown, but is to be accorded the widest scope consistent with the claims.
Overview
In embodiments of the present invention, the problem of incorporating multiple software versions in a switch group is solved by using portable representation of configuration information and converting the attribute values in the portable representation to corresponding attribute values of an appropriate software version. A software version can indicate, but is not limited to, the version of the operating system, one or more applications, switch images, persistent storage framework, and/or plug-ins of the persistent storage framework. A switch in the switch group can be referred to as a member switch.
With existing technologies, a switch can be configured by logging in to the switch (e.g., via telnet) or via a console port (e.g., an RS-232 port). Such configuration can be related to network virtualizations, partitions, and switch groups, and a plurality of network protocols of different network layers. The attribute values (e.g., parameters) of the configuration information are applied to the switch (e.g., loaded to the switch modules). In some embodiments, one of the switches in the switch group is elected as a principal switch, which manages and distributes configuration information in the switch group. The rest of the switches of the switch group can be referred to as managed switches. When a user accesses this principal switch, the principal switch allows the user to configure the global configuration information of the switch group. The principal switch, in turn, synchronizes that global configuration information with a respective managed switch. Furthermore, the principal switch also allows the user to configure a respective managed switch in the switch group. The principal switch, in turn, synchronizes that local configuration information to that managed switch.
Typically, to facilitate synchronization among the member switches, the member switches of a switch group have the same software version (e.g., software version). This causes a switch with an upgraded software version to leave the switch group until other switches are upgraded as well. For example, an attribute type in the configuration information can be different for different software versions. As a result, when the principal switch tries to synchronize the value of the attribute, an inconsistency can occur among the switches with different software versions. Hence, a software version upgrade may not be performed gradually while keeping a respective member switch operational in the switch group. This leads to unavailability and reduced operational capability of the switch group.
To solve this problem, a respective switch of the switch group uses a portable representation (e.g., an XML representation) to send its update information to the principal switch. The information in a portable representation is recognizable by the principal switch of a switch group. Examples of update information include, but are not limited to, an update to global or local configuration information (e.g., an updated attribute value), a new configuration (e.g., a new VLAN), a new switch image, and a new database image e.g., a database dump). Upon receiving the update information from a user (e.g., a network administrator) via a user command, a member switch sends the update information to the principal switch in the portable representation. The principal switch extracts the update information from the portable representation and creates an instruction to update the corresponding attributes with the received attribute values in the portable representation for a respective software version in the switch group. The principal switch then sends the instruction for a software version to the managed switches with that software version.
In some embodiments, the principal switch supports type conversion. If an attribute type (e.g., an IP address) in a software version is different than the attribute type (e.g., an integer) of the switch version of the originating switch, the principal switch, if supported, can convert the attribute types (e.g., from integer to IP address). Otherwise, the principal switch uses a default value for that different attribute type (e.g., a default IP address). In this way, the configuration information of a respective member switch, even with different software versions, remains synchronized, thereby facilitating interoperability in the switch group.
In some embodiments, a respective switch of the switch group is equipped with a persistent storage framework which stores the configuration information in a local persistent storage. Such a persistent storage can be an object relational database. The configuration information is loaded from this persistent storage to the switch modules (e.g., the application-specific integrated circuit (ASIC) chips of the switch). The principal switch maintains the global configuration information of the switch group and respective local configuration information of a respective switch in its local persistent storage. The principal switch provides a copy of its local persistent storage to a respective managed switch, which in turn, stores that copy in its local persistent storage. In this way, a respective switch in the switch group has the same local and global configuration information. This allows a user to access the configuration information of the switch group from any member switch. In this disclosure, the description in conjunction with
In some embodiments, the framework supports Model Driven Architecture (MDA), Object Oriented Programming (OOP), and/or Model/View/Controller (MVC) design patterns to facilitate modular development and operation of the units. The framework can also support class frameworks based on Unified Modeling Language (UML). Upon receiving class models (e.g., class name, attributes, and methods) and their relations based on UML, the framework automatically generates the corresponding code, thereby ensuring structure in the operational units of a switch. In some embodiments, the class models are expressed in YANG, which is a data modeling language used to model configuration and state data manipulated by the Network Configuration Protocol (NETCONF).
Since the units operate on the framework in a modular way, their associated attribute values can be stored in a persistent storage in a structured way. In some embodiments, the framework uses Object-Relational Mapping to store the attribute values of the units in a structured way in an object relational database. The framework allows different classes to be defined for a unit based on MDA, OOP, and/or MVC design patterns. The framework then seamlessly maps a respective class to a database table and vice-versa. Furthermore, the framework also seamlessly represents the relationships among the classes (e.g., an association or a composition) in the database. As a result, when a unit becomes operational on the switch, attribute values associated with a respective class in that unit is automatically loaded from the database. Moreover, if a class changes (e.g., a new attribute or a new relationship), the framework seamlessly incorporates that change into the database.
In some embodiments, the switch group can be a fabric switch. The switch can include one or more units which allow the switch to join and operate as a member switch of the fabric switch. These units can also run on the framework. In a fabric switch, any number of switches coupled in an arbitrary topology may logically operate as a single switch. The fabric switch can be an Ethernet fabric switch or a virtual cluster switch (VCS), which can operate as a single Ethernet switch. Any member switch may join or leave the fabric switch in “plug-and-play” mode without any manual configuration. In some embodiments, a respective switch in the fabric switch is a Transparent Interconnection of Lots of Links (TRILL) routing bridge (RBridge). In some further embodiments, a respective switch in the fabric switch is an Internet Protocol (IP) routing-capable switch (e.g., an IP router).
It should be noted that a fabric switch is not the same as conventional switch stacking. In switch stacking, multiple switches are interconnected at a common location (often within the same rack), based on a particular topology, and manually configured in a particular way. These stacked switches typically share a common address, e.g., an IP address, so they can be addressed as a single switch externally. Furthermore, switch stacking requires a significant amount of manual configuration of the ports and inter-switch links. The need for manual configuration prohibits switch stacking from being a viable option in building a large-scale switching system. The topology restriction imposed by switch stacking also limits the number of switches that can be stacked. This is because it is very difficult, if not impossible, to design a stack topology that allows the overall switch bandwidth to scale adequately with the number of switch units.
In contrast, a fabric switch can include an arbitrary number of switches with individual addresses, can be based on an arbitrary topology, and does not require extensive manual configuration. The switches can reside in the same location, or be distributed over different locations. These features overcome the inherent limitations of switch stacking and make it possible to build a large “switch farm,” which can be treated as a single, logical switch. Due to the automatic configuration capabilities of the fabric switch, an individual physical switch can dynamically join or leave the fabric switch without disrupting services to the rest of the network.
Furthermore, the automatic and dynamic configurability of the fabric switch allows a network operator to build its switching system in a distributed and “pay-as-you-grow” fashion without sacrificing scalability. The fabric switch's ability to respond to changing network conditions makes it an ideal solution in a virtual computing environment, where network loads often change with time.
In switch stacking, multiple switches are interconnected at a common location (often within the same rack), based on a particular topology, and manually configured in a particular way. These stacked switches typically share a common address, e.g., an IP address, so they can be addressed as a single switch externally. Furthermore, switch stacking requires a significant amount of manual configuration of the ports and inter-switch links. The need for manual configuration prohibits switch stacking from being a viable option in building a large-scale switching system. The topology restriction imposed by switch stacking also limits the number of switches that can be stacked. This is because it is very difficult, if not impossible, to design a stack topology that allows the overall switch bandwidth to scale adequately with the number of switch units.
It should also be noted that a fabric switch is distinct from a virtual local area network (VLAN). A fabric switch can accommodate a plurality of VLANs. A VLAN is typically identified by a VLAN tag. In contrast, the fabric switch is identified a fabric identifier (e.g., a VCS identifier), which is assigned to the fabric switch. A respective member switch of the fabric switch is associated with the fabric identifier. Furthermore, when a member switch of a fabric switch learns a media access control (MAC) address of an end device (e.g., via layer-2 MAC address learning), the member switch generates a notification message, includes the learned MAC address in the payload of the notification message, and sends the notification message to all other member switches of the fabric switch. In this way, a learned MAC address is shared among a respective member switch of the fabric switch.
The term “fabric switch” refers to a number of interconnected physical switches which form a single, scalable logical switch. These physical switches are referred to as member switches of the fabric switch. In a fabric switch, any number of switches can be connected in an arbitrary topology, and the entire group of switches functions together as one single, logical switch. This feature makes it possible to use many smaller, inexpensive switches to construct a large fabric switch, which can be viewed as a single logical switch externally. Although the present disclosure is presented using examples based on a fabric switch, embodiments of the present invention are not limited to a fabric switch. Embodiments of the present invention are relevant to any computing device that includes a plurality of devices operating as a single device.
The term “end device” can refer to any device external to a fabric switch. Examples of an end device include, but are not limited to, a host machine, a conventional layer-2 switch, a layer-3 router, or any other type of network device. Additionally, an end device can be coupled to other switches or hosts further away from a layer-2 or layer-3 network. An end device can also be an aggregation point for a number of network devices to enter the fabric switch. An end device hosting one or more virtual machines can be referred to as a host machine. In this disclosure, the terms “end device” and “host machine” are used interchangeably.
The term “switch” is used in a generic sense, and it can refer to any standalone or fabric switch operating in any network layer. “Switch” should not be interpreted as limiting embodiments of the present invention to layer-2 networks. Any device that can forward traffic to an external device or another switch can be referred to as a “switch.” Any physical or virtual device (e.g., a virtual machine/switch operating on a computing device) that can forward traffic to an end device can be referred to as a “switch.” Examples of a “switch” include, but are not limited to, a layer-2 switch, a layer-3 router, a TRILL RBridge, or a fabric switch comprising a plurality of similar or heterogeneous smaller physical and/or virtual switches.
The term “edge port” refers to a port on a fabric switch which exchanges data frames with a network device outside of the fabric switch (i.e., an edge port is not used for exchanging data frames with another member switch of a fabric switch). The term “inter-switch port” refers to a port which sends/receives data frames among member switches of a fabric switch. The terms “interface” and “port” are used interchangeably.
The term “switch identifier” refers to a group of bits that can be used to identify a switch. Examples of a switch identifier include, but are not limited to, a media access control (MAC) address, an Internet Protocol (IP) address, and an RBridge identifier. Note that the TRILL standard uses “RBridge ID” (RBridge identifier) to denote a 48-bit intermediate-system-to-intermediate-system (IS-IS) System ID assigned to an RBridge, and “RBridge nickname” to denote a 16-bit value that serves as an abbreviation for the “RBridge ID.” In this disclosure, “switch identifier” is used as a generic term, is not limited to any bit format, and can refer to any format that can identify a switch. The term “RBridge identifier” is also used in a generic sense, is not limited to any bit format, and can refer to “RBridge ID,” “RBridge nickname,” or any other format that can identify an RBridge.
The term “packet” refers to a group of bits that can be transported together across a network. “Packet” should not be interpreted as limiting embodiments of the present invention to layer-3 networks. “Packet” can be replaced by other terminologies referring to a group of bits, such as “message,” “frame,” “cell,” or “datagram.”
Network Architecture
In some embodiments, network 100 is a TRILL network and a respective switch of network 100, such as switch 102, is a TRILL RBridge. Under such a scenario, communication among the switches in network 100 is based on the TRILL protocol. For example, upon receiving an Ethernet frame from end device 112, switch 102 encapsulates the received Ethernet frame in a TRILL header and forwards the TRILL packet. In some further embodiments, network 100 is an IP network and a respective switch of network 100, such as switch 102, is an IP-capable switch, which calculates and maintains a local IP routing table (e.g., a routing information base or RIB), and is capable of forwarding packets based on its IP addresses. Under such a scenario, communication among the switches in network 100 is based on IP. For example, upon receiving an Ethernet frame from end device 112, switch 102 encapsulates the received Ethernet frame in an IP header and forwards the IP packet.
In some embodiments, network 100 is a fabric switch (under such a scenario, network 100 can also be referred to as fabric switch 100). Fabric switch 100 is assigned with a fabric switch identifier. A respective member switch of fabric switch 100 is associated with that fabric switch identifier. This allows the member switch to indicate that it is a member of fabric switch 100. In some embodiments, whenever a new member switch joins fabric switch 100, the fabric switch identifier is automatically associated with that new member switch. Furthermore, a respective member switch of fabric switch 100 is assigned a switch identifier (e.g., an RBridge identifier, a Fibre Channel (FC) domain ID (identifier), or an IP address). This switch identifier identifies the member switch in fabric switch 100.
Switches in fabric switch 100 use edge ports to communicate with end devices (e.g., non-member switches) and inter-switch ports to communicate with other member switches. For example, switch 102 is coupled to end device 112 via an edge port and to switches 101, 103, 104, and 105 via inter-switch ports and one or more links. Data communication via an edge port can be based on Ethernet and via an inter-switch port can be based on the IP and/or TRILL protocol. It should be noted that control message exchange via inter-switch ports can be based on a different protocol (e.g., the IP or FC protocol).
A switch, such as switch 102, stores configuration information needed to operate switch 102 as an individual switch or as a member switch of fabric switch 100. Furthermore, switch 102 can participate in various services and operations, such as network virtualization (e.g., a virtual local area networks (VLAN)), switch partitioning, and link aggregations (e.g., a multi-chassis trunk). Furthermore, switch 102 operates on a plurality of network protocols of different network layers (e.g., Ethernet, TRILL, FC, and IP). As a result, switch 102 runs protocol daemons for each of these protocols. However, to incorporate the services and operations, the protocol daemons need to be directly modified, which can lead to conflicts and errors.
Furthermore, each of the operations, services, and the protocols is associated with one or more attributes. These attribute values (e.g., parameters) is typically applied to switch 102 when switch 102 powers up. As a result, these attribute values are stored in a persistent storage so that these values can be retrieved even when switch 102 is powered off or restarts. With existing technologies, switch 102 may store such attribute values in a local storage in an unstructured format (e.g., a string comprising the attribute values). During the boot up process, switch 102 reads and parses the attribute values in the unstructured format, and loads the attribute values into switch memory. Managing persistent storage in unstructured format is inefficient and requires runtime structuring.
To solve this problem, switch 102 is equipped with a persistent storage framework 120 which facilitates structured persistent storage to the attribute values associated with different operational units of switch 102 (e.g., modules and services of switch 102). It should be noted that other switches of network 100 can be equipped with a persistent storage framework and support persistent storage. In some embodiments, some switch of network 100 may not be equipped with a persistent storage framework. Different units of switch 102, each of which facilitates an aspect of switch 102's operations, operate on framework 120 in a structured and modular way. This allows a respective unit to be independently introduced to framework 120 in such a way that the unit can interoperate with other units (e.g., modules and services) of switch 102. Framework 120 supports MDA, OOP, and/or MVC design patterns to facilitate structured development and operation of the units in switch 102.
Since the units operate on framework 120 in a structured way, their associated attribute values can be stored in a persistent storage in a structured way. In some embodiments, framework 120 uses Object-Relational Mapping to store the attribute values of the units in a structured way in an object relational database. Framework 120 allows different classes to be defined for a unit during development based on MDA, OOP, and/or MVC design patterns. Framework 120 supports class models based on UML. In some embodiments, class models are expressed in YANG, which is a data modeling language used to model configuration and state data manipulated by NETCONF. Upon receiving class models (e.g., class name, attributes, and methods) and their relationships based on UML, framework 120 automatically generates the corresponding code, thereby ensuring structure in the operational units of switch 102.
Framework 120 seamlessly maps a respective class to a database table and vice-versa. Furthermore, framework 120 also seamlessly represents the relations among the classes (e.g., an association or a composition) in the database. As a result, when a unit becomes operational on switch 102 (e.g., when switch 102 powers up), attribute values associated with a respective class in that unit is automatically loaded from the database. Moreover, if a class changes (e.g., a new attribute or a new relationship), framework 120 seamlessly incorporates that change into the database.
Persistent Storage Framework
Switch 102 further includes an input interface 122 to switch 102 (e.g., a graphical user interface (GUI) and/or a command line interface (CLI). A user can access input interface 122 via end device 114. The user can obtain information from and provide instruction to switch 102 via input interface 122. Switch 102 also includes a configuration daemon 124, which can receive configuration (e.g., an IP address) for switch 102 from end device 114 (e.g., from a user) via input interface 122. Configuration daemon 124 provides this configuration information to framework 120. Framework 120 can include a configuration daemon gateway module 132, which communicates with configuration daemon 124. Upon receiving the configuration information, framework 120 can identify different attribute values (e.g., a VLAN identifier) and assigns those attribute values to the corresponding attribute of an operational unit of switch 102.
On the other hand, switch 102 can receive an instruction via input interface 122 to provide its configuration associated with one or more units. For example, a user can issue a command to show the IP addresses assigned to switch 102 from end device 114. Input interface 122 provides this instruction to configuration daemon 124, which in turn, sends an internal command to configuration daemon gateway module 132 for the requested configuration information. In response, framework 120 identifies the attributes (e.g., IP addresses) associated with the requested configuration information and obtains the corresponding attribute values (e.g., assigned IP addresses to switch 120) from a persistent storage. Configuration daemon gateway module 132 provides the obtained attribute values to configuration daemon 124. Upon receiving the attribute values, configuration daemon 124 provides the attribute values as the requested configuration information to input interface 122, which in turn, provides the configuration information to end device 114.
Framework 120 includes a core management module 130, which facilitates structured persistent storage to the attribute values associated with different operational units of switch 102 (e.g., modules and services of switch 102). Different units of switch 102 operate on core management module 130 in a structured way. This allows a respective unit to be independently introduced to framework 120 such a way that the unit can interoperate with other units (e.g., modules and services) of switch 102. Framework 120 supports MDA, OOP, and/or MVC design pattern to facilitate structured development and operation of the units in switch 102.
For example, instead of modifying protocol daemons 140, switch 102 can have plug-ins 134 for protocol daemons 140. Core management module 130 facilitates inter-operations between plug-in 134 and protocol daemons 140. Suppose that a modification to standard Ethernet protocol is needed. Instead of modifying the native protocol daemon of Ethernet, a plug-in for the protocol daemon of Ethernet can be introduced to core management module 130. Similarly, to facilitate membership to a fabric switch, fabric switch module 136 can be introduced to core management module 130. Fabric switch module 136 allows switch 102 to run a control plane with automatic configuration capability and join a fabric switch based on the control plane. Plug-ins 134 and fabric switch module 136 can be developed using MDA, OOP, and/or MVC design patterns, supported by framework 120.
Since the units of switch 102 operate core management module 130 in a structured way, their associated attribute values can be stored in a persistent storage in a structured way. In some embodiments, core management module 130 uses Object-Relational Mapping to store the attribute values of the units in a structured way in an object relational database 150. Core management module 130 allows different classes to be defined for a unit during development based on MDA, OOP, and/or MVC design patterns and expressed as a UML model, and seamlessly maps a respective class to a database table in database 150 and vice-versa.
Furthermore, core management module 130 also seamlessly represents the relations among the classes (e.g., an association or a composition) in database 150. As a result, when a unit becomes operational on switch 102 (e.g., when switch 102 powers up), attribute values associated with a respective class in that unit is automatically loaded from database 150. Moreover, if a class changes (e.g., a new attribute or a new relationship), core management module 130 seamlessly incorporates that change into database 150. It should be noted that a class defined by a user may not include explicit instructions (e.g., a Structured Query Language (SQL) query) for inserting and retrieving attribute values from database 150. The class simply includes an instruction indicating that persistent storage is required for some operations and core management module 130 facilitates the object relational mapping, and the corresponding database operations (e.g., SQL insert and select).
Attribute Data Types
To facilitate seamless object relational mapping, a persistent storage framework defines a set of data types for different categories of attributes. These attributes can be used to define class attributes of different operational units of a switch. In some embodiments, the framework can identify these class attributes expressed in a UML model. It should be noted that such expression can be represented in various forms, such as graphical, textual, XML, etc. The framework ensures these attributes are compatible with an object relational database. As a result, during operation, the framework can seamlessly map the class attributes to an object relational database and provide persistent storage to the attributes.
A data type of an attribute is basic entity provided by the framework that can be persisted or transported in the object relational database. A data type is associated with an identifier (e.g., a name). A data type can be, persisted or ephemeral, configuration or operational and read-only or read-write. The framework can serialize or de-serialize a data type to or from: XML, remote procedure call (RPC), SQL, JavaScript Object Notation (JSON), and Open vS witch Database (OVSDB) management protocol.
The framework supports different categories of attributes. Such categories include, but are not limited to, integers and numbers, string, date and time, messaging, UML relations, network, and others. In addition, the framework supports user defined data types and corresponding attributes. Table 1 includes different categories of attributes and their corresponding data types supported by the framework. It should be noted that the categories and data types listed in Table 1 is not exhaustive. The framework can support more categories and data types.
The framework provides extensive list of built-in data types, as described in conjunction with Table 1. These data types capture the attribute values (e.g., data fields) of objects. In some embodiments, the framework includes one or more attributes that provide run time introspection that allows runtime identification of classes. Since attributes can be serialized to and de-serialized from a variety of formats, the framework provides extensive support for custom behavior overriding in serialization and de-serialization. Furthermore, the framework supports user defined data types.
Object Identifier
In the example in
However, a switch can have a plurality of line cards. For another line card, another object (i.e., another instance) of the class is created in the memory of switch 102, and framework 120 stores the attribute values of that other object in the table associated with the class in database 150. In this way, the same table can store attribute values of different objects of the same class. To identify different objects of a class in the table, framework 120 generates and assigns an object identifier (object ID or OID) to a respective object of a respective class. This object identifier operates as the primary identifier of that object. In the database table, this primary identifier is the primary key of that table. It should be noted that an object identifier is referred to be associated with a class in a generic sense, which indicates an object identifier of an object of the class.
In some embodiments, object identifier includes a class identifier (a class ID or CID) 220 and an instance identifier (an instance ID or IID) 230. Class identifier 220 represents the class from which the object is instantiated. In some embodiments, class identifier 220 is generated based on a hash function (e.g., Rabin Polynomial hash function) applied to the name of the class. Instance identifier 230 represents that particular instance of the object. Hence, if two objects of the same class are created, class identifier 220 of object identifier 210 remains the same for both the objects. However, the two objects differ in their respective instance identifier 230. Typically, class identifier 220 and instance identifier 230 are 32 and 64 bits long, respectively. However, these lengths can vary.
In some embodiments, instance identifier 230 includes a group identifier 232, a location identifier 234, a management module identifier 236, and a serial identifier 238. Group identifier 232 identifies a group in which the switch is a member. For example, if the switch is a member switch of a fabric switch, group identifier 232 can be a fabric switch identifier, which identifies a fabric switch. Location identifier 234 identifies the switch in the group. For example, if the switch is a member switch of a fabric switch, location identifier 234 can be a switch identifier, which identifies the switch within that fabric switch. Typically, group identifier 232 and location identifier 234 are 10 and 20 bits long, respectively.
Management module identifier 236 identifies the type of management module is operating the switch. For example, if the switch is participating in an active-standby high availability protocol (e.g., Virtual Router Redundancy Protocol (VRRP) and Virtual Switch Redundancy Protocol (VSRP)), management module identifier 236 can indicate whether the switch is an active or a standby switch. Typically, management module identifier 236 is 1 bit long. However, length of management module identifier 236 can be increased by incorporating adjacent bits from location identifier 234.
Serial identifier 238 provides identification of a specific instance of an object and can be a wrapped-around monotonically increasing number (e.g., an unsigned integer). Typically, serial identifier 238 is 32 bits long. In this way, object identifier 210 uniquely identifies an object of a class created by a management module in a switch, which can be in a fabric switch. In other words, object identifier 210 can be unique among objects, classes, management modules, fabric switches, and switch locations within a corresponding fabric switch.
Base Classes
In the example in
The framework supports a set of base classes and multiple inheritance from these base classes.
Class 304 outlines the most common attributes and operations of the objects managed by the framework. In other words, all class hierarchies derive virtually from the PersistableObject class. Since a class can inherit from multiple classes and each of these classes can inherit from the PersistableObject class, there can potentially be a conflict during execution of a managed object. This problem is generally referred to as the diamond problem. To solve this problem, the framework supports virtual derivation from the PersistableObject class. Another base class 306, denoted as the LocalManagedObjectBase class, outlines the attributes and operations locally managed within a switch. For example, a port is locally managed in a switch.
Base class 308, denoted as the LocalManagedObject class, is virtually derived from the ManagedObject class and the ManagedObjectBase class. Hence, the LocalManagedObject class outlines the attributes and operations of a switch which are locally and globally managed. For example, a port is locally managed within a switch and a VLAN configured for the port is managed globally. In some embodiments, an application (e.g., a protocol plug-in) running on a switch can specify more base classes for that application. Typically, base classes are not directly mapped to the tables of the object relational database. These base classes provide object relational mapping support. The attributes (i.e., the data fields) of these classes become part of a respective managed object derived from these base classes. As a result, the managed objects can share states and behavior.
In some embodiments, the attributes of a managed object can be any of the attribute data types supported by the framework, as described in conjunction with Table 1. The framework also supports vector variants (e.g., arrays and lists) for a number of the data types. Furthermore, the framework provides support to check whether a particular attribute is user configured. As described in conjunction with
Object Relational Mapping
In some embodiments, a persistent storage framework of a switch supports, both one-to-one and one-to-many, association, aggregation, and composition UML relationships. Association and aggregation are supported via ObjectID and ObjectIDVector data types, and ObjectIDAssociation and ObjectIDAssociationVector attributes, respectively. On the other hand, composition is supported via ManagedObectPointer and ManagedObectPointerVector data types and corresponding attributes. In some embodiments, the framework supports smart pointers and vector to facilitate seamless development.
Similarly, a switch can include one or more line cards. Hence, the Node class has a relationship with class 406, denoted as the LineCard class, which represents a line card. A line card can be in one switch and a switch includes (i.e., is composed of) a plurality of line cards. Hence, the relationship between the Node class and the LineCard class is a one-to-many composition, which is denoted as “includes.” On the other hand, a switch typically has a power source, which may not be inside of the switch. So, the Node class has a relationship with class 408, denoted as the PowerSource class, which represents a power source of a node. Suppose that, at a time, a power source can power one switch and a switch can receive power from one source. Hence, the relationship between the Node class and the PowerSource class is a one-to-one association, which is denoted as “getsPower.”
A power source can be based on alternating current (AC) or direct current (DC). So, class 408-A, denoted as the ACPowerSource class, and class 408-B, denoted as the DCPowerSource class, are derived from the PowerSource class. The ACPowerSource class and the DCPowerSource class represent AC and DC power sources, respectively. Hence, based on the getsPower association, a Node can get power from a generic power source, an AC power source, or a DC power source. In this UML diagram, since the relationship between the Node class and class 408 is one-to-one, an object of the Node class can have only one of the power sources. In this example, the PowerSource class, the ACPowerSource class, and the DCPowerSource class can be referred to as the inheritance chain of the PowerSource class (class 408).
The framework can receive the UML diagram of
XML definition 400 includes a respective attribute, such as NodeID, and its type (i.e., data type, as described in conjunction with Table 1). XML definition 400 also includes one-to-one and one-to-many relationships for which the Node class is a container class. For example, a node contains line cards. Hence, XML definition 400 specifies aggregation “includes” as an attribute, its type, and the class to which Node is related. It should be noted that the isMemberOf relationship is not represented in XML definition 400 even though the isMemberOf relationship to the Node class. This is because the container class for the isMemberOf relationship is the SwitchGroup class. Hence, the isMemberOf relationship is represented in an XML definition corresponding to the SwitchGroup class (not shown in
Persistent Storage in a Switch
Upon receiving XML definitions associated with the classes of a UML model, the framework creates a respective table for a respective class, their derivations, their instances (i.e., objects), and their one-to-many relationships in an object relational database.
Table 420 also includes a column for a respective attribute of the Node class. For example, table 420 includes a column 424 for a NodeID of the Node class. Furthermore, since the Node class has a one-to-one association with the PowerSource class, for which the Node class is the container class, the framework includes a column 425 for an object identifier of an object of the PowerSource class (i.e., an object associated with the PowerSource class). Column 425 includes two columns 426 and 427 for the class identifier and instance identifier, respectively, of the object identifier associated with the PowerSource class. The framework also creates a table 410 for the PowerSource class, comprising column 411 for the object identifier associated with the PowerSource class. Column 411 includes two columns 412 and 413 for the class identifier and instance identifier, respectively, of the object identifier of the PowerSource class.
Similarly, the framework also creates a table 440 for the ACPowerSource class, comprising column 441 for an object identifier of an object of the ACPowerSource class (i.e., an object associated with the ACPowerSource class). Column 441 includes two columns 442 and 443 for the class identifier and instance identifier, respectively, of the object identifier associated with the ACPowerSource class. In the same way, the framework also creates a table 450 for the DCPowerSource class, comprising column 451 for an object identifier of an object of the PowerSource class. Column 451 includes two columns 452 and 453 for the class identifier and instance identifier, respectively, of the object identifier associated with the DCPowerSource class.
In some embodiments, the framework creates auxiliary tables to enforce consistency on columns 426 and 427. For example, the framework creates an auxiliary table 430 for the derivations of the PowerSource class (e.g., based on the UML model in
When a class identifier is generated for any class of the inheritance chain of the PowerSource class, that class identifier is inserted into table 430. The framework identifies the PowerSource, the ACPowerSource, and the DCPowerSource classes of the inheritance chain of the PowerSource class from the UML model in
When an object identifier is inserted into table 410, 440, or 450, the instance identifier of the object identifier is concurrently inserted into table 460 (denoted with dotted arrow). Suppose that, upon detecting an object in the memory of the switch, the framework inserts an object identifier comprising a class identifier 432 and instance identifier 435 into table 410. Similarly, an object identifier comprising a class identifier 433 and instance identifier 444, and an object identifier comprising a class identifier 433 and instance identifier 445 are inserted into table 440. An object identifier comprising a class identifier 434 and instance identifier 454 is inserted into table 450. The framework ensures that instance identifiers 435, 444, 445, and 454 are also inserted into table 460. Column 461 of table 460 provides consistency enforcement to column 426 of table 420 (denoted with a dashed arrow). In some embodiments, consistency enforcement to column 427 is based on a foreign key constraint on column 461 of table 460.
During operation, an object of the Node class is instantiated in the memory of the switch. The framework identifies the object in the memory and generates an object identifier for the object comprising a class identifier 464 and an instance identifier 465. The framework identifies the attribute values of the object, which includes NodelD 466 and an object identifier of a power source object. Suppose that the power source for the switch is an AC power source, and the object identifier comprises a class identifier 433 and an instance identifier 444, as stored in table 440 corresponding to the ACPowerSource class. The framework creates an entry in table 420 by inserting class identifier 464, instance identifier 465, NodeID 466, class identifier 433, and instance identifier 444 into table 420. Since consistency is enforced on columns 426 and 427, the relational database ensures that class identifier 433 and instance identifier 444 appear in columns 431 and 461, respectively.
However, even though the Node class is related to the LineCard class, since it is a one-to-many relationship, table 420 does not include an object identifier associated with the LineCard class. The framework creates an auxiliary table to represent the relationship the Node class and the LineCard class.
During operation, an object of the LineCard class is instantiated in the memory of the switch. The framework identifies the object in memory and generates an object identifier comprising a class identifier 474 and an instance identifier 475 for the object. The framework then creates an entry in table 470 by inserting the object identifier into table 470. Similarly, the framework generates an object identifier comprising a class identifier 474 and an instance identifier 476 for another object of the LineCard class, and a third object identifier comprising a class identifier 474 and an instance identifier 477 for an object of the LineCard class. The framework creates respective entries in table 470 by inserting these object identifiers into table 470.
In some embodiments, the framework creates an auxiliary table 480 to represent the one-to-many “includes” relationship between the Node class and the LineCard class. In the relationship, the Node class is the container class and the LineCard class is the related class. Table 480 includes a column 481 for an object identifier associated with the Node class, and a column 484 for an object identifier associated with the LineCard class. Column 481 includes two columns 482 and 483 for the class identifier and instance identifier, respectively, of the object identifier associated with the Node class. Similarly, column 484 includes two columns 485 and 486 for the class identifier and instance identifier, respectively, of the object identifier associated with the LineCard class.
Suppose that the object of the Node class, which is associated with class identifier 464 and instance identifier 465, includes two line cards. Hence, the object of the Node class include two objects (e.g., an ManagedObjectVector) of the LineCard class. Suppose that instance identifiers 475 and 476 belong to these two objects. As a result, the framework inserts class identifier 464, instance identifier 465, class identifier 474, and instance identifier 475 into table 480. The framework also inserts class identifier 464, instance identifier 465, class identifier 474, and instance identifier 476 into table 480. In this way, the relationship between the object of the Node class (associated with instance identifier 465) and two objects of the LineCard class (associated with instance identifier 475 and 476) is stored in table 480.
In some embodiments, similar to tables 430 and 460, the framework creates auxiliary table 490 for the derivations of the Node class (e.g., based on the UML model in
In the same way, the framework creates auxiliary table 495 for the derivations of the LineCard class (and its derivations, if any). Table 495 includes a column 496 for the class identifier associated with the derivations of the LineCard class. Similarly, the framework creates an auxiliary table 497 for the objects instantiated from the derivations of the LineCard class. In this example, table 497 corresponds to the LineCard class (and its derivations, if any). Table 497 includes a column 498 for the instance identifiers of the objects instantiated from the derivations of the LineCard class.
When a class identifier is generated for the Node class or the LineCard class, that class identifier is inserted into table 490 or 495, respectively. The framework inserts class identifiers 464 and 474 associated with the Node and the LineCard classes, respectively, into tables 490 and 495, respectively. In this example, an entry in a table is denoted with dotted lines. Column 491 of table 490 provides consistency enforcement to column 482 of table 480 (denoted with a dashed arrow). In some embodiments, consistency enforcement of column 482 is based on a foreign key constraint on column 491 of table 490. In the same way, column 496 of table 495 provides consistency enforcement to column 485 of table 480 (denoted with a dashed arrow). In some embodiments, consistency enforcement of column 485 is based on a foreign key constraint on column 496 of table 495.
On the other hand, when the framework identifies objects of the Node or the LineCard class, the framework generates a corresponding object identifier and inserts the object identifier, comprising a class identifier and an instance identifier, into table 420 or 470, respectively. When an object identifier is inserted into table 420 or 470, the instance identifier of the object identifier is concurrently inserted into table 492 or 497, respectively (denoted with dotted arrow). For example, when the framework inserts an object identifier comprising a class identifier 464 and instance identifier 465 into table 420, instance identifier 465 is inserted into table 492. In the same way, when the framework inserts an object identifier comprising a class identifier 474 and instance identifier 475 into table 470, instance identifier 475 is inserted into table 497.
Similar to table 480, the framework also creates an auxiliary table to represent the one-to-many “isMemberOf” relationship between the Node class and the SwitchGroup class, as described in conjunction with
It should be noted that the framework distinguishes between a composition relationship (e.g., “includes” in
Operations of a Persistent Storage Framework
The framework identifies an attribute of the identified class (operation 536). It should be noted that the relationships for which the class is a container class are can also be attributes, as described in conjunction with
The framework enforces consistency on the class identifier and the instance identifier based on the corresponding auxiliary tables of the related classes (operation 548), as described in conjunction with
The framework adds a column for an object identifier comprising columns for class identifier and instance identifier of the container class (operation 564), as described in conjunction with
Distributed Configuration Management
In the example in
With existing technologies, a respective switch in switch group 100 is configured individually and the configuration process is repeated for a respective switch in switch group 100. Such configuration can be related to network virtualizations, partitions, and switch groups, and a plurality of network protocols of different network layers. The attribute values (e.g., parameters) of the configuration are applied to a respective switch, such as switch 102 (e.g., loaded to the switch modules, such as the processing hardware of switch 102). This process is repeated for switches 101, 103, 104, and 105. This process can be repetitive, tedious, and error-prone (e.g., human errors, such spelling mistake).
Furthermore, some of the configuration information is applicable for entire switch group 100 (e.g., a VLAN for switch group 100). Such configuration can be referred to as global configuration. This global configuration is repeated for a respective switch in switch group 100. This can lead to inconsistency among the global configuration in different switches in switch group 100. For example, the user repeats global configuration for switch 101, 102, 103, 104, and 105 in switch group 100. The user may, mistakenly, configure VLAN 110 in switch 102 and VLAN 111 in switch 101. As a result, traffic belonging to VLAN 110 may be erroneously dropped at switch 101.
To solve this problem, one of the switches in switch group 100 is elected as a principal switch, which manages and distributes configuration information in switch group 100. Suppose that switch 102 is elected as the principal switch. In some embodiments, a respective switch in switch group 100 broadcasts its information in switch group 100, and the switch with the lowest (or highest) identifier value is elected as the principal switch. Other switches of switch group 100, such as switches 101, 103, 104, and 105, can be referred to as managed switches. When the user accesses switch 102 (e.g., via end device 112 or 114), switch 102 allows the user to configure the global configuration of switch group 100. Switch 102, in turn, provides that global configuration to switches 101, 103, 104, and 105. Furthermore, switch 102 also allows the user to configure switches 101, 103, 104, and 105 in the switch group. Switch 102, in turn, provides the corresponding local configuration information to switches 101, 103, 104, and 105, respectively.
In some embodiments, a respective switch of switch group 100 is equipped with a persistent storage framework, as described in conjunction with
Since switch 102 can also be accessed via console port 604 via end device 614, the user can configure switches of switch group 100 without the presence of a management network. It should be noted that, even though a user can configure the switches of switch group 100 from principal switch 102, the user can obtain the local configuration information of a respective switch of switch group 100 via any of the member switches of the switch group. For example, switch 101 can be accessed by logging in to switch 101 using a management IP address 631 via a management port 606. The user can use an end device 616 to log in to switch 101 based on management IP address 631. Since switch 101 includes the configuration information in object relational database 640 of switch 101, the user can obtain the global configuration information as well the local configuration information of a respective switch of switch group 100 from switch 101.
In some embodiments, switch group 100 is associated with a virtual IP address 610. During operation, virtual IP address 610 is associated with the principal switch of switch group 100. As a result, the user can access principal switch of group 100 via virtual IP address 610 without knowing the management IP address 632. Furthermore, if switch 102 becomes unavailable (e.g., due to a failure), another switch in switch group 100, such as switch 101, is elected as the principal switch and virtual IP address 610 becomes associated with switch 101. Since switch 101 has the configuration information in database 640 is the same as the configuration information in database 150, switch 101 can continue to facilitate distributed configuration for the switches of switch group 100. As a result, the user can continue to access the principal switch of switch group 100, which is switch 101, via virtual IP address 610 and configure the switches of switch group 100.
In some embodiments, a respective switch in switch group 100 is associated with an internal control IP address (ICIPA), which is used for communication among member switches of switch group 100. For example, an internal control IP address can be used for communication via an inter-switch port. When the principal switch of a switch group provides the configuration information to a managed switch, the principal switch communicates with that managed switch based on respective internal control IP addresses. Suppose that internal control IP addresses 621 and 622 are associated with switches 101 and 102, respectively. When switch 102 provides the configuration information to switch 101, switch 102 uses IP addresses 621 and 622 as the destination and source addresses for the packets comprising the configuration information.
In some embodiments, a respective switch in a switch group can receive configuration information from a user.
Suppose that configuration information 662 and 664 can cause a conflict in the switch. For example, configuration information 662 and 664 can be associated with conflicting quality of service settings for switch 105. If both switches 101 and 103 are allowed to operate as a principal switch, switches 101 and 103 can individually apply these conflicting configurations to switch 105. This can lead to race condition in switch group 100. To solve this problem, a switch group internally has one principal switch, and other switches forward their received configuration information to that principal switch. That principal switch receives the configuration information, serializes them, resolves any conflict, and applies them based on distributed commit. In this way, it would appear to the user that the switch group is being configured from any of the member switches, though internally, the switch group is configured from the principal switch.
For example, switches 101 and 103 forward configuration information 662 and 664, respectively, to principal switch 102 of switch group 100. Switch 102 receives configuration information 662 and 664, and serializes them (e.g., based on the order of arrival or an associated priority). In some embodiments, if switch 102 detects any conflict, switch 102 resolves the conflict based on its serialization. For example, configuration information 664 is serialized before configuration information 662, switch 102 resolves the conflict by applying the configuration of configuration information 664 to switch 105 based on distributed commit, and discarding configuration information 662. In this way, it would appear to the user that switch 105 is being configured switches 101 and/or 103, though internally, switch 102 configures switch 105.
In some embodiments, in switch group 100, different member switch can be the principal switch for different services. A service can be a modular component which manages a related set of configuration. In the example in
During operation, a user logs in to switch 102 using management IP address 632 and provides configuration information 666, which is associated with port management (e.g., creating a port channel). Even though switch 102 is a principal switch, switch 102 forwards configuration information 666 to switch 101 since switch 101 is the principal switch for port management. On the other hand, suppose that configuration information 662 is associated with network virtualization (e.g., creating a new VLAN). Even though switch 101 is a principal switch, switch 101 forwards configuration information 662 to switch 102 since switch 102 is the principal switch for network virtualization.
Interoperability of Software Versions
In the example in
Typically, member switches of switch group 100 have the same software version (e.g., either software version 1 or 2). This allows information from one switch to be synchronized with other member switches. If a switch has an upgraded software version, the switch leaves the switch group until other switches are upgraded as well. As a result, switches 101 and 103 may not be in the same switch group as switches 102, 104, and 105. This is because an attribute field can be different for different software versions. For example, if principal switch 102 tries to synchronize that different value of the attribute field with switch 101, an inconsistency can occur. Hence, a software version upgrade may not be performed gradually while keeping a respective member switch operational in switch group 100. This leads to unavailability and reduced operational capability of switch group 100.
To solve this problem, a respective member switch of switch group 100 uses a portable representation (e.g., an XML representation) to synchronize its update information. The information in a portable representation is recognizable by principal switch 102 of switch group 100. Since both local and global configuration of switch group 100 is replicated at a respective member switch, to ensure interoperability, principal switch 102 synchronizes attribute values of different software versions. Suppose that nodeID of class Node in
Upon receiving the notification message, principal switch 102 extracts the portable presentation from the message and obtains the integer value of the nodeID from the portable representation. Principal switch 102 creates a respective instruction to update the corresponding attribute (i.e., the nodeID of switch 101) with the received integer value for software versions 1 and 2 in the switch group. In some embodiments, this instruction is an SQL transaction executable in the persistent storages of the switches with the corresponding software version. For software version 1, the instruction can be an SQL transaction executable in database 640. Since the received nodeID is compatible with software version 1, principal switch 102 can directly incorporate the received nodeID in the instruction for software version 1.
However, since the received value is an integer and the nodeID in software version 2 is a string, principal switch 102, if supported, converts the nodeID from integer to a string. If not supported, principal switch 102 can use a default value for the string. Principal switch 102 incorporates the converted nodeID (i.e., the nodeID string) in the instruction for software version 2. Principal switch sends the instruction for software version 1 to switches 101 and 103, and the instruction for software version 2 to switches 102, 104, and 105. In this way, the configuration information of a respective member switch, even with different software versions 1 and 2, remains synchronized. As a result, both local and global configuration of switch group 100 is replicated at a respective member switch, even with a different software version, of switch group 100.
In some embodiments, the instructions from principal switch 102 are executed in a distributed commit. During operation, principal switch 102 creates a respective instruction to prepare the update information (e.g., an SQL transaction) for software versions 1 and 2 in the corresponding persistent storage, and sends the instruction for software version 1 to switches 101 and 103, and the instruction for software version 2 to switches 102, 104, and 105. A respective managed switch receives the instruction for the local software version, prepares the update information (e.g., the SQL transaction) specified in the instruction in a local persistent storage, and acknowledges the preparation (e.g., with a corresponding transaction identifier).
Upon receiving an acknowledgement from a respective managed switch, principal switch 102 sends an instruction to a respective switch (including the local switch) to commit the transaction in the local persistent storage (e.g., database 640 in switch 101 and database 150 in switch 102) and the local switch modules (e.g., switch ASICs). If one of the switches does not acknowledge the preparation, the transaction is not executable in that switch, and hence, cannot be synchronized in switch group 100. Under such a circumstance, principal switch 102 instructs the managed switches with a successful preparation to roll back the transaction.
It should be noted that, to ensure interoperability, a member switch with the most updated software version (e.g., version 2) is elected as the principal switch of switch group 100. In some embodiments, a principal switch is selected based on a priority value. The member switches with the most updated software version is configured with (e.g., automatically or by a user) a higher priority value than the member switches with an older software version. In the example in
XML definitions 722 and 732 include values for a respective attribute, such as nodeID, of the Node class. XML definitions 722 and 732 also include one-to-one and one-to-many relationships for which the Node class is a container class, as described in conjunction with
Suppose that, in software version 2, attribute type of the nodeID is of type string and the power source type is of AC power source (e.g., an object of the ACPowerSource class). Hence, if a switch with software version 2, such as switch 104 of
Synchronization Management
As described in conjunction with
In switch group 100, a respective switch can receive update information (e.g., update to local or global configuration information), as described in conjunction with
Upon receiving the portable representation, switch 102 converts the attribute values in the portable representation to corresponding attribute values of a respective software version, as described in conjunction with
Switch 102 receives acknowledgement from a respective managed switch (operation 759). Switch 102 then determines that the transaction is prepared at a respective switch and ready to be committed. Switch 102 sends an instruction to a respective switch (including the local switch) to commit the transaction in respective local persistent storage and switch modules. For example, switch 102 sends an instruction to switch 101 to commit the transaction (operation 760). Upon receiving the instruction, switch 101 commit the transaction in the local database (i.e., in table 420-1 of database 640) and switch modules (operation 761).
Operations
The switch creates a notification message comprising the portable representation (operation 806). This notification message can be a message of a notification service of a switch group in which the switch is a member. The switch encapsulates the notification message in a fabric encapsulation and sets the identifier of the principal switch as the egress switch identifier or the encapsulation header (operation 808). The identifier of the principal switch can be one or more of: a MAC address, an RBridge identifier, and an IP address. The switch then identifies an egress port associated with the identifier of the principal switch (e.g., from a local forwarding table) and transmits the notification message via the identified port (operation 810).
The switch generates an instruction message instructing to prepare the persistent storage for the update information expressed for a respective identified software version (operation 838). In some embodiments, the switch uses different namespace to maintain a copy of the same class in a respective software version. Based on that copy, the switch generates the instruction for the corresponding software version. The switch identifies the switch(es) of one of the identified software versions (e.g., from the mapping) (operation 840) and sends the corresponding instruction message to a respective identified switch (operation 842). The switch checks whether it has sent the instruction for all software versions (operation 844). If not, the switch identifies the switch(es) of another identified software versions (operation 840) and sends the corresponding instruction message to a respective identified switch (operation 842). In some embodiments, the switch prepares the update information in the local persistent storage as well.
Upon sending the instruction for all software versions, the switch receives a response from a respective managed switch (operation 846) and checks whether all persistent storages have been prepared (operation 848). If all persistent storages have not been prepared (i.e., one or more switches have not prepared the persistent storage, for example, due to a failure), the switch rolls back the configuration information from the local persistent storage (operation 850). The switch then generates a control message instructing to roll back the configuration information from the persistent storage and sends the message to a respective successful managed switch (i.e., a switch with successful storage preparation) (operation 852). In other words, the switch may not send the control message to the managed switches which have not succeeded in preparing their corresponding local persistent storage.
If all persistent storages have been prepared, the switch generates a control message instructing to commit the configuration information to the persistent storage and the switch modules, and sends the message to a respective switch (operation 854). It should be noted that a respective switch also includes the local switch, wherein the switch instructs the local switch to commit the configuration information. Commit refers to finalizing information (i.e., ending the current transaction and making the configuration information permanent) in the switch modules or in the persistent storage. By checking whether all persistent storages have been prepared (operation 848), the switch determines whether the configuration information is eligible for a distributed commit in the switch group. If the switch determines eligibility, the switch instructs the managed switches to commit. This process can be referred to as a distributed commit.
If a version conversion is needed, the switch then determines a namespace for the determined software version (operation 866) and obtains the class chain associated with the update information in the determined namespace (operation 868). In some embodiments, the namespace is a programming language namespace. Examples of such namespace include, but are not limited to, C++, Java, C#, Python, and PHP namespace. In the example in
If a version conversion is not needed (operation 864) or the class chain has been instantiated (operation 870), the switch assigns values to the unchanged attributes in the target update information (i.e., the update information expressed for the target software version) (operation 872). The switch then identifies an attribute type difference (e.g., between an integer and a string) with the originating software version of the originating switch (i.e., the switch which has originated the portable representation) (operation 874). In the example in
If a type conversion is supported, the switch converts the value of the corresponding attribute type in the update information to the target attribute type and include in the target update information (operation 878). The target attribute type is the corresponding attribute type in the software version of the target managed switch(es). Suppose that, in the example in
If all attribute type difference has been determined, the switch assigns a default value to the attributes which are not present in the originating software version (i.e., the software version of the switch originating the portable representation) (operation 884). In the example in
Exemplary Switch
In some embodiments, switch 900 maintains a membership in a switch group, as described in conjunction with
Communication ports 902 can include inter-switch communication channels for communication within the fabric switch. This inter-switch communication channel can be implemented via a regular communication port and based on any open or proprietary format. Communication ports 902 can also include one or more extension communication ports for communication between neighbor fabric switches. Communication ports 902 can include one or more TRILL ports capable of receiving frames encapsulated in a TRILL header. Communication ports 902 can also include one or more IP ports capable of receiving IP packets. An IP port is capable of receiving an IP packet and can be configured with an IP address. Packet processor 910 can process TRILL-encapsulated frames and/or IP packets.
During operation, persistent storage module 930 stores configuration information associated with the switch group in a table, which includes one or more columns for attribute values of the configuration information, in object relational database 940 in storage device 950. If switch 900 is a principal switch, synchronization module 932 obtains update information from a portable representation, which is extracted from a notification message received via one of communication ports 902. The update information includes a first set of attribute values for a first table (i.e., a first data structure) in database 940. Synchronization module 932 then identifies a software version of a second switch in the switch group and generates an update instruction instructing to update the first table in a persistent storage in the second switch with the target update information. It should be noted that the software version of the second switch can be the same as switch 900, and the first and second sets of attribute values can be the same.
Synchronization module 932 can identify the software version of the remote switch from a mapping between an identifier of the remote switch and the software version. In some embodiments, switch 900 also includes a conversion module 970, which converts a first attribute value to a second attribute value (e.g., from an integer to a string). Conversion module 970 assigns a default attribute value to an attribute in the target update information (e.g., if conversion is not supported or for attributes without a value specified in the portable representation).
In some embodiments, synchronization module 932 verifies whether the target update information is prepared for storage in a persistent storage of a respective switch of the switch group. If the update information is prepared, synchronization module 932 generates an instruction message instructing to commit the target update information to the persistent storage and switch modules of a respective switch of the switch group. If the update information is not prepared, synchronization module 932 rolls back the target update information from a persistent storage of a switch, which has succeeded in storage preparation, of the switch group.
If switch 900 is a managed switch, synchronization module 932 identifies update information from a received user command. The update information comprises a first set of attribute values for a first table in database 640. Synchronization module 932 generates a portable representation, which is recognizable by the principal switch of the switch group, of the update information. Synchronization module 932 also generates a notification message comprising the portable representation. This notification message is destined for the principal switch. In some embodiments, synchronization module 932 verifies whether the update information is prepared for storage in the local persistent storage. If the update information is prepared, synchronization module 932 generates an acknowledgement message comprising a transaction identifier associated with the preparation.
Note that the above-mentioned modules can be implemented in hardware as well as in software. In one embodiment, these modules can be embodied in computer-executable instructions stored in a memory which is coupled to one or more processors in switch 900. When executed, these instructions cause the processor(s) to perform the aforementioned functions.
In summary, embodiments of the present invention provide a first switch and a method which supports multiple software versions in a switch group. In one embodiment, the first switch includes one or more ports, a switch group module, a persistent storage module, and a synchronization module. The switch group module maintains a membership in a switch group. The switch group includes a plurality of switches and operates as a single switch. The persistent storage module stores configuration information associated with the switch group in a table, which includes one or more columns for attribute values of the configuration information, in a local persistent storage. The synchronization module obtains update information from a portable representation. This update information comprises a first set of attribute values for a first data structure in the persistent storage. The synchronization module identifies a software version of a second switch in the switch group and generates an update instruction to update the first data structure in a persistent storage in the second switch with target update information. The target update information comprises a second set of attribute values updated from the first set of attribute values.
The methods and processes described herein can be embodied as code and/or data, which can be stored in a computer-readable non-transitory storage medium. When a computer system reads and executes the code and/or data stored on the computer-readable non-transitory storage medium, the computer system performs the methods and processes embodied as data structures and code and stored within the medium.
The methods and processes described herein can be executed by and/or included in hardware modules or apparatus. These modules or apparatus may include, but are not limited to, an application-specific integrated circuit (ASIC) chip, a field-programmable gate array (FPGA), a dedicated or shared processor that executes a particular software module or a piece of code at a particular time, and/or other programmable-logic devices now known or later developed. When the hardware modules or apparatus are activated, they perform the methods and processes included within them.
The foregoing descriptions of embodiments of the present invention have been presented only for purposes of illustration and description. They are not intended to be exhaustive or to limit this disclosure. Accordingly, many modifications and variations will be apparent to practitioners skilled in the art. The scope of the present invention is defined by the appended claims.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5390173 | Spinney | Feb 1995 | A |
| 5802278 | Isfeld | Sep 1998 | A |
| 5878232 | Marimuthu | Mar 1999 | A |
| 5879173 | Poplawski | Mar 1999 | A |
| 5959968 | Chin | Sep 1999 | A |
| 5973278 | Wehrli, III et al. | Oct 1999 | A |
| 5983278 | Chong | Nov 1999 | A |
| 5995262 | Hirota | Nov 1999 | A |
| 6041042 | Bussiere | Mar 2000 | A |
| 6085238 | Yuasa | Jul 2000 | A |
| 6092062 | Lohman | Jul 2000 | A |
| 6104696 | Kadambi | Aug 2000 | A |
| 6185214 | Schwartz | Feb 2001 | B1 |
| 6185241 | Sun | Feb 2001 | B1 |
| 6295527 | McCormack | Sep 2001 | B1 |
| 6331983 | Haggerty | Dec 2001 | B1 |
| 6438106 | Pillar | Aug 2002 | B1 |
| 6498781 | Bass | Dec 2002 | B1 |
| 6542266 | Phillips | Apr 2003 | B1 |
| 6553029 | Alexander | Apr 2003 | B1 |
| 6571355 | Linnell | May 2003 | B1 |
| 6583902 | Yuen | Jun 2003 | B1 |
| 6633761 | Singhal | Oct 2003 | B1 |
| 6636963 | Stein | Oct 2003 | B1 |
| 6771610 | Seaman | Aug 2004 | B1 |
| 6870840 | Hill | Mar 2005 | B1 |
| 6873602 | Ambe | Mar 2005 | B1 |
| 6937576 | DiBenedetto | Aug 2005 | B1 |
| 6956824 | Mark | Oct 2005 | B2 |
| 6957269 | Williams | Oct 2005 | B2 |
| 6975581 | Medina | Dec 2005 | B1 |
| 6975864 | Singhal | Dec 2005 | B2 |
| 7016352 | Chow | Mar 2006 | B1 |
| 7061877 | Gummalla | Jun 2006 | B1 |
| 7062177 | Grivna | Jun 2006 | B1 |
| 7173934 | Lapuh | Feb 2007 | B2 |
| 7197308 | Singhal | Mar 2007 | B2 |
| 7206288 | Cometto | Apr 2007 | B2 |
| 7310664 | Merchant | Dec 2007 | B1 |
| 7313637 | Tanaka | Dec 2007 | B2 |
| 7316031 | Griffith | Jan 2008 | B2 |
| 7330897 | Baldwin | Feb 2008 | B2 |
| 7380025 | Riggins | May 2008 | B1 |
| 7397794 | Lacroute | Jul 2008 | B1 |
| 7430164 | Bare | Sep 2008 | B2 |
| 7453888 | Zabihi | Nov 2008 | B2 |
| 7477894 | Sinha | Jan 2009 | B1 |
| 7480258 | Shuen | Jan 2009 | B1 |
| 7508757 | Ge | Mar 2009 | B2 |
| 7558195 | Kuo | Jul 2009 | B1 |
| 7558273 | Grosser | Jul 2009 | B1 |
| 7571447 | Ally | Aug 2009 | B2 |
| 7599901 | Mital | Oct 2009 | B2 |
| 7688960 | Aubuchon | Mar 2010 | B1 |
| 7690040 | Frattura | Mar 2010 | B2 |
| 7716370 | Devarapalli | May 2010 | B1 |
| 7720076 | Dobbins | May 2010 | B2 |
| 7729296 | Choudhary | Jun 2010 | B1 |
| 7787480 | Mehta | Aug 2010 | B1 |
| 7792920 | Istvan | Sep 2010 | B2 |
| 7796593 | Ghosh | Sep 2010 | B1 |
| 7801021 | Triantafillis | Sep 2010 | B1 |
| 7808992 | Homchaudhuri | Oct 2010 | B2 |
| 7836332 | Hara | Nov 2010 | B2 |
| 7843906 | Chidambaram et al. | Nov 2010 | B1 |
| 7843907 | Abou-Emara | Nov 2010 | B1 |
| 7860097 | Lovett | Dec 2010 | B1 |
| 7898959 | Arad | Mar 2011 | B1 |
| 7912091 | Krishnan | Mar 2011 | B1 |
| 7924837 | Shabtay | Apr 2011 | B1 |
| 7937438 | Miller | May 2011 | B1 |
| 7937756 | Kay | May 2011 | B2 |
| 7945941 | Sinha | May 2011 | B2 |
| 7949638 | Goodson | May 2011 | B1 |
| 7957386 | Aggarwal | Jun 2011 | B1 |
| 8018938 | Fromm | Sep 2011 | B1 |
| 8027354 | Portolani | Sep 2011 | B1 |
| 8054832 | Shukla | Nov 2011 | B1 |
| 8068442 | Kompella | Nov 2011 | B1 |
| 8078704 | Lee | Dec 2011 | B2 |
| 8090805 | Chawla | Jan 2012 | B1 |
| 8102781 | Smith | Jan 2012 | B2 |
| 8102791 | Tang | Jan 2012 | B2 |
| 8116307 | Thesayi | Feb 2012 | B1 |
| 8125928 | Mehta | Feb 2012 | B2 |
| 8134922 | Elangovan | Mar 2012 | B2 |
| 8155150 | Chung | Apr 2012 | B1 |
| 8160063 | Maltz | Apr 2012 | B2 |
| 8160080 | Arad | Apr 2012 | B1 |
| 8170038 | Belanger | May 2012 | B2 |
| 8175107 | Yalagandula | May 2012 | B1 |
| 8194674 | Pagel | Jun 2012 | B1 |
| 8195774 | Lambeth | Jun 2012 | B2 |
| 8204061 | Sane | Jun 2012 | B1 |
| 8213313 | Doiron | Jul 2012 | B1 |
| 8213336 | Smith | Jul 2012 | B2 |
| 8230069 | Korupolu | Jul 2012 | B2 |
| 8239960 | Frattura | Aug 2012 | B2 |
| 8249069 | Raman | Aug 2012 | B2 |
| 8270401 | Barnes | Sep 2012 | B1 |
| 8295291 | Ramanathan | Oct 2012 | B1 |
| 8295921 | Wang | Oct 2012 | B2 |
| 8301686 | Appajodu | Oct 2012 | B1 |
| 8339994 | Gnanasekaran | Dec 2012 | B2 |
| 8351352 | Eastlake | Jan 2013 | B1 |
| 8369335 | Jha | Feb 2013 | B2 |
| 8369347 | Xiong | Feb 2013 | B2 |
| 8392496 | Linden | Mar 2013 | B2 |
| 8451717 | Venkataraman et al. | May 2013 | B2 |
| 8462774 | Page | Jun 2013 | B2 |
| 8520595 | Yadav | Aug 2013 | B2 |
| 8599850 | Jha | Dec 2013 | B2 |
| 8599864 | Chung | Dec 2013 | B2 |
| 8615008 | Natarajan | Dec 2013 | B2 |
| 8619788 | Sankaran | Dec 2013 | B1 |
| 8705526 | Hasan | Apr 2014 | B1 |
| 8706905 | McGlaughlin | Apr 2014 | B1 |
| 8717895 | Koponen | May 2014 | B2 |
| 8724456 | Hong | May 2014 | B1 |
| 8798045 | Aybay | Aug 2014 | B1 |
| 8804736 | Drake | Aug 2014 | B1 |
| 8806031 | Kondur | Aug 2014 | B1 |
| 8826385 | Congdon | Sep 2014 | B2 |
| 8918631 | Kumar | Dec 2014 | B1 |
| 8937865 | Kumar | Jan 2015 | B1 |
| 8995272 | Agarwal | Mar 2015 | B2 |
| 9178793 | Marlow | Nov 2015 | B1 |
| 9350680 | Thayalan | May 2016 | B2 |
| 9438447 | Basso | Sep 2016 | B2 |
| 20010005527 | Vaeth | Jun 2001 | A1 |
| 20010055274 | Hegge | Dec 2001 | A1 |
| 20020021701 | Lavian | Feb 2002 | A1 |
| 20020039350 | Wang | Apr 2002 | A1 |
| 20020054593 | Morohashi | May 2002 | A1 |
| 20020087723 | Williams | Jul 2002 | A1 |
| 20020091795 | Yip | Jul 2002 | A1 |
| 20030026290 | Umayabashi | Feb 2003 | A1 |
| 20030041085 | Sato | Feb 2003 | A1 |
| 20030093567 | Lolayekar | May 2003 | A1 |
| 20030097464 | Martinez | May 2003 | A1 |
| 20030097470 | Lapuh | May 2003 | A1 |
| 20030123393 | Feuerstraeter | Jul 2003 | A1 |
| 20030147385 | Montalvo | Aug 2003 | A1 |
| 20030174706 | Shankar | Sep 2003 | A1 |
| 20030189905 | Lee | Oct 2003 | A1 |
| 20030189930 | Terrell | Oct 2003 | A1 |
| 20030208616 | Laing | Nov 2003 | A1 |
| 20030216143 | Roese | Nov 2003 | A1 |
| 20030223428 | BlanquerGonzalez | Dec 2003 | A1 |
| 20030233534 | Bernhard | Dec 2003 | A1 |
| 20040001433 | Gram | Jan 2004 | A1 |
| 20040003094 | See | Jan 2004 | A1 |
| 20040008868 | Bornowski | Jan 2004 | A1 |
| 20040010600 | Baldwin | Jan 2004 | A1 |
| 20040049699 | Griffith | Mar 2004 | A1 |
| 20040057430 | Paavolainen | Mar 2004 | A1 |
| 20040081171 | Finn | Apr 2004 | A1 |
| 20040095900 | Siegel | May 2004 | A1 |
| 20040117508 | Shimizu | Jun 2004 | A1 |
| 20040120326 | Yoon | Jun 2004 | A1 |
| 20040165595 | Holmgren | Aug 2004 | A1 |
| 20040165596 | Garcia | Aug 2004 | A1 |
| 20040205234 | Barrack | Oct 2004 | A1 |
| 20040213232 | Regan | Oct 2004 | A1 |
| 20040225725 | Enomoto | Nov 2004 | A1 |
| 20040243673 | Goyal | Dec 2004 | A1 |
| 20050007951 | Lapuh | Jan 2005 | A1 |
| 20050044199 | Shiga | Feb 2005 | A1 |
| 20050074001 | Mattes | Apr 2005 | A1 |
| 20050094568 | Judd | May 2005 | A1 |
| 20050094630 | Valdevit | May 2005 | A1 |
| 20050108375 | Hallak-Stamler | May 2005 | A1 |
| 20050122979 | Gross | Jun 2005 | A1 |
| 20050152335 | Lodha | Jul 2005 | A1 |
| 20050157751 | Rabie | Jul 2005 | A1 |
| 20050169188 | Cometto | Aug 2005 | A1 |
| 20050195813 | Ambe | Sep 2005 | A1 |
| 20050207423 | Herbst | Sep 2005 | A1 |
| 20050213561 | Yao | Sep 2005 | A1 |
| 20050220096 | Friskney | Oct 2005 | A1 |
| 20050259586 | Hafid | Nov 2005 | A1 |
| 20050265356 | Kawarai | Dec 2005 | A1 |
| 20050278565 | Frattura | Dec 2005 | A1 |
| 20060007869 | Hirota | Jan 2006 | A1 |
| 20060018302 | Ivaldi | Jan 2006 | A1 |
| 20060029055 | Perera | Feb 2006 | A1 |
| 20060034292 | Wakayama | Feb 2006 | A1 |
| 20060036648 | Frey | Feb 2006 | A1 |
| 20060036765 | Weyman | Feb 2006 | A1 |
| 20060059163 | Frattura | Mar 2006 | A1 |
| 20060062187 | Rune | Mar 2006 | A1 |
| 20060072550 | Davis | Apr 2006 | A1 |
| 20060083172 | Jordan | Apr 2006 | A1 |
| 20060083254 | Ge | Apr 2006 | A1 |
| 20060098589 | Kreeger | May 2006 | A1 |
| 20060126511 | Youn | Jun 2006 | A1 |
| 20060140130 | Kalkunte | Jun 2006 | A1 |
| 20060168109 | Warmenhoven | Jul 2006 | A1 |
| 20060184937 | Abels | Aug 2006 | A1 |
| 20060209886 | Silberman | Sep 2006 | A1 |
| 20060221960 | Borgione | Oct 2006 | A1 |
| 20060227776 | Chandrasekaran | Oct 2006 | A1 |
| 20060235995 | Bhatia | Oct 2006 | A1 |
| 20060242311 | Mai | Oct 2006 | A1 |
| 20060242398 | Fontijn | Oct 2006 | A1 |
| 20060245439 | Sajassi | Nov 2006 | A1 |
| 20060251067 | DeSanti | Nov 2006 | A1 |
| 20060256767 | Suzuki | Nov 2006 | A1 |
| 20060265515 | Shiga | Nov 2006 | A1 |
| 20060285499 | Tzeng | Dec 2006 | A1 |
| 20060291480 | Cho | Dec 2006 | A1 |
| 20060294413 | Filz | Dec 2006 | A1 |
| 20070036178 | Hares | Feb 2007 | A1 |
| 20070053294 | Ho | Mar 2007 | A1 |
| 20070074052 | Hemmah | Mar 2007 | A1 |
| 20070081530 | Nomura | Apr 2007 | A1 |
| 20070083625 | Chamdani | Apr 2007 | A1 |
| 20070086362 | Kato | Apr 2007 | A1 |
| 20070094464 | Sharma | Apr 2007 | A1 |
| 20070097968 | Du | May 2007 | A1 |
| 20070098006 | Parry | May 2007 | A1 |
| 20070116224 | Burke | May 2007 | A1 |
| 20070116422 | Reynolds | May 2007 | A1 |
| 20070130295 | Rastogi | Jun 2007 | A1 |
| 20070156659 | Lim | Jul 2007 | A1 |
| 20070177525 | Wijnands | Aug 2007 | A1 |
| 20070177597 | Ju | Aug 2007 | A1 |
| 20070183313 | Narayanan | Aug 2007 | A1 |
| 20070206762 | Chandra | Sep 2007 | A1 |
| 20070211712 | Fitch | Sep 2007 | A1 |
| 20070220059 | Lu | Sep 2007 | A1 |
| 20070226214 | Smits | Sep 2007 | A1 |
| 20070238343 | Velleca | Oct 2007 | A1 |
| 20070258449 | Bennett | Nov 2007 | A1 |
| 20070274234 | Kubota | Nov 2007 | A1 |
| 20070280223 | Pan | Dec 2007 | A1 |
| 20070289017 | Copeland, III | Dec 2007 | A1 |
| 20080052487 | Akahane | Feb 2008 | A1 |
| 20080056135 | Lee | Mar 2008 | A1 |
| 20080057918 | Abrant | Mar 2008 | A1 |
| 20080065760 | Damm | Mar 2008 | A1 |
| 20080080517 | Roy | Apr 2008 | A1 |
| 20080095160 | Yadav | Apr 2008 | A1 |
| 20080101386 | Gray | May 2008 | A1 |
| 20080112133 | Torudbakken | May 2008 | A1 |
| 20080133760 | Berkvens | Jun 2008 | A1 |
| 20080159277 | Vobbilisetty | Jul 2008 | A1 |
| 20080172492 | Raghunath | Jul 2008 | A1 |
| 20080181196 | Regan | Jul 2008 | A1 |
| 20080186981 | Seto | Aug 2008 | A1 |
| 20080205377 | Chao | Aug 2008 | A1 |
| 20080219172 | Mohan | Sep 2008 | A1 |
| 20080225852 | Raszuk | Sep 2008 | A1 |
| 20080225853 | Melman | Sep 2008 | A1 |
| 20080240129 | Elmeleegy | Oct 2008 | A1 |
| 20080267179 | LaVigne | Oct 2008 | A1 |
| 20080285458 | Lysne | Nov 2008 | A1 |
| 20080285555 | Ogasahara | Nov 2008 | A1 |
| 20080288020 | Einav | Nov 2008 | A1 |
| 20080298248 | Roeck | Dec 2008 | A1 |
| 20080304498 | Jorgensen | Dec 2008 | A1 |
| 20080310342 | Kruys | Dec 2008 | A1 |
| 20090022069 | Khan | Jan 2009 | A1 |
| 20090024734 | Merbach | Jan 2009 | A1 |
| 20090037607 | Farinacci | Feb 2009 | A1 |
| 20090042270 | Dolly | Feb 2009 | A1 |
| 20090044270 | Shelly | Feb 2009 | A1 |
| 20090067422 | Poppe | Mar 2009 | A1 |
| 20090067442 | Killian | Mar 2009 | A1 |
| 20090079560 | Fries | Mar 2009 | A1 |
| 20090080345 | Gray | Mar 2009 | A1 |
| 20090083445 | Ganga | Mar 2009 | A1 |
| 20090092042 | Yuhara | Apr 2009 | A1 |
| 20090092043 | Lapuh | Apr 2009 | A1 |
| 20090094354 | Rastogi | Apr 2009 | A1 |
| 20090106298 | Furusho | Apr 2009 | A1 |
| 20090106405 | Mazarick | Apr 2009 | A1 |
| 20090113408 | Toeroe | Apr 2009 | A1 |
| 20090116381 | Kanda | May 2009 | A1 |
| 20090122700 | Aboba | May 2009 | A1 |
| 20090129384 | Regan | May 2009 | A1 |
| 20090138577 | Casado | May 2009 | A1 |
| 20090138752 | Graham | May 2009 | A1 |
| 20090144720 | Roush | Jun 2009 | A1 |
| 20090161584 | Guan | Jun 2009 | A1 |
| 20090161670 | Shepherd | Jun 2009 | A1 |
| 20090168647 | Holness | Jul 2009 | A1 |
| 20090199177 | Edwards | Aug 2009 | A1 |
| 20090204965 | Tanaka | Aug 2009 | A1 |
| 20090213783 | Moreton | Aug 2009 | A1 |
| 20090222879 | Kostal | Sep 2009 | A1 |
| 20090232031 | Vasseur | Sep 2009 | A1 |
| 20090245112 | Okazaki | Oct 2009 | A1 |
| 20090245137 | Hares | Oct 2009 | A1 |
| 20090245242 | Carlson | Oct 2009 | A1 |
| 20090246137 | Hadida | Oct 2009 | A1 |
| 20090249444 | Macauley | Oct 2009 | A1 |
| 20090252049 | Ludwig | Oct 2009 | A1 |
| 20090252061 | Small | Oct 2009 | A1 |
| 20090252503 | Ishigami | Oct 2009 | A1 |
| 20090260083 | Szeto | Oct 2009 | A1 |
| 20090279558 | Davis | Nov 2009 | A1 |
| 20090292858 | Lambeth | Nov 2009 | A1 |
| 20090323698 | LeFaucheur | Dec 2009 | A1 |
| 20090323708 | Ihle | Dec 2009 | A1 |
| 20090327392 | Tripathi | Dec 2009 | A1 |
| 20090327462 | Adams | Dec 2009 | A1 |
| 20090328392 | Tripathi | Dec 2009 | |
| 20100002382 | Aybay | Jan 2010 | A1 |
| 20100027420 | Smith | Feb 2010 | A1 |
| 20100042869 | Szabo | Feb 2010 | A1 |
| 20100046471 | Hattori | Feb 2010 | A1 |
| 20100054260 | Pandey | Mar 2010 | A1 |
| 20100061269 | Banerjee | Mar 2010 | A1 |
| 20100074175 | Banks | Mar 2010 | A1 |
| 20100097941 | Carlson | Apr 2010 | A1 |
| 20100103813 | Allan | Apr 2010 | A1 |
| 20100103939 | Carlson | Apr 2010 | A1 |
| 20100114818 | Lier | May 2010 | A1 |
| 20100131636 | Suri | May 2010 | A1 |
| 20100157844 | Casey | Jun 2010 | A1 |
| 20100158024 | Sajassi | Jun 2010 | A1 |
| 20100165877 | Shukla | Jul 2010 | A1 |
| 20100165995 | Mehta | Jul 2010 | A1 |
| 20100168467 | Johnston | Jul 2010 | A1 |
| 20100169467 | Shukla | Jul 2010 | A1 |
| 20100169948 | Budko | Jul 2010 | A1 |
| 20100182920 | Matsuoka | Jul 2010 | A1 |
| 20100189119 | Sawada | Jul 2010 | A1 |
| 20100195489 | Zhou | Aug 2010 | A1 |
| 20100215042 | Sato | Aug 2010 | A1 |
| 20100215049 | Raza | Aug 2010 | A1 |
| 20100220724 | Rabie | Sep 2010 | A1 |
| 20100226368 | Mack-Crane | Sep 2010 | A1 |
| 20100226381 | Mehta | Sep 2010 | A1 |
| 20100246388 | Gupta | Sep 2010 | A1 |
| 20100254703 | Kirkpatrick | Oct 2010 | A1 |
| 20100257263 | Casado | Oct 2010 | A1 |
| 20100265849 | Harel | Oct 2010 | A1 |
| 20100271960 | Krygowski | Oct 2010 | A1 |
| 20100272107 | Papp | Oct 2010 | A1 |
| 20100281106 | Ashwood-Smith | Nov 2010 | A1 |
| 20100284414 | Agarwal | Nov 2010 | A1 |
| 20100284418 | Gray | Nov 2010 | A1 |
| 20100284698 | McColloch | Nov 2010 | A1 |
| 20100287262 | Elzur | Nov 2010 | A1 |
| 20100287548 | Zhou | Nov 2010 | A1 |
| 20100290464 | Assarpour | Nov 2010 | A1 |
| 20100290473 | Enduri | Nov 2010 | A1 |
| 20100299527 | Arunan | Nov 2010 | A1 |
| 20100303071 | Kotalwar | Dec 2010 | A1 |
| 20100303075 | Tripathi | Dec 2010 | A1 |
| 20100303083 | Belanger | Dec 2010 | A1 |
| 20100309820 | Rajagopalan | Dec 2010 | A1 |
| 20100309912 | Mehta | Dec 2010 | A1 |
| 20100329110 | Rose | Dec 2010 | A1 |
| 20110007738 | Berman | Jan 2011 | A1 |
| 20110019678 | Mehta | Jan 2011 | A1 |
| 20110032945 | Mullooly | Feb 2011 | A1 |
| 20110035489 | McDaniel | Feb 2011 | A1 |
| 20110035498 | Shah | Feb 2011 | A1 |
| 20110044339 | Kotalwar | Feb 2011 | A1 |
| 20110044352 | Chaitou | Feb 2011 | A1 |
| 20110051723 | Rabie | Mar 2011 | A1 |
| 20110055274 | Scales et al. | Mar 2011 | A1 |
| 20110058547 | Waldrop | Mar 2011 | A1 |
| 20110064086 | Xiong | Mar 2011 | A1 |
| 20110064089 | Hidaka | Mar 2011 | A1 |
| 20110072208 | Gulati | Mar 2011 | A1 |
| 20110085560 | Chawla | Apr 2011 | A1 |
| 20110085563 | Kotha | Apr 2011 | A1 |
| 20110088011 | Ouali | Apr 2011 | A1 |
| 20110110266 | Li | May 2011 | A1 |
| 20110134802 | Rajagopalan | Jun 2011 | A1 |
| 20110134803 | Dalvi | Jun 2011 | A1 |
| 20110134925 | Safrai | Jun 2011 | A1 |
| 20110142053 | Van | Jun 2011 | A1 |
| 20110142062 | Wang | Jun 2011 | A1 |
| 20110149526 | Turner | Jun 2011 | A1 |
| 20110158113 | Nanda | Jun 2011 | A1 |
| 20110161494 | McDysan | Jun 2011 | A1 |
| 20110161695 | Okita | Jun 2011 | A1 |
| 20110176412 | Stine | Jul 2011 | A1 |
| 20110188373 | Saito | Aug 2011 | A1 |
| 20110194403 | Sajassi | Aug 2011 | A1 |
| 20110194563 | Shen | Aug 2011 | A1 |
| 20110228767 | Singla | Sep 2011 | A1 |
| 20110228780 | Ashwood-Smith | Sep 2011 | A1 |
| 20110231570 | Altekar | Sep 2011 | A1 |
| 20110231574 | Saunderson | Sep 2011 | A1 |
| 20110235523 | Jha | Sep 2011 | A1 |
| 20110243133 | Villait | Oct 2011 | A9 |
| 20110243136 | Raman | Oct 2011 | A1 |
| 20110246669 | Kanada | Oct 2011 | A1 |
| 20110255538 | Srinivasan | Oct 2011 | A1 |
| 20110255540 | Mizrahi | Oct 2011 | A1 |
| 20110261828 | Smith | Oct 2011 | A1 |
| 20110268118 | Schlansker | Nov 2011 | A1 |
| 20110268120 | Vobbilisetty | Nov 2011 | A1 |
| 20110268125 | Vobbilisetty | Nov 2011 | A1 |
| 20110273988 | Tourrilhes | Nov 2011 | A1 |
| 20110273990 | Rajagopalan | Nov 2011 | A1 |
| 20110274114 | Dhar | Nov 2011 | A1 |
| 20110280572 | Vobbilisetty | Nov 2011 | A1 |
| 20110286357 | Haris | Nov 2011 | A1 |
| 20110286457 | Ee | Nov 2011 | A1 |
| 20110286462 | Kompella | Nov 2011 | A1 |
| 20110292947 | Vobbilisetty | Dec 2011 | A1 |
| 20110296052 | Guo | Dec 2011 | A1 |
| 20110299391 | Vobbilisetty | Dec 2011 | A1 |
| 20110299413 | Chatwani | Dec 2011 | A1 |
| 20110299414 | Yu | Dec 2011 | A1 |
| 20110299527 | Yu | Dec 2011 | A1 |
| 20110299528 | Yu | Dec 2011 | A1 |
| 20110299531 | Yu | Dec 2011 | A1 |
| 20110299532 | Yu | Dec 2011 | A1 |
| 20110299533 | Yu | Dec 2011 | A1 |
| 20110299534 | Koganti | Dec 2011 | A1 |
| 20110299535 | Vobbilisetty | Dec 2011 | A1 |
| 20110299536 | Cheng | Dec 2011 | A1 |
| 20110317559 | Kern | Dec 2011 | A1 |
| 20120011240 | Hara | Jan 2012 | A1 |
| 20120014261 | Salam | Jan 2012 | A1 |
| 20120014387 | Dunbar | Jan 2012 | A1 |
| 20120020220 | Sugita | Jan 2012 | A1 |
| 20120027017 | Rai | Feb 2012 | A1 |
| 20120033663 | Guichard | Feb 2012 | A1 |
| 20120033665 | Jacob | Feb 2012 | A1 |
| 20120033668 | Humphries | Feb 2012 | A1 |
| 20120033669 | Mohandas | Feb 2012 | A1 |
| 20120033672 | Page | Feb 2012 | A1 |
| 20120042095 | Kotha | Feb 2012 | A1 |
| 20120063363 | Li | Mar 2012 | A1 |
| 20120075991 | Sugita | Mar 2012 | A1 |
| 20120099567 | Hart | Apr 2012 | A1 |
| 20120099602 | Nagapudi | Apr 2012 | A1 |
| 20120099863 | Xu | Apr 2012 | A1 |
| 20120102160 | Breh | Apr 2012 | A1 |
| 20120106339 | Mishra | May 2012 | A1 |
| 20120117438 | Shaffer | May 2012 | A1 |
| 20120131097 | Baykal | May 2012 | A1 |
| 20120131289 | Taguchi | May 2012 | A1 |
| 20120134266 | Roitshtein | May 2012 | A1 |
| 20120136999 | Roitshtein | May 2012 | A1 |
| 20120147740 | Nakash | Jun 2012 | A1 |
| 20120158997 | Hsu | Jun 2012 | A1 |
| 20120163164 | Terry | Jun 2012 | A1 |
| 20120170491 | Kern | Jul 2012 | A1 |
| 20120177039 | Berman | Jul 2012 | A1 |
| 20120210416 | Mihelich | Aug 2012 | A1 |
| 20120221636 | Surtani | Aug 2012 | A1 |
| 20120230225 | Matthews | Sep 2012 | A1 |
| 20120239918 | Huang | Sep 2012 | A1 |
| 20120243539 | Keesara | Sep 2012 | A1 |
| 20120250502 | Brolin | Oct 2012 | A1 |
| 20120260079 | Mruthyunjaya | Oct 2012 | A1 |
| 20120275297 | Subramanian | Nov 2012 | A1 |
| 20120275347 | Banerjee | Nov 2012 | A1 |
| 20120278804 | Narayanasamy | Nov 2012 | A1 |
| 20120287785 | Kamble | Nov 2012 | A1 |
| 20120294192 | Masood | Nov 2012 | A1 |
| 20120294194 | Balasubramanian | Nov 2012 | A1 |
| 20120320800 | Kamble | Dec 2012 | A1 |
| 20120327766 | Tsai et al. | Dec 2012 | A1 |
| 20130003535 | Sarwar | Jan 2013 | A1 |
| 20130003549 | Matthews | Jan 2013 | A1 |
| 20130003737 | Sinicrope | Jan 2013 | A1 |
| 20130003738 | Koganti | Jan 2013 | A1 |
| 20130028072 | Addanki | Jan 2013 | A1 |
| 20130034015 | Jaiswal | Feb 2013 | A1 |
| 20130034021 | Jaiswal | Feb 2013 | A1 |
| 20130066947 | Ahmad | Mar 2013 | A1 |
| 20130067466 | Combs | Mar 2013 | A1 |
| 20130070762 | Adams | Mar 2013 | A1 |
| 20130083693 | Himura | Apr 2013 | A1 |
| 20130097345 | Munoz | Apr 2013 | A1 |
| 20130114595 | Mack-Crane et al. | May 2013 | A1 |
| 20130124707 | Ananthapadmanabha | May 2013 | A1 |
| 20130127848 | Joshi | May 2013 | A1 |
| 20130132296 | Koppenhagen | May 2013 | A1 |
| 20130135811 | Dunwoody | May 2013 | A1 |
| 20130136123 | Ge | May 2013 | A1 |
| 20130148546 | Eisenhauer | Jun 2013 | A1 |
| 20130156425 | Kirkpatrick | Jun 2013 | A1 |
| 20130194914 | Agarwal | Aug 2013 | A1 |
| 20130219473 | Schaefer | Aug 2013 | A1 |
| 20130223221 | Xu | Aug 2013 | A1 |
| 20130223449 | Koganti | Aug 2013 | A1 |
| 20130250951 | Koganti | Sep 2013 | A1 |
| 20130259037 | Natarajan | Oct 2013 | A1 |
| 20130266015 | Qu | Oct 2013 | A1 |
| 20130268590 | Mahadevan | Oct 2013 | A1 |
| 20130272135 | Leong | Oct 2013 | A1 |
| 20130294451 | Li | Nov 2013 | A1 |
| 20130297757 | Han | Nov 2013 | A1 |
| 20130301425 | Udutha et al. | Nov 2013 | A1 |
| 20130301642 | Radhakrishnan | Nov 2013 | A1 |
| 20130308492 | Baphna | Nov 2013 | A1 |
| 20130308647 | Rosset | Nov 2013 | A1 |
| 20130315586 | Kipp | Nov 2013 | A1 |
| 20130322427 | Stiekes | Dec 2013 | A1 |
| 20130332660 | Talagala | Dec 2013 | A1 |
| 20130336104 | Talla | Dec 2013 | A1 |
| 20130346583 | Low | Dec 2013 | A1 |
| 20140013324 | Zhang | Jan 2014 | A1 |
| 20140019608 | Kawakami | Jan 2014 | A1 |
| 20140025736 | Wang | Jan 2014 | A1 |
| 20140044126 | Sabhanatarajan | Feb 2014 | A1 |
| 20140050223 | Foo | Feb 2014 | A1 |
| 20140056298 | Vobbilisetty | Feb 2014 | A1 |
| 20140059225 | Gasparakis | Feb 2014 | A1 |
| 20140064056 | Sakata | Mar 2014 | A1 |
| 20140086253 | Yong | Mar 2014 | A1 |
| 20140105034 | Sun | Apr 2014 | A1 |
| 20140157251 | Hocker | Jun 2014 | A1 |
| 20140258446 | Bursell | Sep 2014 | A1 |
| 20140269733 | Venkatesh | Sep 2014 | A1 |
| 20140298091 | Carlen | Oct 2014 | A1 |
| 20140355477 | Velayudhan et al. | Dec 2014 | A1 |
| 20150010007 | Matsuhira | Jan 2015 | A1 |
| 20150030031 | Zhou | Jan 2015 | A1 |
| 20150127618 | Alberti | May 2015 | A1 |
| 20150143369 | Zheng | May 2015 | A1 |
| 20150172098 | Agarwal | Jun 2015 | A1 |
| 20150195093 | Mahadevan | Jul 2015 | A1 |
| 20150222506 | Kizhakkiniyil | Aug 2015 | A1 |
| 20150248298 | Gavrilov | Sep 2015 | A1 |
| 20150263991 | Macchiano | Sep 2015 | A1 |
| 20150281066 | Koley | Oct 2015 | A1 |
| 20150301901 | Rath | Oct 2015 | A1 |
| 20150347468 | Bester | Dec 2015 | A1 |
| 20160072899 | Tung | Mar 2016 | A1 |
| Number | Date | Country |
|---|---|---|
| 102801599 | Nov 2012 | CN |
| 0579567 | May 1993 | EP |
| 0579567 | Jan 1994 | EP |
| 0993156 | Dec 2000 | EP |
| 1398920 | Mar 2004 | EP |
| 2001167 | Aug 2007 | EP |
| 1916807 | Apr 2008 | EP |
| 2874359 | May 2015 | EP |
| 2008056838 | May 2008 | WO |
| 2009042919 | Apr 2009 | WO |
| 2010111142 | Sep 2010 | WO |
| 2011140028 | Nov 2011 | WO |
| 2014031781 | Feb 2014 | WO |
| Entry |
|---|
| Mahalingam “VXLAN: a Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks” Oct. 17, 2013 pp. 1-22, Sections 1, 4 and 4.1. |
| Office action dated Apr. 30, 2015, U.S. Appl. No. 13/351,513, filed Jan. 17, 2012. |
| Office Action dated Apr. 1, 2015, U.S. Appl. No. 13/656,438, filed Oct. 19, 2012. |
| Office Action dated May 21, 2015, U.S. Appl. No. 13/288,822, filed Nov. 3, 2011. |
| Siamak Azodolmolky et al. “Cloud computing networking: Challenges and opportunities for innovations”, IEEE Communications Magazine, vol. 51, No. 7, Jul. 1, 2013. |
| Office Action dated Apr. 1, 2015 U.S. Appl. No. 13/656,438, filed Oct. 19, 2012. |
| Office action dated Jun. 8, 2015, U.S. Appl. No. 14/178,042, filed Feb. 11, 2014. |
| Office Action Dated Jun. 10, 2015, U.S. Appl. No. 13/890,150, filed May 8, 2013. |
| Office Action dated 06/18/215, U.S. Appl. No. 13/098,490, filed May 2, 2011. |
| Office Action dated Jun. 16, 2015, U.S. Appl. No. 13/048,817, filed Mar. 15, 2011. |
| Eastlake, D. et al., ‘RBridges: TRILL Header Options’, Dec. 24, 2009, pp. 1-17, TRILL Working Group. |
| Perlman, Radia et al., ‘RBridge VLAN Mapping’, TRILL Working Group, Dec. 4, 2009, pp. 1-12. |
| Touch, J. et al., ‘Transparent Interconnection of Lots of Links (TRILL): Problem and Applicability Statement’, May 2009, Network Working Group, pp. 1-17. |
| ‘RBridges: Base Protocol Specification’, IETF Draft, Perlman et al., Jun. 26, 2009. |
| ‘Switched Virtual Networks. Internetworking Moves Beyond Bridges and Routers’ Data Communications, McGraw Hill. New York, US, vol. 23, No. 12, Sep. 1, 1994 (Sep. 1, 1994), pp. 66-70,72,74, XP000462385 ISSN: 0363-6399. |
| Office Action dated Apr. 9, 2014, U.S. Appl. No. 13/092,724, filed Apr. 22, 2011. |
| Office action dated Aug. 29, 2014, U.S. Appl. No. 13/042,259, filed Mar. 7, 2011. |
| Office Action dated Mar. 14, 2014, U.S. Appl. No. 13/092,460, filed Apr. 22, 2011. |
| Office Action dated Mar. 26, 2014, U.S. Appl. No. 13/092,701, filed Apr. 22, 2011. |
| Office Action dated Apr. 9, 2014, U.S. Appl. No. 13/092,752, filed Apr. 22, 2011. |
| Office action dated Aug. 21, 2014, U.S. Appl. No. 13/184,526, filed Jul. 16, 2011. |
| Office Action dated Mar. 6, 2014, U.S. Appl. No. 13/425,238, filed Mar. 20, 2012. |
| Office Action dated Jun. 18, 2014, U.S. Appl. No. 13/440,861, filed Apr. 5, 2012. |
| Office Action dated Feb. 28, 2014, U.S. Appl. No. 13/351,513, filed Jan. 17, 2012. |
| Office Action dated May 9, 2014, U.S. Appl. No. 13/484,072, filed May 30, 2012. |
| Office Action dated May 14, 2014, U.S. Appl. No. 13/533,843, filed Jun. 26, 2012. |
| Office Action dated Feb. 20, 2014, U.S. Appl. No. 13/598,204, filed Aug. 29, 2012. |
| Office Action dated Jun. 6, 2014, U.S. Appl. No. 13/669,357, filed Nov. 5, 2012. |
| Brocade, ‘Brocade Fabrics OS (FOS) 6.2 Virtual Fabrics Feature Frequently Asked Questions’, pp. 1-6, 2009 Brocade Communications Systems, Inc. |
| Brocade, ‘Fastlron and Turbolron 24x Configuration Guide’, Feb. 16, 2010. |
| Brocade, ‘The Effortless Network: Hyperedge Technology for the Campus LAN’ 2012. |
| Brocade ‘An Introduction to Brocade VCS Fabric Technology’, Dec. 3, 2012. |
| Christensen, M. et al., ‘Considerations for Internet Group Management Protocol (IGMP) and Multicast Listener Discovery (MLD) Snooping Switches’, May 2006. |
| Fastlron Configuration Guide Supporting Ironware Software Release 07.0.00, Dec. 18, 2009. |
| Foundary Fastlron Configuration Guide, Software Release FSX 04.2.00b, Software Release FWS 04.3.00, Software Release FGS 05.0.00a, Sep. 2008. |
| Huang, Nen-Fu et al., ‘An Effective Spanning Tree Algorithm for a Bridged LAN’, Mar. 16, 1992. |
| Knight, ‘Network Based IP VPN Architecture using Virtual Routers’, May 2003. |
| Knight P et al: ‘Layer 2 and 3 Virtual Private Networks: Taxonomy, Technology, and Standardization Efforts’, IEEE Communications Magazine, IEEE Service Center, Piscataway, US, vol. 42, No. 6, Jun. 1, 2004(Jun. 1, 2004), pp. 124-131, XP001198207, ISSN:0163-6804, DOI: 10.1109/MCOM.2004.1304248. |
| Knight S et al: ‘Virtual Router Redundancy Protocol’ Internet Citation Apr. 1, 1998 (Apr. 1, 1998), XP002135272 Retrieved from the Internet: URL:ftp://ftp.isi.edu/in-notes/rfc2338.txt [retrieved on Apr. 10, 2000]. |
| Lapuh, Roger et al., ‘Split Multi-link Trunking (SMLT)’, draft-lapuh-network-smlt-08, Jul. 2008. |
| Lapuh, Roger et al., ‘Split Multi-Link Trunking (SMLT)’, Network Working Group, Oct. 2012. |
| Louati, Wajdi et al., ‘Network-based virtual personal overlay networks using programmable virtual routers’, IEEE Communications Magazine, Jul. 2005. |
| Narten, T. et al., ‘Problem Statement: Overlays for Network Virtualization d raft-narten-n vo3-over I ay-problem—statement-01’, Oct. 31, 2011. |
| Office Action for U.S. Appl. No. 13/042,259, filed Mar. 7, 2011, from Jaroenchonwanit, Bunjob, dated Jan. 16, 2014. |
| Office Action for U.S. Appl. No. 13/092,752, filed Apr. 22, 2011, from Park, Jung H., dated Jul. 18, 2013. |
| Office Action for U.S. Appl. No. 13/365,993, filed Feb. 3, 2012, from Cho, Hong Sol., dated Jul. 23, 2013. |
| Office Action for U.S. Appl. No. 12/725,249, filed Mar. 16, 2010, dated Apr. 26, 2013. |
| Office Action for U.S. Appl. No. 12/725,249, filed Mar. 16, 2010, dated Sep. 12, 2012. |
| Office Action for U.S. Appl. No. 12/950,968, filed Nov. 19, 2010, dated Jan. 4, 2013. |
| Office Action for U.S. Appl. No. 12/950,968, filed Nov. 19, 2010, dated Jun. 7, 2012. |
| Office Action for U.S. Appl. No. 12/950,974, filed Nov. 19, 2010, dated Dec. 20, 2012. |
| Office Action for U.S. Appl. No. 12/950,974, filed Nov. 19, 2010, dated May 24, 2012. |
| Office Action for U.S. Appl. No. 13/030,688, filed Feb. 18, 2011, dated Apr. 25, 2013. |
| Office Action for U.S. Appl. No. 13/030,806, filed Feb. 18, 2011, dated Dec. 3, 2012. |
| Office Action for U.S. Appl. No. 13/030,806, filed Feb. 18, 2011, dated Jun. 11, 2013. |
| Office Action for U.S. Appl. No. 13/042,259, filed Mar. 7, 2011, dated Mar. 18, 2013. |
| Office Action for U.S. Appl. No. 13/042,259, filed Mar. 7, 2011, dated Jul. 31, 2013. |
| Office Action for U.S. Appl. No. 13/044,301, filed Mar. 9, 2011, dated Feb. 22, 2013. |
| Office Action for U.S. Appl. No. 13/044,301, filed Mar. 9, 2011, dated Jun. 11, 2013. |
| Office Action for U.S. Appl. No. 13/044,326, filed Mar. 9, 2011, dated Oct. 2, 2013. |
| Office Action for U.S. Appl. No. 13/050,102, filed Mar. 17, 2011, dated Oct. 26, 2012. |
| Office Action for U.S. Appl. No. 13/050,102, filed Mar. 17, 2011, dated May 16, 2013. |
| Office Action for U.S. Appl. No. 13/087,239, filed Apr. 14, 2011, dated May 22, 2013. |
| Office Action for U.S. Appl. No. 13/092,460, filed Apr. 22, 2011, dated Jun. 21, 2013. |
| Office Action for U.S. Appl. No. 13/092,580, filed Apr. 22, 2011, dated Jun. 10, 2013. |
| Office Action for U.S. Appl. No. 13/092,701, filed Apr. 22, 2011, dated Jan. 28, 2013. |
| Office Action for U.S. Appl. No. 13/092,701, filed Apr. 22, 2011, dated Jul. 3, 2013. |
| Office Action for U.S. Appl. No. 13/092,724, filed Apr. 22, 2011, dated Feb. 5, 2013. |
| Office Action for U.S. Appl. No. 13/092,724, filed Apr. 22, 2011, dated Jul. 16, 2013. |
| Office Action for U.S. Appl. No. 13/092,752, filed Apr. 22, 2011, dated Feb. 5, 2013. |
| Office Action for U.S. Appl. No. 13/092,864, filed Apr. 22, 2011, dated Sep. 19, 2012. |
| Office Action for U.S. Appl. No. 13/092,873, filed Apr. 22, 2011, dated Jun. 19, 2013. |
| Office Action for U.S. Appl. No. 13/092,877, filed Apr. 22, 2011, dated Mar. 4, 2013. |
| Office Action for U.S. Appl. No. 13/092,877, filed Apr. 22, 2011, dated Sep. 5, 2013. |
| Office Action for U.S. Appl. No. 13/098,360, filed Apr. 29, 2011, dated May 31, 2013. |
| Office Action for U.S. Appl. No. 13/098,490, filed May 2, 2011, dated Dec. 21, 2012. |
| Office Action for U.S. Appl. No. 13/098,490, filed May 2, 2011, dated Mar. 27, 2014. |
| Office Action for U.S. Appl. No. 13/098,490, filed May 2, 2011, dated Jul. 9, 2013. |
| Office Action for U.S. Appl. No. 13/184,526, filed Jul. 16, 2011, dated Jan. 28, 2013. |
| Office Action for U.S. Appl. No. 13/184,526, filed Jul. 16, 2011, dated May 22, 2013. |
| Office Action for U.S. Appl. No. 13/312,903, filed Dec. 6, 2011, dated Jun. 13, 2013. |
| Office Action for U.S. Appl. No. 13/092,873, filed Apr. 22, 2011, dated Nov. 29, 2013. |
| Office Action for U.S. Appl. No. 13/184,526, filed Jul. 16, 2011, dated Dec. 2, 2013. |
| Office Action for U.S. Appl. No. 13/533,843, filed Jun. 26, 2012, dated Oct. 21, 2013. |
| Office Action for U.S. Appl. No. 13/598,204, filed Aug. 29, 2012, dated Feb. 20, 2014. |
| Perlman, Radia et al., ‘Challenges and Opportunities in the Design of TRILL: a Routed layer 2 Technology’, 2009. |
| Perlman, Radia et al., ‘RBridges: Base Protocol Specification; Draft-ietf-trill-rbridge-protocol-16.txt’, Mar. 3, 2010, pp. 1-117. |
| S. Nadas et al., ‘Virtual Router Redundancy Protocol (VRRP) Version 3 for IPv4 and IPv6’, Internet Engineering Task Force, Mar. 2010. |
| Zhai F. Hu et al. ‘RBridge: Pseudo-Nickname; draft-hu-trill-pseudonode-nickname-02.txt’, May 15, 2012. |
| ‘An Introduction to Brocade VCS Fabric Technology’, BROCADE white paper, http://community.brocade.com/docs/DOC-2954, Dec. 3, 2012. |
| U.S. Appl. No. 13/030,806 Office Action dated Dec. 3, 2012. |
| Office action dated Jan. 10, 2014, U.S. Appl. No. 13/092,580, filed Apr. 22, 2011. |
| Office action dated Jan. 16, 2014, U.S. Appl. No. 13/042,259, filed Mar. 7, 2011. |
| Office action dated Jul. 31, 2013, U.S. Appl. No. 13/042,259, filed Mar. 7, 2011. |
| Office action dated Jan. 6, 2014, U.S. Appl. No. 13/092,877, filed Apr. 22, 2011. |
| Office action dated Oct. 2, 2013, U.S. Appl. No. 13/044,326, filed Mar. 9, 2011. |
| Office action dated Dec. 2, 2013, U.S. Appl. No. 13/184,526, filed Jul. 16, 2011. |
| Office action dated Nov. 29, 2013, U.S. Appl. No. 13/092,873, filed Apr. 22, 2011. |
| Office action dated Nov. 12, 2013, U.S. Appl. No. 13/312,903, filed Dec. 6, 2011. |
| Brocade ‘Brocade Unveils’ The Effortless Network, http://newsroom.brocade.com/press-releases/brocade-unveils-the-effortless-network-nasdaq-brcd-0859535, 2012. |
| Kreeger, L. et al., ‘Network Virtualization Overlay Control Protocol Requirements draft-kreeger-nvo3-overlay-cp-00’, Jan. 30, 2012. |
| Office Action for U.S. Appl. No. 13/365,808, filed Jul. 18, 2013, dated Jul. 18, 2013. |
| Office Action for U.S. Appl. No. 13/092,887, dated Jan. 6, 2014. |
| Office action dated Apr. 26, 2012, U.S. Appl. No. 12/725,249, filed Mar. 16, 2010. |
| Office action dated Sep. 12, 2012, U.S. Appl. No. 12/725,249, filed Mar. 16, 2010. |
| Office action dated Dec. 21, 2012, U.S. Appl. No. 13/098,490, filed May 2, 2011. |
| Office action dated Mar. 27, 2014, U.S. Appl. No. 13/098,490, filed May 2, 2011. |
| Office action dated Jul. 9, 2013, U.S. Appl. No. 13/098,490, filed May 2, 2011. |
| Office action dated May 22, 2013, U.S. Appl. No. 13/087,239, filed Apr. 14, 2011. |
| Office action dated Dec. 5, 2012, U.S. Appl. No. 13/087,239, filed Apr. 14, 2011. |
| Office action dated Feb. 5, 2013, U.S. Appl. No. 13/092,724, filed Apr. 22, 2011. |
| Office action dated Jun. 10, 2013, U.S. Appl. No. 13/092,580, filed Apr. 22, 2011. |
| Office action dated Mar. 18, 2013, U.S. Appl. No. 13/042,259, filed Mar. 7, 2011. |
| Office action dated Jun. 21, 2013, U.S. Appl. No. 13/092,460, filed Apr. 22, 2011. |
| Office action dated Jan. 28, 2013, U.S. Appl. No. 13/092,701, filed Apr. 22, 2011. |
| Office action dated Jul. 3, 2013, U.S. Appl. No. 13/092,701, filed Apr. 22, 2011. |
| Office action dated Jul. 18, 2013, U.S. Appl. No. 13/092,752, filed Apr. 22, 2011. |
| Office action dated Dec. 20, 2012, U.S. Appl. No. 12/950,974, filed Nov. 19, 2010. |
| Office action dated May 24, 2012, U.S. Appl. No. 12/950,974, filed Nov. 19, 2010. |
| Office action dated Sep. 5, 2013, U.S. Appl. No. 13/092,877, filed Apr. 22, 2011. |
| Office action dated Mar. 4, 2013, U.S. Appl. No. 13/092,877, filed Apr. 22, 2011. |
| Office action dated Jan. 4, 2013, U.S. Appl. No. 12/950,968, filed Nov. 19, 2010. |
| Office action dated Jun. 7, 2012, U.S. Appl. No. 12/950,968, filed Nov. 19, 2010. |
| Office action dated Sep. 19, 2012, U.S. Appl. No. 13/092,864, filed Apr. 22, 2011. |
| Office action dated May 31, 2013, U.S. Appl. No. 13/098,360, filed Apr. 29, 2011. |
| Office action dated Dec. 3, 2012, U.S. Appl. No. 13/030,806, filed Feb. 18, 2011. |
| Office action dated Apr. 22, 2014, U.S. Appl. No. 13/030,806, filed Feb. 18, 2011. |
| Office action dated Jun. 11, 2013, U.S. Appl. No. 13/030,806, filed Feb. 18, 2011. |
| Office action dated Apr. 25, 2013, U.S. Appl. No. 13/030,688, filed Feb. 18, 2011. |
| Office action dated Feb. 22, 2013, U.S. Appl. No. 13/044,301, filed Mar. 9, 2011. |
| Office action dated Jun. 11, 2013, U.S. Appl. No. 13/044,301, filed Mar. 9, 2011. |
| Office action dated Oct. 26, 2012, U.S. Appl. No. 13/050,102, filed Mar. 17, 2011. |
| Office action dated May 16, 2013, U.S. Appl. No. 13/050,102, filed Mar. 17, 2011. |
| Office action dated Aug. 4, 2014, U.S. Appl. No. No. 13/050,102, filed Mar. 17, 2011. |
| Office action dated Jan. 28, 2013, U.S. Appl. No. 13/148,526, filed Jul. 16, 2011. |
| Office action dated May 22, 2013, U.S. Appl. No. 13/148,526, filed Jul. 16, 2011. |
| Office action dated Jun. 19, 2013, U.S. Appl. No. 13/092,873, filed Apr. 22, 2011. |
| Office action dated Jul. 18, 2013, U.S. Appl. No. 13/365,808, filed Feb. 3, 2012. |
| Office action dated Jun. 13, 2013, U.S. Appl. No. 13/312,903, filed Dec. 6, 2011. |
| Lapuh, Roger et al., ‘Split Multi-link Trunking (SMLT) draft-lapuh-network-smlt-08’, Jan. 2009. |
| Office Action for U.S. Appl. No. 13/030,688, filed Feb. 18, 2011, dated Jul. 17, 2014. |
| Office Action for U.S. Appl. No. 13/044,326, filed Mar. 9, 2011, dated Jul. 7, 2014. |
| Office Action for U.S. Appl. No. 13/092,752, filed Apr. 22, 2011, dated Apr. 9, 2014. |
| Office Action for U.S. Appl. No. 13/092,873, filed Apr. 22, 2011, dated Jul. 25, 2014. |
| Office Action for U.S. Appl. No. 13/092,877, filed Apr. 22, 2011, dated Jun. 20, 2014. |
| Office Action for U.S. Appl. No. 13/312,903, filed Dec. 6, 2011, dated Aug. 7, 2014. |
| Office Action for U.S. Appl. No. 13/351,513, filed Jan. 17, 2012, dated Jul. 24, 2014. |
| Office Action for U.S. Appl. No. 13/425,238, filed Mar. 20, 2012, dated Mar. 6, 2014. |
| Office Action for U.S. Appl. No. 13/556,061, filed Jul. 23, 2012, dated Jun. 6, 2014. |
| Office Action for U.S. Appl. No. 13/742,207 dated Jul. 24, 2014, filed Jan. 15, 2013. |
| Office Action for U.S. Appl. No. 13/950,974, filed Nov. 19, 2010, from Haile, Awet A., dated Dec. 2, 2012. |
| Office Action for U.S. Appl. No. 13/087,239, filed Apr. 14, 2011, dated Dec. 5, 2012. |
| Office Action for U.S. Appl. No. 13/351,513, filed Jan. 17, 2012, dated Feb. 28, 2014. |
| Perlman R: ‘Challenges and opportunities in the design of TRILL: a routed layer 2 technology’, 2009 IEEE GLOBECOM Workshops, Honolulu, HI, USA, Piscataway, NJ, USA, Nov. 30, 2009 (Nov. 30, 2009), pp. 1-6, XP002649647, DOI: 10.1109/GLOBECOM.2009.5360776 ISBN: 1-4244-5626-0 [retrieved on Jul. 19, 2011]. |
| TRILL Working Group Internet-Draft Intended status: Proposed Standard RBridges: Base Protocol Specificaiton Mar. 3, 2010. |
| Office action dated Aug. 14, 2014, U.S. Appl. No. 13/092,460, filed Apr. 22, 2011. |
| Office action dated Jul. 7, 2014, for U.S. Appl. No. 13/044,326, filed Mar. 9, 2011. |
| Office Action dated Dec. 19, 2014, for U.S. Appl. No. 13/044,326, filed Mar. 9, 2011. |
| Office Action for U.S. Appl. No. 13/092,873, filed Apr. 22, 2011, dated Nov. 7, 2014. |
| Office Action for U.S. Appl. No. 13/092,877, filed Apr. 22, 2011, dated Nov. 10, 2014. |
| Office Action for U.S. Appl. No. 13/157,942, filed Jun. 10, 2011. |
| Mckeown, Nick et al. “OpenFlow: Enabling Innovation in Campus Networks”, Mar. 14, 2008, www.openflow.org/documents/openflow-wp-latest.pdf. |
| Office Action for U.S. Appl. No. 13/044,301, dated Mar. 9, 2011. |
| Office Action for U.S. Appl. No. 13/184,526, filed Jul. 16, 2011, dated Jan. 5, 2015. |
| Office Action for U.S. Appl. No. 13/598,204, filed Aug. 29, 2012, dated Jan. 5, 2015. |
| Office Action for U.S. Appl. No. 13/669,357, filed Nov. 5, 2012, dated Jan. 30, 2015. |
| Office Action for U.S. Appl. No. 13/851,026, filed Mar. 26, 2013, dated Jan. 30, 2015. |
| Office Action for U.S. Appl. No. 13/786,328, filed Mar. 5, 2013, dated Mar. 13, 2015. |
| Office Action for U.S. Appl. No. 13/092,460, filed Apr. 22, 2011, dated Mar. 13, 2015. |
| Office Action for U.S. Appl. No. 13/425,238, dated Mar. 12, 2015. |
| Office Action for U.S. Appl. No. 13/092,752, filed Apr. 22, 2011, dated Feb. 27, 2015. |
| Office Action for U.S. Appl. No. 13/042,259, filed Mar. 7, 2011, dated Feb. 23, 2015. |
| Office Action for U.S. Appl. No. 13/044,301, filed Mar. 9, 2011, dated Jan. 29, 2015. |
| Office Action for U.S. Appl. No. 13/050,102, filed Mar. 17, 2011, dated Jan. 26, 2015. |
| Rosen, E. et al., “BGP/MPLS VPNs”, Mar. 1999. |
| Office Action for U.S. Appl. No. 14/577,785, filed Dec. 19, 2014, dated Apr. 13, 2015. |
| Office Action for U.S. Appl. No. 13/425,238, filed Mar. 20, 2012, dated Mar. 12, 2015. |
| Abawajy J. “An Approach to Support a Single Service Provider Address Image for Wide Area Networks Environment” Centre for Parallel and Distributed Computing, School of Computer Science Carleton University, Ottawa, Ontario, K1S 5B6, Canada. |
| Office Action dated Feb. 11, 2016, U.S. Appl. No. 14/488,173, filed Sep. 16, 2014. |
| Office Action dated Feb. 24, 2016, U.S. Appl. No. 13/971,397, filed Aug. 20, 2013. |
| Office Action dated Feb. 24, 2016, U.S. Appl. No. 12/705,508, filed Feb. 12, 2010. |
| Office Action dated Jun. 18, 215, U.S. Appl. No. 13/098,490, filed May 2, 2011. |
| Office Action dated Jul. 31, 2015, U.S. Appl. No. 13/598,204, filed Aug. 29, 2014. |
| Office Action dated Jul. 31, 2015, U.S. Appl. No. 14/473,941, filed Aug. 29, 2014. |
| Office Action dated Jul. 31, 2015, U.S. Appl. No. 14/488,173, filed Sep. 16, 2014. |
| Office Action dated Aug. 21, 2015, U.S. Appl. No. 13/776,217, filed Feb. 25, 2013. |
| Office Action dated Aug. 19, 2015, U.S. Appl. No. 14/156,374, filed Jan. 15, 2014. |
| Office Action dated Sep. 2, 2015, U.S. Appl. No. 14/151,693, filed Jan. 9, 2014. |
| Office Action dated Sep. 17, 2015, U.S. Appl. No. 14/577,785, filed Dec. 19, 2014. |
| Office Action dated Sep. 22, 2015 U.S. Appl. No. 13/656,438, filed Oct. 19, 2012. |
| Office Action dated Nov. 5, 2015, U.S. Appl. No. 14/178,042, filed Feb. 11, 2014. |
| Office Action dated Oct. 19, 2015, U.S. Appl. No. 14/215,996, filed Mar. 17, 2014. |
| Office Action dated Sep. 18, 2015, U.S. Appl. No. 13/345,566, filed Jan. 6, 2012. |
| Open Flow Switch Specification Version 1.1.0, Feb. 28, 2011. |
| Open Flow Switch Specification Version 1.0.0, Dec. 31, 2009. |
| Open Flow Configuration and Management Protocol 1.0 (OF-Config 1.0) Dec. 23, 2011. |
| Open Flow Switch Specification Version 1.2 Dec. 5, 2011. |
| Office action dated Feb. 2, 2016, U.S. Appl. No. 13/092,460, filed Apr. 22, 2011. |
| Office Action dated Feb. 2, 2016. U.S. Appl. No. 14/154,106, filed Jan. 13, 2014. |
| Office Action dated Feb. 3, 2016, U.S. Appl. No. 13/098,490, filed May 2, 2011. |
| Office Action dated Feb. 4, 2016, U.S. Appl. No. 13/557,105, filed Jul. 24, 2012. |
| Office Action dated Jul. 6, 2016, U.S. Appl. No. 14/618,941, filed Feb. 10, 2015. |
| Office Action dated Jul. 20, 2016, U.S. Appl. No. 14/510,913, filed Oct. 9, 2014. |
| Office Action dated Jul. 29, 2016, U.S. Appl. No. 14/476,941, filed Aug. 29, 2014. |
| Office Action dated Jul. 28, 2016, U.S. Appl. No. 14/284,212, filed May 21, 2016. |
| Office Action dated Jan. 31, 2017, U.S. Appl. No. 13/184,526, filed Jul. 16, 2011. |
| Office Action dated Jan. 27, 2017, U.S. Appl. No. 14/216,292, filed Mar. 17, 2014. |
| Office Action dated Jan. 26, 2017, U.S. Appl. No. 13/786,328, filed Mar. 5, 2013. |
| Office Action dated Dec. 2, 2016, U.S. Appl. No. 14/512,268, filed Oct. 10, 2014. |
| Office Action dated Dec. 1, 2016, U.S. Appl. No. 13/899,849, filed May 22, 2013. |
| Office Action dated Dec. 1, 2016, U.S. Appl. No. 13/656,438, filed Oct. 19, 2012. |
| Office Action dated Nov. 30, 2016, U.S. Appl. No. 13/598,204, filed Aug. 29, 2012. |
| Office Action dated Nov. 21, 2016, U.S. Appl. No. 13/669,357, filed Nov. 5, 2012. |
| Office Action dated Feb. 8, 2017, U.S. Appl. No. 14/473,941, filed Aug. 29, 2014. |
| Office Action dated Feb. 8, 2017, U.S. Appl. No. 14/822,380, filed Aug. 10, 2015. |
| Number | Date | Country | |
|---|---|---|---|
| 20160191316 A1 | Jun 2016 | US |
