Patent Grant
11374817
As more networks move to the cloud, it is more common for corporations or other entities to have networks spanning multiple sites. While logical networks that operate within a single site are well established, there are various challenges in having logical networks span multiple physical sites (e.g., datacenters). The sites should be self-contained, while also allowing for data to be sent from one site to another easily. Various solutions are required to solve these issues.
Some embodiments of the invention provide a network management system for managing a logical network spanning multiple federated sites (e.g., multiple datacenters). The network management system of some embodiments includes a global manager that manages the entire logical network spanning all of the sites, as well as local managers at each site that directly manage the logical network at their respective sites. The logical network includes logical network elements that span one or more sites and logical network policies that apply to the elements at those sites. In some embodiments, the global manager receives a global desired configuration for the logical network (e.g., from an administrator of the network), identifies a relevant portion of the global desired configuration for each site in the federation, and provides the identified portion to the site's corresponding local manager. In addition, the global manager collects information about the realized state of the logical network across the sites from the local managers and provides this information (e.g., to an administrator of the logical network) for troubleshooting and management purposes.
In some embodiments, the global manager executes on a computing device at one of the sites spanned by the logical network, and each local manager also executes on a computing device at its respective site. In some embodiments, the global manager executes on the same computing device at one of the physical sites as the local manager managing that site. The global manager and the local managers are in some embodiments separate modules of a single application. Some embodiments deploy each manager at a physical site as a cluster of machines, with each machine executing on a different computing device at the same site.
Some embodiments employ a primary global manager and a secondary global manager, in an active-standby arrangement. The primary global manager is asynchronously synchronized with the secondary global manager as a standby for failover scenarios. The secondary global manager executes on a different computing device, located in some embodiments for resiliency at a different site than the primary global manager, and maintains an independent database. The secondary global manager is also deployed in some embodiments as a set of machines and may also execute on the same computing device as a local manager.
The primary global manager's database is a distributed shared log implemented across a set of storage devices at the physical site where the primary global manager resides in some embodiments. Data regarding the global desired configuration is received and stored in the database using a series of database transactions which are initiated through a series of application programming interface (API) calls to the global manager. The database, in some embodiments, generates an update stream from the database transactions, that is used to provide the data regarding the desired configuration to the secondary global manager for replication of the database. The update stream also includes metadata associated with each transaction, such as timestamp information that can be used for data ordering, as well as database status to prevent race conditions for access. In some embodiments, the database is shared by the primary global manager with other applications (e.g., a local manager) on the same computing device. In some such embodiments, data for replication to the secondary global manager is tagged so that only data associated with the primary global manager is replicated and other data associated with other applications on the computing device is not replicated.
Each global manager's database also includes in some embodiments a set of queues, each of which is reserved for a local manager at each of the physical sites, as well as one queue reserved for the secondary global manager. The database at the secondary global manager also includes these queues. When the primary global manager receives the global desired configuration for the logical network, the global manager stores portions of the global configuration in each queue, based on the relevance of the portions to the configuration of the logical network at the queue's corresponding physical site. In some embodiments, a broker service of the global manager identifies the relevant portions for each physical site (e.g., based on the span of the logical network elements), as described in further detail below.
In some embodiments, a set of asynchronous channels connect the primary global manager to the local managers and the secondary global manager at different sites. Some embodiments execute a set of site managers at each global manager to manage the channels, with each site manager corresponding to one of the physical sites. The channels retrieve data from the queues in the database and provide the retrieved data to the destination database at the corresponding physical site. These channels maintain the connections between physical sites and in some embodiments guarantee various connection parameters (e.g., the minimum bandwidth, the maximum roundtrip time, etc.) that are required for replication of data to the secondary global manager and dissemination of data to the local managers. The channels also identify the active machine for each manager, when the managers are implemented by a cluster of machines.
In some embodiments, the global desired configuration of the logical network is expressed as a hierarchical tree (also referred to as a global policy tree) with nodes and connections between the nodes. In some embodiments, the nodes represent logical network elements that span one or more sites and logical network policies that apply to those elements, and the connections represent relationships between the nodes (e.g., parent-child relationships, logical network connections, etc.). The logical network elements include logical forwarding elements that forward data in the logical network, e.g. logical routers, logical switches, etc. These logical network elements also include logical constructs in some embodiments, such as logical ports associated with the logical forwarding elements, domains that are logical groupings of one or more sites (e.g., geographic regions), and groups of logical network endpoints that share one or more attributes (e.g., operating system, region, etc.). Each logical network element is implemented in some embodiments by physical forwarding elements (PFEs) executing on computing devices at the sites that are spanned by that logical network element. The logical network policies include forwarding policies, service policies, and security policies, and are applied in some embodiments to govern the behavior of the logical forwarding elements.
The global manager performs a span calculation in some embodiments for each logical network element, to determine which logical network elements are implemented at each site. In some embodiments, the span calculation is based on the relationships between the nodes in the global policy tree. For example, when the relationship between two nodes is a parent-child relationship, the child node may inherit its span from the parent node. In other cases, however, a child node does not inherit the entire span of its parent node. As another example, when the relationship between two nodes is a dependence of one node on another node, expressed as a reference from one node to the other, the span of the dependent node will depend on the referenced node. Some logical network elements also have a pre-defined span in some embodiments, e.g., defined by an administrator of the network.
The relevant portion of the global desired configuration that is provided to each local manager in some embodiments is a site-specific policy subtree of the global policy tree, corresponding to the logical network elements implemented at that site. The subtree for each site only includes nodes for logical network elements that span the site. In some embodiments, the subtree is generated by first identifying the span of each node in the global policy tree. The global policy tree is then parsed, using the identified span for each node, to generate the policy subtree for each site.
The local manager at each site uses the relevant portion of the global desired configuration, received from the global manager, to manage the logical network at the site. For example, in some embodiments, the local manager uses the relevant portion to generate and provide configuration data to the control plane of the logical network (e.g., a cluster of controllers at each site). In some embodiments, these controllers identify computing devices at the site which execute physical forwarding elements, and distribute the configuration data to the identified computing devices. Some embodiments have local controllers (also referred to as chassis controllers) that execute on one or more of the computing devices, and which receive the configuration data from the controller cluster. The local controllers use the configuration data to configure the physical forwarding elements to implement the logical network elements. Each site's controller cluster also creates mappings between logical addresses (e.g., MAC addresses of logical network endpoints executing on the computing devices) and physical addresses (e.g., IP addresses of tunnel endpoints at the computing devices), and distributes these mappings to each computing device to which they are relevant, as well as to other controller clusters at other sites that require the data.
In some embodiments, the computing devices at each site also execute machines along with the physical forwarding elements and the local controllers. These machines include logical network endpoints, which are sources and destinations of data message traffic (e.g. computers, virtual machines, containers, etc.), and service machines, which perform services on the data traffic (e.g. firewalls, load balancers, etc.). A machine is located at a single site at a given time, but may be migrated between sites in some embodiments. These machines connect in some embodiments to the physical forwarding elements in order to exchange data messages in the network. In some embodiments, logical network endpoints at each site are logically organized into groups which can span multiple sites. The service machines in some embodiments apply some of the logical network policies to the data messages exchanged between groups of endpoints based on policies that are defined in terms of these groups.
Policies are defined in some embodiments at the global manager, e.g., by an administrator of the logical network. In some embodiments, policies are one or more service rules which are enforced at the sites on data message flows based on a set of flow attributes. The global manager in some embodiments distributes the service rules to local managers at the sites at which the policy is to be enforced (i.e., the policy's span). The local manager uses the service rules to generate configuration data for distribution by controllers, to configure the data plane (i.e., the forwarding elements and the service machines) to enforce the received service rules on data message flows that are associated with groups of logical network endpoints.
The policies are defined in some embodiments by reference to groups of logical network endpoints that span one or more sites. The groups are defined in some embodiments at the global manager, e.g. by an administrator of the logical network. The service rules refer to these groups in some embodiments by using a group identifier that is assigned at the global manager when the groups are defined. The definitions of these groups are distributed to the sites spanned by the policy, so that the controllers at these sites are able to configure the data plane to enforce the policy.
As discussed above, the service rules for a given policy are enforced on data messages that are associated with a group of logical network endpoints spanning one or more sites. Service rules have multiple fields in some embodiments, including source fields, destination fields, and action fields. Some embodiments refer to these groups in these fields by using group identifiers that are assigned to the groups (e.g., by the global manager when the groups are defined). For example, when the group identifier for the group is referenced in a source field of the service rule, the service rule is enforced on data messages that originate from machines in that group. When the group identifier for the group is referenced in a destination field of the service rule, the service rule is enforced on data messages that are directed to machines in that group.
The service rule is enforced on data messages whose attributes match those specified in the rule. For example, the rule may specify a flow 5-tuple (source IP address and port number, destination IP address and port number, and transmission protocol). Alternatively, the rule may specify other attributes of the flow. The PFEs at each site identify matching flows by performing match classification operations on each data message, to determine if the flow attributes match those specified in the service rule. When a data message matches the service rule, it is provided to a service machine in some embodiments, which performs the action that is specified by the rule on the matching data message. These actions include dropping the data message, allowing the data message, or performing other service operations on the data message. For example, a service operation may modify the header of the data message, to redirect the message to a different destination. These service operations include load-balancing, firewall, Dynamic Host Configuration Protocol (DHCP), Network Address Translation (NAT), and other services.
Some embodiments define domains that are groups of one or more sites. The domains are defined in some embodiments at the global manager, e.g. by an administrator of the logical network. Some domains are specific to a single physical site, and are referred to as locations. Other domains are logical groups of one or more sites, and are referred to as regions. Some embodiments restrict each site to membership in a single location and a single region. In other words, a location may not have more than one site, and a site may not be in more than two regions. The member sites of each domain define the domain's span. In some embodiments, domains are defined at the global manager and represented as nodes in the policy tree. As noted above, policies are also defined as nodes in the policy tree, and in some embodiments, policies are defined as child nodes of domain nodes. In such cases, the policy (and therefore, any service rule of the policy) inherits the span of its parent domain, i.e. the policy is enforced at the sites which are members of its parent domain.
Furthermore, groups are also represented in some embodiments as child nodes of a domain. In such cases, the group inherits the span of its parent domain, i.e. the group is defined at the sites which are members of its parent domain. Logical network endpoints located at a site may join a group if the group is defined at that site (i.e., if the group spans that site).
In some embodiments, a policy's service rules distributed to a first set of sites for applying to a first group of machines defined at those sites refer to a second group of machines defined at a second, different set of sites. In other words, the span of the referenced group does not include the span of the policy. Since such service rules reference a group of machines that is undefined for one or more of the sites in the first set of sites, not all of the machines in the first group are able to enforce the policy. Therefore, some embodiments identify which sites in the first set of sites do not have the definition of the referenced group, and distribute the definition of the referenced group to those identified sites.
For example, the policy might be defined in a first domain, which has the first set of sites as members, while the group referenced by the policy is defined in a second, different domain, which has the second set of sites as members. The policy therefore inherits span from the first domain, and the referenced group inherits span from the second domain. In such embodiments, the definition of the referenced group is distributed to the identified sites by extending the span of the group to include the span of the policy. This is done at the global manager (e.g., by an administrator of the logical network) in some embodiments, by creating a reference group in the first domain which references the group in the second domain. Upon receiving this modification, the global manager repeats the span calculation and determines that the group now spans the first set of sites. The global manager then provides the portion of the desired configuration (which now includes the referenced group) to the local managers at the first set of sites, which are now able to configure their respective controllers to configure the physical forwarding elements and service machines at the first set of sites to enforce the rule.
Some embodiments provide, upon request, the realized state of one or more of the logical network elements of the logical network. In some embodiments, the global manager receives the request for the realized state of the logical elements from an administrator of the logical network. For each logical network element in the request, the global manager identifies the sites spanned by the element and sends requests to those identified sites' local managers. These requests are sent in parallel in some embodiments. After receiving the realized state from each of the sites spanned by each of the logical network elements, the global manager combines the received realized state from each site and presents the realized state in response to the initial request. For example, the realized state may be presented in a report for an administrator of the logical network in some embodiments.
The realized state of a logical network element indicates in some embodiments whether the desired configuration (also referred to as the desired state) of the logical network element has been realized at each of the sites that the element spans. Some embodiments describe the realized state partly in terms of a deviation from the desired state. In some embodiments, the realized state for a logical network element (e.g., a logical router, logical switch, etc.) describes how the logical network element is implemented at each of the physical sites that it spans. For example, the realized state includes data from PFEs executing on computing devices at each of the spanned sites to implement the logical network element. Identifying the spanned sites is achieved in some embodiments by performing a span calculation as described above, or determined from the definition of the logical network element (e.g., by an administrator of the logical network). For each logical network element, the realized state received from each local manager is a subset of the total realized state of that logical network element, just as the desired state for the logical network element that is provided to the local manager is a subset of the global desired state of that logical network element defined at the global manager.
In some embodiments, the global manager also receives notifications from the local managers. Unlike requests for realized state which are provided on-demand (e.g., pulled from the local managers), these notifications are automatically pushed from the local managers. The notifications pertain in some embodiments to site-specific implementation problems or situations. Notifications are provided in some embodiments to the GM as the events they describe occur (e.g., in real-time or near-real-time rather than on-demand). These notifications are also displayed in some embodiments as a report (e.g., for review by an administrator of the logical network).
One type of notification in some embodiments is an alarm from a local manager that the implementation of a logical network element has failed at the specific site. For example, the failure could be due to an IP address conflict with another network element defined locally at the local manager, of which the global manager was unaware.
Another type of notification in some embodiments is a message from the local manager that an administrator of the network has overridden the desired state of a logical network element at that particular site. This occurs in some embodiments by an administrator (e.g., a different administrator, possibly located in a different region or even country) using the local manager to modify the logical network rather than using the global manager. Some embodiments restrict overriding the desired state to networking-related configurations only, and prevent such overrides for security-related configurations (e.g., security policies).
A third type of notification in some embodiments is a message from the local manager that the deletion of a logical network element has failed at the site. In other words, the logical network element was deleted at the global manager (e.g. for being unneeded or unused), but at the local manager of one site, the logical network element is still in use. An example may be a first logical router defined locally at one site's local manager, and which connects to a second logical router defined at the global manager. The global manager is unaware of the first logical router, which does not exist at other sites, and a global administrator may attempt to delete the second logical router. The local manager at the site where the second logical router is in use as a link for the first logical router then sends a notification to inform the global manager that the second logical router has not yet been deleted from that site.
An additional example of a type of notification in some embodiments is a message from the local manager that a logical network endpoint (e.g., a virtual machine) has been migrated from one site to another site. Such endpoints are typically attached to a logical port of a logical network element (e.g., a logical switch) that spans both sites. When the endpoint is migrated to a different site, it retains its association with that same logical port in some embodiments, and seamlessly inherits the same security policies that are associated with that port, even at the new site.
In some embodiments, a machine may be migrated from one physical site to another physical site, while preserving the state of network policies that apply to the machine. For example, for a logical segment (logical switch) that spans at least two physical sites, a machine attached to a logical port of the segment at one of the spanned sites is migrated in some embodiments to a new logical port of the same segment at another one of the spanned sites. Migrating a machine in some embodiments comprises creating a new machine at a new host computer (at the new site), transferring configuration data to the new machine from the original machine, and removing the original machine from its host computer. In some embodiments, the configuration data comprises physical state data and logical network policies.
The state of the machine (e.g., runtime data) is transferred in some embodiments from a computing device executing the original machine at the source site (i.e., the source host) to a computing device executing the new machine at the destination site (i.e., the destination host). In some embodiments, the state includes a local port identifier assigned by a local manager at the source physical site, and a global port identifier assigned by the global manager. The global port identifier is associated with logical network policies defined through a user client (e.g., by an administrator of the network) in some embodiments as part of the desired configuration of the logical network. Upon receiving the transferred state data, the destination host overwrites at least a portion of the state of the new machine with the received state of the old machine.
In some embodiments, the destination host extracts the global port identifier from the received state data and provides it to the destination site's local manager. The local manager uses the global port identifier in some embodiments to retrieve logical network policies associated with the global port identifier from the global manager, and applies these policies to the new machine. In some embodiments, the local manager generates configuration data from the policies and provides the configuration data to the control plane of the logical network at the destination site (e.g., a set of controllers) which then distribute the configuration data to the destination host.
The preceding Summary is intended to serve as a brief introduction to some embodiments of the invention. It is not meant to be an introduction or overview of all of the inventive subject matter disclosed in this document. The Detailed Description that follows and the Drawings that are referred to in the Detailed Description will further describe the embodiments described in the Summary as well as other embodiments. Accordingly, to understand all the embodiments described by this document, a full review of the Summary, Detailed Description and the Drawings is needed. Moreover, the claimed subject matters are not to be limited by the illustrative details in the Summary, Detailed Description and the Drawings, but rather are to be defined by the appended claims, because the claimed subject matters can be embodied in other specific forms without departing from the spirit of the subject matters.
The novel features of the invention are set forth in the appended claims. However, for purposes of explanation, several embodiments of the invention are set forth in the following figures.
In the following detailed description of the invention, numerous details, examples, and embodiments of the invention are set forth and described. However, it will be clear and apparent to one skilled in the art that the invention is not limited to the embodiments set forth and that the invention may be practiced without some of the specific details and examples discussed.
Some embodiments of the invention provide a network management system for managing a federated logical network spanning multiple physical sites (e.g., datacenters). The desired configuration of the logical network is defined by one or more network administrators using a set of network managers. The logical network includes logical network elements that span one or more sites and logical network policies that apply to the elements at those sites. Any such elements or policies that span multiple physical sites are defined through the global manager.
In some embodiments, the global manager receives (e.g., via a user client) a global desired configuration for the logical network, identifies a relevant portion of the global desired configuration for each physical site, and provides the identified portion to the site's corresponding local manager. In addition, the global manager collects information about the realized state of the logical network across the sites from the local managers and provides this information (e.g., to an administrator of the logical network) for troubleshooting and management purposes.
The logical network elements of some embodiments include logical forwarding elements (LFEs), such as logical switches (to which logical network endpoints attach) and logical routers. Each LFE (e.g., logical switch or logical router) is implemented across one or more physical sites, depending on how the LFE is defined by the network administrator. In some embodiments, the LFEs are implemented within the physical sites by physical forwarding elements (PFEs) executing on host computers that also host logical network endpoints of the logical network (e.g., with the PFEs executing in virtualization software of the host computers) and/or on edge devices within the physical sites. The LFEs transmit data traffic between logical network endpoints (e.g., data compute nodes (DCNs)) (i) within a datacenter, (ii) between logical network endpoints in different datacenters, and (iii) between logical network endpoints in a datacenter and endpoints external to the logical network (e.g., external to the datacenters). The edge devices, in some embodiments, are computing devices that may be bare metal machines executing a datapath and/or computers on which logical network endpoints execute to a datapath. These datapaths, in some embodiments, perform various gateway operations (e.g., gateways for stretching logical switches across physical sites, gateways for executing centralized features of logical routers such as performing stateful services and/or connecting to external networks).
As in this example, logical routers, in some embodiments, may include T0 logical routers (e.g., router 105) that connect directly to external networks 125 and T1 logical routers (e.g., router 110) that segregate a set of logical switches from the rest of the logical network and may perform stateful services for endpoints connected to those logical switches. These logical routers 105-110, in some embodiments, are defined by the network managers to have one or more routing components, depending on how the logical router has been configured by the network administrator.
T1 logical routers may be connected to T0 logical routers in some embodiments (e.g., T1 logical router 110 connecting to T0 logical router 105). These T0 logical routers 105, as mentioned, handle data messages exchanged between the logical network endpoints and external network endpoints. As shown, the T0 logical router 105 includes a DR 225 as well as a set of SRs 230-240. In some embodiments, T0 logical routers include an SR (or multiple SRs) operating in each physical site spanned by the logical router. In some or all of these physical sites, the T0 SRs 230-240 connect to external routers 241-243 (or to top of rack (TOR) switches that provide connections to external networks).
In addition to the logical switches 115 and 120 (which span all of the physical sites spanned by the T1 DR 205),
Lastly, the network management system also defines backplane logical switches that connect each set of SRs. In this case, there is a backplane logical switch 280 connecting the T1 SRs 210-220 and a backplane logical switch 285 connecting the T0 SRs 230-240. These backplane logical switches, unlike the transit logical switches, are stretched across the physical sites spanned by their respective logical routers. When one SR for a particular logical router routes a data message to another SR for the same logical router, the data message is sent according to the appropriate backplane logical switch.
As mentioned, the LFEs of a logical network may be implemented by PFEs executing on source host computers as well as by the edge devices.
The edge devices 325, in some embodiments, execute datapaths 327 (e.g., data plane development kit (DPDK) datapaths) that implement one or more LFEs. In some embodiments, SRs of logical routers are assigned to edge devices and implemented by these edge devices (the SRs are centralized, and thus not distributed in the same manner as the DRs or logical switches). The datapaths 327 of the edge devices 325 may execute in the primary operating system of a bare metal computing device and/or execute within a VM or other data compute node (that is not a logical network endpoint) operating on the edge device, in different embodiments.
In some embodiments, as shown, the edge devices 325 connect the physical sites 305-315 to each other (and to external networks 125). In such embodiments, the host computers 320 within a physical site can send data messages directly to each other, but send data messages to host computers 320 in other physical sites via the edge devices 325. When a source logical network endpoint (e.g., VM 330) in the first physical site 305 sends a data message to a destination logical network endpoint (e.g., VM 335) in the second physical site 310, this data message is first processed by the PFE 340 executing on the same host computer 320 as the source VM 330, then by an edge device 325 in the first physical site 305, then an edge device 325 in the second physical site 310, and then by the PFE 345 in the same host computer 320 as the destination logical network endpoint 335.
More specifically, when a logical network endpoint sends a data message to another logical network endpoint, the PFE executing on the host computer at which the source logical network endpoint resides performs logical network processing. In some embodiments, the source host computer PFE set (collectively referred to herein as the source PFE) performs processing for as much of the logical network as possible (referred to as first-hop logical processing). That is, the source PFE processes the data message through the logical network until either (i) the destination logical port for the data message is determined or (ii) the data message is logically forwarded to an LFE for which the source PFE cannot perform processing (e.g., an SR of a logical router).
In some embodiments, the global manager 420 receives a global desired configuration for the logical network via one or more user clients 440. Each of the local managers 425-435 also receives in some embodiments a (site-specific) desired configuration for the logical network via the user clients 440. The desired configuration is provided to the managers 420-435 and 460 from a user client 440 in some embodiments using a representational state transfer (REST) application programming interface (API), and is represented by dashed lines in
In some embodiments, as illustrated in
Some embodiments employ a secondary global manager 460, in an active-standby arrangement with the (primary) global manager 420. The primary global manager 420 is asynchronously synchronized (e.g., not real-time) with the secondary global manager 460 as a standby for failover scenarios. This asynchronous replication is represented by a dot-dash line in
The secondary global manager 460 executes in some embodiments on the same computing device 450 as the local manager 430 managing its site 410, as illustrated in
The primary global manager 420, the secondary global manager 460, and the local managers 425-435 are in some embodiments separate modules of a single application, and in other embodiments are separate applications. These applications in some embodiments execute as one or more processes within machines that execute on host computers at each physical site. Some embodiments deploy one or more of the managers 420-435 and 460 as a cluster of machines at their physical site, with each machine executing on a different computing device at the same site, as described in further detail below with reference to
The global manager 420 also includes a number of additional modules, including an API processor 706 for receiving the user client input via a REST API, a core global manager service 707 that writes data to the database 710, a persistent work queue 711 in the database 710 to maintain causality for incoming create/update/delete (CUD) events, a log replication module 730 to replicate CUD events to the database 460 at the secondary global manager 460, a broker/span service 735 to perform span calculations on the CUD events (and the logical network elements referenced therein), an asynchronous replication (AR) module 300 which includes dedicated persistent queues 725-727 for disseminating CUD events to different local managers at different physical sites, and site managers 745 for maintaining connection parameters used by the AR module to establish channels to the other local managers. Each of these modules and their functionality are described in further detail below.
In some embodiments, the databases 710 and 712 are distributed databases (e.g., a shared log) implemented across a set of storage devices at the managers' respective physical sites. In addition, in some embodiments, the global managers 420 and 460 are implemented as a cluster of machines executing on separate computing devices at its respective physical site.
In some embodiments, the manager application that runs on the machines 805-815 has separate modules for the global manager and local manager, which can be enabled or disabled as required. Alternatively or conjunctively, in some embodiments, at least one of the machines 805-815 is a dedicated global manager machine, with a separate machine (not shown) for running an application for the local manager, executing on either the same host computers 820-830, or different host computers (as described above with reference to
The manager cluster stores desired configuration data in a distributed database system that is managed by one or more instances 840-850 of a database that execute on the host computers 820-830 in some embodiments. The database executes within the local manager machine on the host in some embodiments, though they are shown as separate in the figure for clarity. The database instances 840-850 communicate with each other over the physical network 835 at the physical site (e.g., the datacenter network fabric) that is used by the manager cluster. The database instances 840-850 collectively appear to the manager cluster as the single logical database 710. In some embodiments, the instances are shards or slices of the database. In other embodiments, each instance is a node with a full copy of the data (e.g., as illustrated in the example of
In some embodiments, the database instances 840-850 are nodes of a distributed log that is stored on the host computers 820-830. Entries in the distributed log provide an ordered, persisted history of updates to the state of different logical network elements and logical network policies, which the manager cluster accesses via application programming interfaces (APIs) provided by the database instances 840-850. The distributed log and the database APIs are described in more detail by U.S. Pat. No. 10,540,119, which is incorporated herein by reference.
Returning to the example of
In some embodiments, the manager service 707 is the cluster of manager machines 805-815, as described above with reference to
In some embodiments, the database 710 generates one or more update streams from the series of transactions.
The process 1000 begins by receiving at 1005 data describing a desired configuration of the logical network. The received data is in some embodiments one or more create, update, or delete (CUD) events received at the global manager 420 as a series of API transactions, each CUD event affecting one or more logical network elements spanning one or more of the physical sites. For example, in
At 1010, the process 1000 tags the received CUD event using metadata associated with each transaction, such as timestamp information that can be used for data ordering, and database status to prevent race conditions for access. The metadata also includes in some embodiments parameters associated with the API call, such as user ID, source address, etc. Furthermore, in some embodiments, the metadata includes span information that was specified (e.g., by an administrator) when the CUD event was defined through the user client 440. In the example of
In some embodiments, not all data stored by the global manager in the database is necessarily intended for replication. Data to be replicated in some embodiments includes policy tables, permissions, physical site information, and other data that the secondary global manager would require in order to assume active status in the event of failure of the primary global manager. Other database tables, such as those that pertain to managing the network in active mode, are not necessary for replication to the secondary global manager in standby mode. In addition, state information about the realization status of the logical network would not need to be replicated to the secondary global manager, since the realization status would be obsolete by the time a failover scenario occurred. Some embodiments distinguish data to be replicated from data that is not to be replicated, by tagging the data for replication.
In some embodiments, the process 1000 also tags (at 1010) the CUD event with an identifier that indicates that the event is to be replicated. For example, in
At 1015, the process 1000 stores the tagged CUD event, along with at least some of the metadata, in the database 710. Specifically, in some embodiments, the process 1000 stores the tagged data in each of the database nodes 840-850. In embodiments where the database 710 is a distributed shared log, the tagged data is appended to the log. In the example of
As noted above, in some embodiments the database 710 is shared by the primary global manager 420 with a local manager 425 (e.g., on the same computing device 445, as illustrated in the example of
The process 1000 pushes at 1020 the CUD event 905 and associated metadata 910 to the log replication module 730. In some embodiments, the process 1000 pushes the CUD event to a work queue 711 in the database (not shown in
As noted above, the database 710 generates an update stream which pushes newly-written CUD events to a log replication module 730, for replication to the secondary global manager 460. Returning to
When the primary global manager 420 receives the global desired configuration for the logical network, the global manager stores various portions of the global configuration in the persistent queues 725-727, based on the relevance of the portions to the configuration of the logical network at the queue's corresponding physical site. In some embodiments, a broker service 735 of the global manager 420 identifies the relevant portions the global desired configuration for each physical site, for example based on the span of the logical network elements, as described in further detail below. Span is determined in some embodiments by a span service, which in some embodiments is part of the broker service 735 (as depicted in
The process 1200 begins at 1205 by receiving a notification of a new CUD event that has been stored in the database 710. This notification is sent from the global manager 420 or a database instance 840 in some embodiments (e.g., as part of operation 1020 of process 1000). In response to the notification, the broker service 735 retrieves the CUD event. In other embodiments, the broker service 735 periodically queries the database for new CUD events. In either case, in some embodiments, the CUD events are stored in a work queue 711 in the database 712, to preserve their causality and order. The broker service 735 then sequentially retrieves each CUD event from this queue. In the example of
At 1215, the process 1200 determines whether the database 710 already stores span information 1147 for the CUD event. In some cases, the span for the CUD event is pre-defined (e.g., when specifying the CUD event through the user client 440) and is stored in the database 712 as metadata (e.g., metadata 910) associated with the event. If the span for the CUD event is available, at 1217 the process 1200 retrieves the span information and proceeds to operation 1230, which is described below.
If the span for the CUD event is not available, then the process 1200 determines at 1220 which physical sites are spanned by the CUD event. In some embodiments, the broker service 735 invokes a span service 736 to perform a span calculation to determine the span, taking into account the definition of the logical network elements that are referenced by the CUD event, as well as the relationship of those logical network elements with other logical network elements in the logical network. Span calculations are described in further detail below with reference to
Based on the span information, at 1230 the process 1200 stores a copy of the event in each persistent queue that corresponds to one of the spanned physical sites. In the example of
As noted above, a CUD event refers to one or more logical network elements in some embodiments. If the CUD event refers to different logical network elements with different span attributes, then in some embodiments the broker service 735 copies the relevant portions of the CUD event (corresponding to the different logical network elements) to the relevant queues instead of the entire event.
As noted above, the persistent queues are first-in, first-out (FIFO) queues. In the example of
The subsequent CUD events B-F 1120-2940 are similarly queued in the same order as received, as illustrated in
Returning to
In some embodiments, the AR module 740 maintains a dedicated asynchronous channel that connects the primary global manager 420 to the secondary global manager 460. In other embodiments, replication to the secondary global manager 460 is handled by a dedicated log replication module 730, which was described above with reference to
In some embodiments the secondary global manager 460 also has a receiver module 742 for establishing the connection to the log replication module 730 of the primary global manager, receiving the replicated data, and writing the data to the database 712. If the secondary global manager 460 becomes active (e.g., due to failover), then in some embodiments the receiving module 742 assumes the role of the replication module.
The AR modules 740 and 750-765 maintain the channels between the physical sites, and in some embodiments guarantee various connection parameters (e.g., the minimum bandwidth, the maximum roundtrip time, etc.) that are required for replication of data to the secondary global manager and dissemination of data to the local managers. In embodiments where the secondary global manager and/or the local managers are implemented as a cluster of machines, the channels also identify the leader machine for each manager cluster.
Some embodiments execute a set of site managers 745 at the primary global manager 420 that provide information (e.g., connectivity statistics, IP addresses, etc.) about the physical sites to the AR module 740 to use in maintaining the channels, each site manager corresponding to one of the physical sites 405-415. The site managers 745 execute separately from the AR module 740 in some embodiments (as depicted in
As noted above, the broker service 735 enqueues CUD events to the persistent queues 725-727 of the AR module 740. In some embodiments, the AR module 740 polls its persistent queues 725-727 to determine if there are new CUD events. If data (e.g., corresponding to CUD events) is found in a queue, the AR module 740 retrieves the data (i.e., dequeues the CUD events) and transmits the retrieved data over the channel corresponding to that queue to the AR module of the local manager at the corresponding site.
The process 1300 begins at 1305 by querying a persistent queue that is dedicated to one of the local managers 425-435 at one of the physical sites 405-415. The process 1300 queries the selected queue to see if there are any new CUD events in the queue. The queue will potentially include CUD events that were copied to the queue by the broker service 735 after performing a span calculation, as described in
If the process 1300 determines (at 1310) that there is no new data in the selected queue, then the process 1300 returns to 1305 after a specified period of time. The period of time in some embodiments is a time-out value, which varies according to the specific connection parameters of the channel to the physical site. This partly depends in some embodiments on the location of both the global manager 420 and the specific local manager at the site associated with the persistent queue (e.g., the different scenarios for placement of the global manager 420 and the local managers, as described above in
At 1320, the process 1300 then provides the retrieved data to the corresponding physical site, through a channel that connects the AR module 740 of the primary global manager 420 and the corresponding AR module of the physical site's local manager. For example, if the selected queue is queue 725, then the process 1300 transmits the data to the AR module 755 of the local manager 425 at the same site 405, over the network fabric of site 405 through its dedicated channel (represented by a dotted line in
In this manner the AR module 740 sends the data to the destination corresponding to the queue from which it retrieved the data, whether that destination is at the same physical site or a different physical site, through whatever intervening network infrastructure is necessary. The process 1300 then ends.
In the example of
As described above with reference to
The process 1500 begins at 1505 by receiving data from the primary global manager. Specifically, the data is received from the sending AR module 740 of the primary global manager, via a dedicated channel between the sending AR module and the receiving AR module of the local manager (e.g., AR module 765 of local manager 430 in
At 1510, the process 1500 determines if the CUD event (or events) associated with the received data is valid. The validation is based on whether there is any error or inconsistency in applying the CUD event to the configuration of the logical network at the physical site. In addition, other validations are rules in some embodiments that govern whether a logical network element can be updated to stretch its span to a new site. For example, prior to stretching an element to a new site, the security policies that affect the element must be stretched to the new site first. Otherwise, a loophole is created where the security policies applicable to the logical network element are not fully applied. In some embodiments, the validation is performed by the local manager service 1410, which retrieves the CUD event from the ingress queue 1402 and stores it in the database 1405.
If the process 1500 determines that the CUD event is invalid, then a notification for the primary global manager of the failure to validate the CUD event is generated at 1520. The notification in some embodiments is a notification event that is queued in an egress queue (not shown) of the AR module 765, to be sent back to the AR module 740 at the global manager 420 via the same asynchronous channel (e.g., the dotted line in
Invalid events are not dropped in some embodiments, but are also persisted in order to maintain causality. The invalid event will still be accepted, and an intervention (e.g., by an administrator of the network) will be required to resolve the invalidity. The notification events are described in more detail with reference to
At 1530, the process 1500 applies the CUD event to the local desired configuration of the logical network at the physical site. The desired configuration of the logical network is expressed as a policy tree in some embodiments, which is described in further detail below with reference to
For example, if the validated CUD event is a create event, then a logical network element defined by the event is created within the desired configuration stored in the database 1405. If the CUD event is an update event, then the desired configuration of a logical network element referenced by the event is updated within the desired configuration stored in the database 1405. If the CUD event is a delete event, then a logical network element referenced by the event is deleted within the desired configuration stored in the database 1405.
At 1540, the process 1500 uses the (now modified) desired configuration of the logical network to generate and provide configuration data to the control plane of the logical network (e.g., a central controller or cluster of controllers at each site). In some embodiments, these controllers identify computing devices at the site which execute physical forwarding elements, and distribute the configuration data to the identified computing devices. In some embodiments, different logical network elements span different computing devices (e.g., host computers, edge devices, etc.). Each logical network element is implemented in some embodiments by physical forwarding elements executing on the identified computing devices at the sites that are spanned by that logical network element. In other words, a logical network element is implemented by at least one physical forwarding element at each site which it spans. Some embodiments have local controllers (also referred to as chassis controllers) that execute on one or more of the computing devices alongside the physical forwarding elements, and which receive the configuration data from the controller cluster. The local controllers use the configuration data to configure the physical forwarding elements to implement the logical network elements at each computing device. The process 1500 then ends.
The process 1600 begins at 1605 by receiving a CUD event directly from a user client 440. For example, as illustrated in
The process 1600 determines at 1610 whether the CUD event (or events) is valid. The validation is based on whether there is any error or inconsistency in applying the CUD event to the configuration of the logical network at the physical site. In some embodiments, the validation is performed by the local manager service 1410, either directly upon receipt from the API processor 1415, or after retrieving the event from the database 1405.
If the process 1600 determines that the CUD event is invalid, then a notification for the failure to validate the CUD event is generated at 1617. The notification in some embodiments is a notification event, that is provided to the user client 440 for intervention (e.g., by an administrator of the network). The process 1600 then ends.
If the process determines that the CUD event is valid, then the process determines at 1615 whether the event is a local event. In other words, it determines if the CUD event only references logical network elements defined at the local site. These elements, if defined through the local manager, have no span beyond the physical site, and are not known to the global manager 420 in some embodiments. If the CUD event is a local event, then the process 1600 continues to 1635, which is described below.
If the process 1600 determines that the CUD event is not a local event, i.e. it references a logical network element that was defined at the global manager 420, then the process determines at 1620 whether the event overrides the globally-defined desired configuration of the logical network element. This determination is made in some embodiments by applying a set of priority rules to the CUD event to determine whether the CUD event is allowed to override the globally-defined desired configuration. For example, some embodiments only allow overriding of the desired configuration by a local CUD event for networking-related configurations (e.g., message forwarding rules and policies) or configuration profiles (timers, etc. which are affected by the local site's parameters, such as latency). In such cases, the local CUD event would have priority.
As another example, some embodiments prevent overrides of the desired configuration by a local CUD event for security-related configurations. In such cases, the globally-defined desired configuration would have priority. In addition, in some cases the event is an emergency-related event, which is only recognized by the local manager and therefore does override any related global configuration. If the event does not have priority to override the global configuration (e.g., according to the priority rules), then the process continues to 1617, which was defined above.
At 1625, if the process 1600 determines that the CUD event does have priority to override the globally-defined desired configuration, then a notification for the primary global manager of the override event is generated at 1625. The notification in some embodiments is a notification event that is queued in an egress queue (not shown) of the AR module 765 to be sent back to the AR module 740 at the global manager 420, via the same asynchronous channel (e.g., the dotted line in
At 1630, the process 1600 creates a local copy of the logical network element the configuration of which is to be overridden by the CUD event. The original logical network element from the global manager 420 remains as a read-only object in the local manager's database 1405. This local copy (also referred to as a shadow object) is the target of the CUD event instead of the original.
At 1635, the process 1600 applies the CUD event to the local desired configuration of the logical network at the physical site. If the event is an override, then the process 1600 applies the CUD event to the shadow copy of the object instead of the original object received from the global manager 420. The desired configuration of the logical network is expressed as a policy tree in some embodiments, which is described in further detail below with reference to
For example, if the CUD event is a create event, then a logical network element defined by the event is created within the local desired configuration stored in the database 1405. If the validated CUD event is an update event, then the desired configuration of a logical network element referenced by the event is updated within the local desired configuration stored in the database 1405. If the validated CUD event is a delete event, then a logical network element referenced by the event is deleted within the local desired configuration stored in the database 1405.
At 1640, the process 1600 uses the desired configuration of the logical network to generate and provide configuration data to the control plane of the logical network (e.g., a central controller or cluster of controllers at each site). The process 1600 then ends.
As noted above with reference to
The controllers 1420, in addition to distributing configuration data to computing devices such as host computers 320 and edge devices 325, receive physical network to logical network mapping data from the computing devices in some embodiments and share this information across datacenters. For example, these mappings are in some embodiments between logical addresses (e.g., MAC addresses of logical network endpoints i.e. VMs 1430 executing on the computing devices) and physical addresses (e.g., IP addresses of tunnel endpoints at the computing devices). The controllers 1420 retrieve and share tunnel endpoint to logical network address mapping data between the host computers 320 and edge devices 325 at the physical site 410, and also share that data with the controllers at the other physical sites 405 and 415 so that those controllers can share this data with their own site's host computers and edge devices.
In some embodiments, the computing devices at each site also execute machines alongside physical forwarding elements and local controllers. These machines include logical network endpoints, which are sources and destinations of data message traffic (e.g. computers, virtual machines, containers, etc.), and service machines, which perform services on the data traffic (e.g. firewalls, load balancers, etc.). A machine is located at a single site at a given time, but may be migrated between sites in some embodiments. These machines connect in some embodiments to the physical forwarding elements in order to exchange data messages in the network.
In some embodiments, the global desired configuration of the logical network is expressed as a hierarchical tree (also referred to as a global policy tree) with nodes and connections between the nodes. The global policy tree is stored by the primary global manager 420 in its database 710. A replicated global policy tree is also stored by the secondary global manager 460 in its database 712. In some embodiments, the nodes represent logical network elements that span one or more sites and logical network policies that apply to those elements, and the connections represent relationships between the nodes (e.g., parent-child relationships, logical network connections, etc.). Cross-referencing between nodes is achieved by reference to a path through the tree's hierarchy (e.g., global-infra/sites/London/enforce-point/edge-cluster) which provides information about the span of each node.
The logical network elements include logical forwarding elements that forward data in the logical network, e.g. logical routers, logical switches, etc. For example, in
The locale service nodes for the T0 router and the T1 routers define these routers' span. For example, router T0 1705 spans sites A 1760, B 1777, and C 1765, while router T1B 1715 spans site A 1760. As more locale services are added to a T0 or T1 router, the router is stretched to the corresponding sites. Unlike router T1B 1715, router T1A 1710 does not have a locale service child node, and instead has a reference (dashed line) to router T0 1705. Therefore, router T1A 1710 inherits the span of router T0 1705 (i.e., router T1 A 1710 spans sites A 1760, B 1777, and C 1765). Certain child nodes also inherit that span automatically in some embodiments. Accordingly, the static route definitions 1730 under the T0 router 1705 also span sites A 1760, B 1777, and C 1765. The logical switch 1745 inherits the span of its parent router T1A 1710, which in turn derives its span from the reference to router T0 1705. Therefore, logical switch 1745 also spans sites A 1760, B 1777, and C 1765.
Each node in the global policy tree 1700 has multiple attributes that define configuration parameters, some of which are defined by the user and others of which are inherited. In some embodiments, span is not the only attribute that is inherited by a child node from a parent node. For example, certain T0 or T1 routers that span more than one site have one of the physical sites assigned as a primary site, with the other sites being secondary sites. If such a logical router has multiple service router (SR) components, then the SR component at the primary site takes precedence for certain operations. This configuration is specified (e.g., by an administrator of the network) for the router and is not part of the configuration of the locale services under the router.
The locale service nodes 1735, 1740, and 1750 have references (dashed lines) to edge clusters 1751 and 1752 at the respective sites A 1760 and B 1777. As noted above, in this example the T0 router 1705 also spans site C 1765, but the router's locale service for that site and therefore the corresponding reference to an edge cluster under the site C node 1765 is omitted for the sake of visual clarity. The locale service nodes are associated in some embodiments with the service routers described above with reference to
The logical switch 1745 is shown as a child node under router T1A 1710. Such logical switches, also referred to as segments, are restricted to the parent router if they are connected as child nodes (as in
Another type of segment in some embodiments is a VLAN-backed segment. These are defined with respect to a transport zone, which is a group of host devices at a single physical site. Therefore, the VLAN-backed segment can only span that single site where the transport zone is defined. In some embodiments, VLAN-backed segments are used as uplinks in some embodiments, to connect a logical router to an external physical router outside the logical network. In other words, the VLAN is between the Tier-0 router and the external router. Since multiple Tier-0 routers may connect to same external physical router, VLAN-backed segments are used in some embodiments to distinguish their traffic. Typically, connecting a logical Tier-0 router to physical router happens at a single physical site, since each site has its own connection to the wide-area network (e.g., the Internet) between the sites, i.e. a unique Internet Service Provider (ISP). Accordingly, VLAN-backed segments provide a way of logically isolating traffic from different Tier-0 routers to the same external router, even though the Tier-0 routers may be stretched across multiple sites and overlap in their span.
In the example of
Interfaces in some embodiments are uplinks or service ports. Interfaces connect to logical switches or segments, and then logical network endpoints (such as virtual machines, data compute nodes, or other types of workloads) are attached to those logical switches and segments. These endpoints also have their own services, such as DNS, TCP, etc.
In addition, the logical network elements include nodes for each physical site. For example, in
The logical network elements also include logical constructs in some embodiments, such as domains that are logical groupings of one or more sites (e.g., geographic regions), and groups of logical network endpoints that share one or more attributes (e.g., operating system, region, etc.). Domains are defined and represented as nodes in the global policy tree 1700 beneath the global root 1702. The domains are defined in some embodiments at the global manager 420, e.g. by an administrator of the logical network. Unlike sites, which represent a physical construct, domains are a logical construct, which serve as an envelope to group different logical entities together, e.g. for security purposes. For example, firewall policies or other policy micro-segmentation applied to the domain will automatically be applied to all groups of logical endpoints defined within the domain.
In some embodiments, there are different types of domains. For example, some domains are specific to a single physical site, and are referred to as locations. This type of domain acts as the container for all site-wide and site-specific configuration and policies. In some embodiments, a location domain is automatically created for each physical site in the federated logical network, and cannot be modified by the user.
Other domains are logical groups of one or more sites, and are referred to as regions. Regions are assigned to geographic regions in some embodiments. For example, in the example of
In some embodiments, domains are only created as top-level nodes beneath the global root 1702, and cannot be children of other domains or inherit span from other domains. Instead, the span of a domain is manually defined in some embodiments at the global manager (e.g., by an administrator of the logical network) as the sites that are members of the domain. The span is represented in some embodiments by a domain enforcement point, which is configured to reference the site enforcement point for whichever sites the domain is intended to span. These domain enforcement points are only used by the global manager in some embodiments, and are not used by the local managers. For example, in
In some embodiments, logical network endpoints at each site are logically organized into security groups which can span multiple sites. Service machines as well as managed forwarding elements executing on host computer apply logical network policies (such as network policy 1773) to the data messages exchanged between security groups of endpoints in some embodiments, based on policy rules that are defined in terms of these groups. Such security groups and network policies are defined at the global manager 420 through the user client 440 (e.g., by an administrator of the logical network). In some embodiments, security groups and network policies are represented in the global policy tree 1700 as child nodes of domains, and accordingly inherit their parent domain's span. In some embodiments, the span of a network policy is defined not only by its parent domain, but also by sites and/or domains which are referenced by the policy.
For example, in
Domain A 1770 also has a child node corresponding to a network policy 1773. The network policy is applicable to any groups defined under the same domain (e.g., group A 1771). In order to apply a policy to a security group, the span of the security group in some embodiments must include the span of the policy.
In some embodiments, network policies may also refer to security groups that are not in the same domain. For example, the network policy 1773 also references security group B 1772, which is in domain B 1775, even though the domain deployment map for the parent domain A 1770 does not include domain B 1775. Such cross-domain policies are discussed in further detail below with reference to
In some embodiments, some nodes derive their span from span leaders. Certain types of nodes are span leaders, for example a T0 router, a T1 router that is below the global root, a domain, or locale services. A T1 router that is defined beneath a T0 router is not a span leader, in some embodiments. A domain is the span leader for all security policies defined under the domain. Groups are also span leaders in some embodiments, and do not strictly inherit span from their parent nodes (e.g., domains) since the span of a group may be extended beyond its parent domain through reference groups, which are described in more detail below with reference to
As noted in the discussion above with reference to
As described above with reference to
The first security group is group A 1817, which is defined as all VMs which run operating system OS1 in Paris. This security group is defined logically at the global manager 420 (e.g., by an administrator of the logical network) to have a span of domain A 1770 (i.e., sites A and B). As a result, VMs A and D are automatically added to security group A 1817. Security policies which span domain A 1770 may be applied by an administrator of the network to security group A 1817, and would automatically therefore be applied to VMs A and D.
The second security group is group B 1819, which is defined as all VMs which run operating system OS2 in New York. This security group is defined logically at the global manager 420 (e.g., by an administrator of the logical network) to have a span of domain B 1775 (i.e., site C). As a result, VMs J and K are automatically added to security group B 1819. Security policies which span site C (i.e., domain B 1775) may be applied by an administrator of the network to security group B 1819, and would automatically therefore be applied to VMs J and K.
The third security group is group C 1820, which is defined as all VMs which run operating system OS1 in Europe. This security group is defined logically at the global manager 420 (e.g., by an administrator of the logical network) to have a span of domain A 1770 (i.e., sites A and B). As a result, VMs A, D, E, and G are automatically added to security group C 1820. Security policies which span site A, site B, or sites A and B may be applied by an administrator of the network to security group C 1820, and would automatically therefore be applied to VMs A, D, E, and G.
The fourth security group is group D 1825, which is defined as all VMs which run operating system OS2 in Europe. This security group is defined logically at the global manager 420 (e.g., by an administrator of the logical network) to have a span of sites A and B. As a result, VMs B, C, F, and H are automatically added to security group D 1825. Security policies which either span site A, site B, or sites A and B may be applied by an administrator of the network to security group D 1825, and would automatically therefore be applied to VMs B, C, F, and H.
The fifth security group is group E 1830, which is defined as all VMs which run operating system OS1 in all regions globally. This security group is defined logically at the global manager 420 (e.g., by an administrator of the logical network) to have a span of sites A, B, and C. As a result, VMs A, D, E, G, I, and L are automatically added to security group E 1830. Security policies which either span site A, site B, site C, or any combination of sites A-C, may be applied by an administrator of the network to security group E 1830, and would automatically therefore be applied to VMs A, D, E, G, I, and L.
As noted above, in some embodiments for a network policy to be applicable to a security group, the span of the security group must include the span of the network policy. For example, network policy 1773 in the example of
As discussed above, the global manager 420 performs a span calculation in some embodiments for each logical network element, to determine which logical network elements are implemented at each physical site. This is performed to provide the relevant desired global configurations to each physical site (e.g., through the asynchronous replicator channel, as described above in some embodiments with reference to
At 1910, the process 1900 populates a global policy tree (e.g., the global policy tree 1700 in
At 1915, the process 1900 selects a physical site that implements the logical network. In some embodiments, the physical sites are identified through auto-discovery of the physical sites as discussed above. For the selected physical site, the process 1900 parses at 1920 the global policy tree, to identify policy nodes whose span include the selected physical site. The process 1900 uses the identified nodes to generate (at 1925) relevant desired configuration for the selected physical site. In some embodiments, the process generates the relevant configuration by identifying portions of the global desired configuration. The process 1900 provides the generated relevant configuration (e.g., the portions of the global desired configuration) to the local manager at the selected physical site, through the asynchronous replicator channel as described above in some embodiments with reference to
The process 1900 determines, at 1935, whether there are additional physical sites. If there are additional sites, the process 1900 returns to 1915, which was described above. If there are no additional sites, the process 1900 ends.
The relevant portion of the global desired configuration that is provided to each local manager in some embodiments is a site-specific policy subtree of the global policy tree, corresponding to the logical network elements implemented at that site. The subtree for each site only includes nodes for logical network elements that span the site. The global policy tree 1700 is stored by the global manager 420 in its database 710.
In some embodiments, the global policy tree is parsed to identify the span of each node in order to generate a policy subtree for each physical site. In some embodiments, the subtree is generated by first identifying the span of each node in the global policy tree. The global policy tree is then parsed, using the identified span for each node, to generate the policy subtree for each site. The local manager at each site (or a management plane application, which may be separate from the local manager) uses the relevant portion of the global desired configuration, received from the global manager, along with any desired configuration received directly by the local manager itself, to manage the logical network at the site.
The process 2000 begins at 2005 by receiving a global policy tree (e.g., the global policy tree 1700 in
The process 2000 determines (at 2015) the span attribute for the selected node. In some embodiments, the span attribute is determined by a span calculation performed by the broker service 735 of the global manager 420. In some embodiments, the span calculation is based on the relationships between the nodes in the global policy tree. For example, when the relationship between two nodes is a parent-child relationship, the child node may inherit its span from the parent node. In other cases, however, a child node does not inherit the entire span of its parent node. As another example, when the relationship between two nodes is a dependence of one node on another node, expressed as a reference from one node to the other, the span of the dependent node will depend on the referenced node. Some logical network elements also have a pre-defined span in some embodiments, e.g. defined by an administrator of the network, which is stored in the database 710. In such embodiments, the process 2000 retrieves the span attributes from the database.
At 2020, the process 2000 uses the span attribute to assign the selected node to a policy subtree for each physical site that is spanned by the node. These policy subtrees are stored in some embodiments in the database 710. In some embodiments, the policy subtrees are associated with the dedicated persistent queues 725-727 that each correspond to the physical sites.
The process 2000 determines (at 2025) if there are additional nodes in the global policy tree. If there are additional nodes, then the process returns to 2010, which was described above. If there are no additional nodes, then the process continues to 2030. At 2030, the process uses the generated policy subtrees to generate relevant desired configuration for each physical site. In some embodiments, the process generates the relevant configuration from the policy subtrees by identifying portions of the global desired configuration. The process 2000 then ends.
Network policy 1773 is also preserved in the global policy subtree 2100. This policy 1773 is defined under domain A 1770, so in some embodiments it has a span of site A and site B, even though this subtree is specific to site A. In addition, as noted above with reference to
In some embodiments, a local manager also stores a separate policy tree, that is generated based on desired configuration received directly at the local manager instead of from the global manager 420.
In some embodiments, logical network elements defined in the local policy tree 2200 may reference logical network elements defined in the global policy tree 1700. For example, in
The global policy tree 1700 is stored by the primary global manager 420 in its database 710. A replica of the global policy tree 1700 is also stored by the secondary global manager 460 in its database 712. As noted above, in some embodiments the nodes also represent logical network policies that apply to the logical network elements. The logical network policies include forwarding policies, service policies, and security policies, and are applied in some embodiments to govern the behavior of the logical forwarding elements (e.g., by governing the behavior of the physical forwarding elements that implement the logical forwarding elements).
Policies are defined in some embodiments at the global manager 420 through a user client 440, e.g. by an administrator of the logical network. In some embodiments, policies are one or more service rules which are enforced at the sites on data message flows based on a set of flow attributes. The global manager 420 in some embodiments distributes the service rules to local managers 425-435 at the sites 405-415 at which the policy is to be enforced (i.e., the policy's span). Each local manager uses the received service rules to generate configuration data for distribution by controllers 1420-1425, to configure the data plane (i.e., the physical forwarding elements 345 and edge devices 325) to enforce the received service rules on data message flows between logical network endpoints, such VMs and other data compute nodes.
The policies are defined in some embodiments by reference to groups of logical network endpoints that span one or more sites, e.g. the security groups 1817-1830 described above with reference to
As discussed above, the service rules for a given policy are enforced on data messages that are associated with a group of logical network endpoints (e.g., VMs) spanning one or more sites. Service rules have multiple fields in some embodiments, including source fields, destination fields, and action fields. Some embodiments refer to these groups in these fields by using group identifiers that are assigned to the groups (e.g., by the global manager when the groups are defined). For example, when the group identifier for the group is referenced in a source field of the service rule, the service rule is enforced on data messages that originate from machines in that group. When the group identifier for the group is referenced in a destination field of the service rule, the service rule is enforced on data messages that are directed to machines in that group. In addition to group identifiers, the service rule is also enforced in some embodiments on data message flows with other attributes matching those specified in the rule. For example, the rule may specify a flow 5-tuple (source IP address and port number, destination IP address and port number, and transmission protocol). Alternatively or conjunctively, the rule may specify other attributes of the flow or a referenced group identifier.
In some embodiments, the controllers 1420-1425 provide the service rules to identified hosts executing PFEs, and the PFEs store the service rules as match-action tables.
The match-action table 2300 has multiple flow entries 2305-2315 each specifying different service rules. The number of entries shown is simplified for illustrative purposes only and in practice there are hundreds, even thousands of such entries. These entries refer to the security group definitions introduced earlier with reference to
The second entry 2310 indicates that matching data messages originating from group A (OS1 machines in Paris) and destined for group E (OS1 machines globally) should be allowed. This rule indicates that OS1 machines at one site in a domain are allowed to send data messages to machines at other sites in the domain.
The third entry 2315 indicates that matching data messages originating from group B (OS1 machines in New York) and destined for group A (OS1 machines in Paris) should be dropped. This rule indicates that OS1 machines in one region are not allowed to send data messages to OS2 machines in another region. In this example, the rule specified by this entry 2315 is one possible rule for network policy 1773, which as illustrated in
The PFEs identify matching flows by performing match classification operations on each data message, to determine if the flow attributes and/or group identifier match those specified in the service rule. When a data message matches the service rule, it is provided to a service machine in some embodiments, which performs the action that is specified by the rule on the matching data message. These actions include dropping the data message, allowing the data message, or performing other service operations on the data message. For example, a service operation may modify the header of the data message, to redirect the message to a different destination. These service operations include load-balancing, firewall, Dynamic Host Configuration Protocol (DHCP), Network Address Translation (NAT), and other services.
In some embodiments, policies are defined as nodes in the policy tree, and defined as child nodes of domain nodes. In such cases, the policy (and therefore, any service rule of the policy) inherits the span of its parent domain, i.e. the policy is enforced at the sites which are members of its parent domain. Furthermore, groups are also represented in some embodiments as child nodes of a domain, and the group inherits the span of its parent domain. For a policy to be applicable to members of a group, the span of the group must in some embodiments include the span of the policy. This is so the controllers are able to recognize the identifiers referenced by the service rules, and identify the members of those groups to enforce the service rules.
In some embodiments, a policy's service rules distributed to a first set of sites (e.g., a first domain) for applying to a first group of machines defined at those sites make reference to a second group of machines defined at a second, different set of sites (e.g., a second domain). In other words, the span of the referenced second group does not include the span of the policy. Since such service rules reference a group of machines that is undefined for one or more of the sites in the first set of sites (e.g., the first domain), not all the machines in the first group are able to enforce the policy. Therefore, some embodiments identify which sites in the first set of sites (e.g., the first domain) do not have the definition of the referenced group, and distribute the definition of the referenced group to those identified sites.
Two groups are defined beneath these regions, group A 1817 (OS1 machines in Paris) under domain A 1770 (Europe), and group B 1819 (OS2 machines in New York) under domain B 1775 (North America). Both domains A 1770 and B 1775 and groups A 1817 and B 1819 are defined at the global manager 420. Group A 1817 is defined at the local manager 425 in Paris, and group B 1819 is defined at the local manager 435 in New York. However, both domains 1770-1775 are defined at both sites, since domains are in some embodiments defined at all sites. The corresponding site nodes under each domain are omitted from the trees in
The process 2400 begins at 2405 by receiving a network policy defined in a domain (e.g., which spans a first set of sites) for enforcement on members of a security group in that domain. The network policy is received at the global manager 420 through a user client 440 in some embodiments (e.g., from an administrator of the network). For example, in
At 2410, the process 2400 identifies a reference in the network policy to another security group in another domain. For example, in
The process 2400 determines at 2415 whether the span of the other group includes the span of the policy. If the group span includes the policy span, then the process 2400 ends. If the group span does not include the policy span, as in the example of
The process 2400 extends at 2420 the span of the referenced group include the span of the policy. In effect, this distributes the definition of the group in the other domain to the sites in the original domain where the policy is defined. This is done in some embodiments by performing a new span calculation for the policy, based on separate span calculations for the groups referenced by the policy as well as the policy's parent domain. The span of the referenced group is then extended by sending an update event through a dedicated AR channel from the global manager 420 to the local manager 425, as described above with reference to
As noted above, in some embodiments span is also determined according to span leaders for some nodes of the global policy tree 1700. Security groups are one type of node in some embodiments whose span leader is not their containing domain. For example, some groups serve as span leaders for other groups. In the case of cross-domain policies, the span of the referenced group is extended by specifying the span leader of the referenced group to be the first group in the same domain as the policy. In this manner, the span for the referenced group is broader than just its parent domain.
Upon receiving the update event, the local manager 425 in some embodiments creates a reference group underneath the global policy tree in Paris. For example, in
Some embodiments extend the group definition using process 2400 for all local managers at all sites (e.g., London) in domain A (Europe). For example, if the group definition changes, i.e. a VM migrates from one site in the domain to another site in the domain, then the destination site will also need the reference group in its local manager's global tree.
In some embodiments, the policy tree at the local manager 435 in New York is not updated or modified in any way as a result of extending the span of the group 1819 at the global manager 420 and adding the reference group 2550 to the local manager 425. The administrator of the New York site is not aware in some embodiments that the group 1819 at the site is being referenced by a policy 1773 at a different domain 1770 and could even decide to delete the group. However, due to the reference group 2550 created at the referring domain 1770 the global manager 420 is aware of the reference. The global manager 420 then prevents the deletion of the group 1819 and in some embodiments displays an alarm to the administrators of both the referring site and the referred site, so that the conflict can be resolved.
After the local managers in the original domain receive the reference group, they generate configuration data and distribute to their respective site controllers, to configure the physical forwarding elements and service machines at the first set of sites to enforce the policy.
In some embodiments, the configuration of the logical network can be described in terms of several different types of state. The desired state represents the user intent, e.g. the state of the network that the user (e.g., an administrator) desires to implement, through the global manager 420. The realized state is the actual implementation of the desired state, which includes configuration of the physical forwarding elements in the data plane. The realized state only pertains to logical network elements created by the user, and includes configuration details that the network management and control system automatically determines on behalf of the user.
The third type of state is the discovered state, which represents configuration details of logical network elements that are not specified as part of the user intent. For example, one category of discovered state in some embodiments is site-specific information, such as the available physical resources at a physical site. In some embodiments, the desired state is dependent on the discovered state, or is informed by the discovered state. As the desired state is realized, some parts of the discovered state are also modified in some embodiments, and these modifications are then discovered again, informing the next iteration of desired state and realized state. The discovered state automatically discovered is in some embodiments by the global manager 420 based on information from the physical sites, communicated up from those sites' control planes to their local managers, and from the local managers to the global manager.
For example, the structure of the global policy tree 1700 beneath each site node is automatically discovered in some embodiments, from the controller cluster 1420 at the physical site, up to the local manager 430, and then to the global manager 420. Once the global manager 420 is configured with the access credentials of the local manager 430, it communicates with the local manager (e.g., through a REST API in some embodiments) to automatically populate the enforcement point node and identify which transport zones, edge clusters, and edge nodes are configured and available. In some embodiments, these were previously configured by the physical site's local manager. Corresponding child nodes are accordingly created for each.
In some embodiments, the creation of logical ports and segment ports is not allowed from the global manager 420. Such ports are site-specific in such embodiments, with unique identifiers that are assigned by the management/control plane at their home physical site, and the span of these ports cannot be stretched. Only one logical network endpoint can be attached to a port and a logical network endpoint can only be part of one site at a given point in time. When the span of a switch/segment gets stretched to multiple sites, active logical ports (e.g., with workloads attached) are automatically discovered as child nodes of the logical switch or segment in that tree (e.g., as interface nodes).
The desired configuration of the logical network represents the intentions of the user (e.g., the administrator). The user specifies their intent by specifying the desired configuration, which is why the desired configuration is also referred to as user intent. The global manager 420 is an intent-based policy manager that receives user intent (internally represented as the global policy tree 1700) and communicates that intent to the local managers at each site. The local managers then interpret the received user intent to generate configuration data, and provide the configuration data to the network managers and controllers as described above to implement the desired configuration. In some embodiments, this is referred to as realization of the user intent (e.g., in the control plane and the data plane). The actual implementation details (e.g., configuration parameters of the forwarding elements, IP addresses, internal UUID identifiers for ports, etc.) are not known or dictated by the user or by the global manager when specifying the intent, but are necessary details for realization of that intent. These details include data from the management plane and the control plane, e.g. site managers and controllers.
In a best-case idealized scenario, the realized state of the logical network would be identical at every physical site. However, since each physical site is independently configurable (e.g., via the local managers 425-435), and since certain configuration details are site-specific (e.g., unique identifiers for system-created objects in the management and control planes, such as logical port identifiers), in practice the realized state will be different for every physical site, even for the same logical network element. Even for a single site, the amount of realization data is vastly larger than the amount of data required to specify user intent, and this is then multiplied by the number of sites in the federated logical network. Accordingly, in some embodiments the global manager does not automatically maintain information on realization status (also referred to as the “realized state”) for any given logical network element. Instead, the realized state may be queried from the global manager on an on-demand basis, for troubleshooting or other investigatory purposes.
The realized state of a logical network element indicates in some embodiments whether the desired configuration (also referred to as the desired state) of the logical network element has been realized at each of the sites that the element spans. Some embodiments describe the realized state partly in terms of a deviation from the desired state. In some embodiments, the realized state for a logical network element (e.g., a logical router, logical switch, etc.) describes how the logical network element is implemented at each of the physical sites that it spans. For example, the realized state includes data from PFEs executing on computing devices at each of the spanned sites to implement the logical network element.
For each logical network element, the realized state received in some embodiments from each local manager is a subset of the total realized state of that logical network element, just as the desired state for the logical network element that is provided to the local manager is a subset of the global desired state of that logical network element defined at the global manager.
In some embodiments, the realized states received from each local manager 425-435 are status messages that indicate whether the logical network element was successfully realized at the corresponding physical site. For example, if the logical network element has not been successfully realized at all physical sites, the status message may simply be a flag that indicates error or a pending realized state.
The process 2600 begins at 2605 by receiving an inquiry for the realized state of a logical network element (e.g., a logical router, a logical switch, services such as BGP or VPN, security policies, etc.). In some embodiments, the global manager receives the inquiry from a user client 440 (e.g., over a wide area network 442 such as the Internet), and is initiated in some embodiments by an administrator of the logical network. The inquiry is received and processed in some embodiments as one or more REST API calls from the user client 440 to an API processor module 706.
In some embodiments, the inquiry includes a reference to the logical network element, e.g. a path identifier that uniquely identifies the element. For example, an inquiry for the realization status of the logical switch 1745 includes a path identifier such as /global-infra/router_T1A/ which describes where the node corresponding to the logical switch is positioned in the global policy tree 1700.
In some embodiments, the inquiry is for the realized state of the logical network element at a single physical site, instead of at all sites spanned by the element. In such embodiments, the inquiry includes an identifier for the specified site. The process 2600 determines at 2607 whether the inquiry includes a site identifier. If the process 2600 determines that the inquiry includes a site identifier, then the process provides the site identifier to a realization status processor 2705 and continues to 2615, which is described below.
If the process 2600 determines at 2607 that the inquiry does not include a site identifier, the process 2600 identifies at 2610 the physical sites spanned by the logical network element. Identifying the spanned sites is achieved in some embodiments by performing a span calculation as described above with reference to
The process 2600 sends at 2615 a request for the realized state of the logical network element to each local manager 425-435 at each identified site. In some embodiments, the request is sent from the realization status processor 2705 that receives one or more site identifiers from the broker service 735 or the API processor 706. Some embodiments send the requests in parallel (e.g., “scatter” the requests) to the local managers 425-435.
In some embodiments, the requests are made through an API of the local managers 425-435. In such embodiments, the realization status processor 2705 employs multiple realization API clients 2710 to perform the API calls, with one client assigned to each local manager. In some embodiments, the requests are transmitted through the same AR channels 725-727 as are used to provide the desired configuration data to each local manager.
The process 2600 receives at 2615 realized state from each of the local managers in response to the requests (e.g., “gathers” the realized state). After receiving the realized state from the local managers at each of the sites spanned by each of the logical network elements, the process 2600 combines at 2620 the received realized state from each site. In some embodiments, the received realized states are received by the realization API clients 2710 and combined by the realization status processor 2705 in a single realization state for the logical network element.
At 2625, the process 2600 processes the single combined realization state and presents the realized state in response to the initial request. For example, the realized state may be presented in a report on a user client 440 for an administrator of the logical network in some embodiments. The process 2600 then ends.
The side panel 2805 of
The main panel 2810 provides various configuration information sections for this router, including interfaces, routing, and Border Gateway Protocol (BGP) settings. In some embodiments, one or more of these sections are not configurable via the global manager as desired state, but are instead discovered state based on configuration at the physical sites.
The main panel 2810 also displays in some embodiments information related to the desired state of the logical network element, such as the element's span. In the example, the Locations section 2815 shows a list of all locations that the Tier0Gateway1 router currently spans (e.g., London, Paris, and New York). Other available desired configuration information includes Domain Name System (DNS) information, Network Address Translation (NAT) settings, and firewall configuration. Some or all of these settings may be modified by the administrator to define the desired state of the logical network.
The realization status of the Tier0Gateway1 router is visually represented in some embodiments as a status indicator 2820. In different embodiments, the status indicator may display different text, symbols, colors, or some combination of these. This status indicator 2820 represents in some embodiments a global status indicating the global realization status at all sites in the element's span.
The global status indicator 2820 is in some embodiments an on-demand realization status. In some such embodiments, selecting the logical network element from the interface is the action that initiates the realization status request. In the example of
If the router has been successfully realized at all physical sites, as illustrated in
In some embodiments, the global status indicator 2820 is also a user interface element. Interaction with the global indicator, i.e. clicking, mouseover, or touching (if the interface is on a touchscreen) the global status indicator, creates a pop-up window in such embodiments, as in the example of
The side panel 2905 of
The main panel 2910 provides various configuration information sections for this segment, including segment profiles and DHCP static bindings. In some embodiments, one or more of these sections are not configurable via the global manager as desired state, but are instead discovered state based on configuration at the physical sites. Other available information includes statistics and related groups defined for the segment.
The main panel also displays in some embodiments information related to the desired state of the logical network element, such as the element's span. In the example, the Locations section 2920 shows a list of all locations that the LS-103 switch currently spans (e.g., London, Paris, and New York). Other available desired configuration information 2925 includes VLAN-specific settings such as domain name and address bindings, which are not applicable for this overlay network but which are populated for VLAN segments. Some or all of these settings may be modified by the administrator to define the desired state of the logical network.
The realization status of the LS-103 switch is visually represented in some embodiments as a status indicator 2930. In different embodiments, the status indicator may display different text, symbols, colors, or some combination of these. This status indicator 2930 represents in some embodiments a global status indicating the global realization status at all sites in the element's span. For example, if the switch has been successfully realized at all physical sites, the global status indicator 2820 may display a colored icon (e.g., green) and/or the word “Success.” If the switch is not successfully realized at one or more of the physical sites in the span, then the global status indicator 2820 will display a different color (e.g., yellow or red) and/or different text such as “Pending” or “Error.” In some embodiments, if the global status is not “Success” then there may be one or more alarms. The number of alarms is also displayed alongside the global status indicator in some embodiments, though in this example the number of alarms is zero since the switch has been successfully realized.
The global status indicator 2930 is in some embodiments an on-demand realization status. In some embodiments, selecting the logical network element from the interface is the action that initiates the realization status request. In the example of
In some embodiments, the global status indicator 2930 is also a user interface element. Interaction with the global indicator, i.e. clicking, mouseover, or touching (if the interface is on a touchscreen) the global status indicator, creates a pop-up window in such embodiments, as in the example of
In this example, each of the local managers at each spanned site each has reported “Success” to the global manager (as indicated by the status indicators 2935 next to each of the listed sites), in response to the realization status inquiry from the global manager. If the switch is not successfully realized at one or more of the physical sites in the span, then the local status indicator 2825 for that site will display a different color (e.g., yellow or red) and/or different text such as “Pending” or “Error.” The status may also be refreshed by selecting a refresh icon in some embodiments, either for an individual physical site or for a global refresh, which initiates a new request for realization status at the specified site or sites. If refresh is selected for a specific site, then the request includes a site identifier in some embodiments, which is used as described above with reference to operation 2607 in process 2600.
In some embodiments, the global manager also receives notifications from the local managers. Unlike requests for realized state which are provided on-demand (e.g., pulled from the local managers), these notifications are automatically pushed from the local managers. The notifications pertain in some embodiments to site-specific implementation problems or situations. Notifications are provided in some embodiments to the GM as the events they describe occur, e.g. in real-time or near-real-time rather than on-demand. These notifications are also displayed in some embodiments as a report, e.g. for review by an administrator of the logical network.
One type of notification in some embodiments is an alarm from a local manager that the implementation of a logical network element (e.g., a create event) has failed at the specific site. For example, the failure could be due to an IP address conflict with another network element defined locally at the local manager, of which the global manager was unaware.
In the first stage (upper left left) of
In the second stage (upper right) of
The local manager 430 receives the instruction and attempts to fulfill the create command by creating router T1-G under the Tier-0 router. In some embodiments, the local manager attempts to create the router using process 1500, which was described above. In this example, however, the IP address range of router T1-G 3020 at least partially overlaps with the IP address range of the locally-created router T1-B 3010. Therefore, the local manager fails to create the requested router, since the requested IP range is invalid.
In the third stage (bottom) of
In some embodiments, an update command from the global manager 420 of an existing logical network element also may trigger a failure alarm. For example, the local managers 425-435 have in some embodiments emergency security policies that can override any other configuration in certain exceptional situations (e.g., a security exploit, a physical emergency or data loss, etc.). Emergency policies are not supported or recognized by the global manager in such embodiments, so any attempt to specify a change in policy that affects or contradicts the emergency policies at the local manager will also trigger an alarm in the same manner as described above with reference to
Another type of notification in some embodiments is an alarm from the local manager that a local administrator of the network has attempted to override the desired state of a logical network element at that particular site.
In the first stage (upper left left) of
In the second stage (upper right) of
After receiving the update command, the local manager 430 in this example fails to implement the requested update to the Tier-0 router 3110. One example for why the update fails is because some embodiments restrict the types of configurations that can be overridden at the local manager 430. For example, the local manager 430 is only permitted to modify networking-related configuration in some embodiments. If the update is a security policy-related update, then the local manager is not permitted to override the global manager 420. Another example is when the update is to extend the router's span to a new site (i.e., stretching the span of the router) prior to stretching the security policies that govern the router. Security policies must be stretched before stretching logical network elements in some embodiments, to avoid creating a security loophole after the span is extended.
In the third stage (bottom) of
The local manager 430 sends an alarm 3125 to the global manager 420 in some embodiments, to notify it that an attempt was made to override the configuration of the Tier-0 router 3110, and that the shadow copy 3120 is in use instead. This is necessary because without this information, the global manager 420 would assume that the original copy of the Tier-0 router 3110 is in use, and any user intent based on that assumption would potentially fail or be inconsistent.
In embodiments where the attempt to modify the configuration of a logical network element succeeds (e.g., because the update is a networking-related update, not a security-related update), then the local manager in some embodiments sends a notification (not shown in
A third type of notification in some embodiments is a message from the local manager that the deletion of a logical network element has failed at the site.
In the first stage of the example (upper portion of
In the example, at site A 405 the local manager 425 also has a local policy tree with local root 3230 which was defined locally (e.g., by a local administrator, who may be different from a global administrator defining the global policy tree at the global manager 420). Beneath the local root 3230, a local Tier 1 router 3235 is also defined. This local Tier-1 router has a reference to the globally-defined Tier-0 router 3210 from the global manager (depicted as a dashed arrow from Tier-1 router 3235 to Tier-0 router 3210). The global manager 420 is completely unaware of the existence of Tier-1 router 3235 in some embodiments.
At site B 410, the local manager 430 also has a local policy tree with local root 3240 which was defined locally (e.g., by a local administrator, who may be different from a global administrator defining the global policy tree at the global manager 420). Beneath the local root 3240, a local Tier 1 router 3245 is also defined. This local Tier-1 router 3245 is a completely different router than the local Tier-1 router 3235 defined beneath the local root 3230 at the local manager 425 for site A 405. The local Tier-1 router 3245 does not have any reference to the globally-defined Tier-0 router 3210 from the global manager.
In the second stage of the example (bottom of
In the third stage of the example (top of
In the fourth stage of the example (bottom of
The local manager 430 at site B 410 sends a notification 3265 to the global manager to inform that the Tier-0 router 3215 was successfully deleted from the local root 3225. The global manager 420 receives the notification 3265, but since there is an alarm 3255 instead of a corresponding success notification from local manager 425, in some embodiments the Tier-0 router 3205 is not deleted from the global root 1702.
An additional example of a type of notification in some embodiments is a message from the local manager that a logical network endpoint (e.g., a virtual machine) has been migrated from one site to another site. Such endpoints are typically attached to a logical port of a logical network element (e.g., a logical switch) that spans both sites. When the endpoint is migrated to a different site, it retains its association with that same logical port in some embodiments, and seamlessly inherits the same security policies that are associated with that port, even at the new site.
In some embodiments, a machine may be migrated from one physical site to another physical site, while preserving the state of network policies that apply to the machine. For example, for a logical segment (logical switch) that spans at least two physical sites, a machine attached to a logical port of the segment at one of the spanned sites is migrated in some embodiments to a new logical port of the same segment at another one of the spanned sites. Migrating a machine in some embodiments comprises creating a new machine at a new host computer (at the new site), transferring configuration data to the new machine from the original machine, and removing the original machine from its host computer. In some embodiments, the configuration data comprises physical state data and logical network policies.
The state of the machine (e.g., runtime data) is transferred in some embodiments from a computing device executing the original machine at the source site (i.e., the source host) to a computing device executing the new machine at the destination site (i.e., the destination host). In some embodiments, the state includes a local port identifier assigned by a local manager (e.g., local manager 425) at the source physical site, and a global port identifier assigned by the global manager 420. The global port identifier is associated with logical network policies defined through a user client 440 (e.g., by an administrator of the network) in some embodiments as part of the desired configuration of the logical network. Upon receiving the transferred state data, the destination host overwrites at least a portion of the state of the new machine with the received state of the old machine.
In some embodiments, the destination host extracts the global port identifier from the received state data and provides it to the destination site's local manager (e.g., local manager 430). The destination local manager 430 uses the global port identifier in some embodiments to retrieve logical network policies associated with the global port identifier from the global manager 420, and apply these policies to the new machine. In some embodiments, the local manager 430 generates configuration data from the policies and provides the configuration data to the control plane of the logical network at the destination site (e.g., a set of controllers 1420-1425) which then distributes the configuration data to the destination host.
The initial stage of
The second stage of
In this example, the global manager is not aware of the local port identifiers LP-A or LP-B. In some embodiments, each of the local managers 425-430 maintains a mapping of the global port identifier and the local port identifier for the logical port at the respective site. This mapping allows the local managers to retrieve from the global manager 420 the relevant policy information applicable to the machine, so that these policies are seamlessly applied before and after migration.
In the example of
The process 3500 begins at 3505 by receiving an instruction from a compute manager 3405 to register a new VM, for attachment to a logical segment 3330 that spans the source site. The register command is illustrated in
At 3510, the process 3500 requests a port attachment for the new VM, from the local manager 425 at the source site. The port attachment request is illustrated in
At 3515, the process 3500 powers on the requested VM, as indicated by event 3427. The process 3500 attaches the new VM to the port 3336 of the logical segment 3330, using the local port identifier (e.g., LP-A) received from the local manager 425. In some embodiments, the process 3500 sends a success message to the local manager 425 after attaching the port 3336. The success message is illustrated in
Upon receiving the success message, in some embodiments the local manager 425 provides the local port identifier (as illustrated in
At 3520, the process 3500 receives the global port identifier from the local manager 425, as illustrated in
In some embodiments, a set of tags are also specified at the global manager 420 for attaching to the newly attached VM. These tags are used to define group membership for the VM in some embodiments. Security rules may also be defined dynamically using these tags. In some embodiments, these dynamic references to tags may include regular expressions. Thus, for example, a set of tags dev_paris could be defined to indicate a VM is in development mode, and security rules defined just for the Paris site by referencing the full tag, or security rules could be defined just using a portion of the tag (e.g., “dev_*”) to apply to all VMs tagged as in development mode regardless of location. The tags are directly applied to the VM in some embodiments, and applied by the source host 3415 internally on the logical port.
In some embodiments, the global manager 420 receives configuration data (e.g., desired state, such as security policies) to apply to the logical port. The desired state is received in some embodiments through a user interface 440 (e.g., from an administrator of the logical network). The desired state is then provided from the global manager 420 to the local manager 425, as described above with reference to
At 3525, the process 3500 receives and applies the desired configuration (e.g., security policies, of which some may reference the tags) to the logical port 3336, as illustrated in
At 3530, the process 3500 receives an instruction from the compute manager 3405 to prepare the VM for migration. The migration command is illustrated in
At 3535, the process 3500 saves the realized state data for the VM, as indicated by event 3445. In some embodiments, the saved state data includes the global port identifier, which was received as described above at 3520. The process 3500 sends the saved data to the destination host 3410, as described below with reference to process 3600 of
At 3540, the process 3500 receives an instruction from the compute manager 3405 to de-register the VM. The de-registration command is illustrated in
At 3545, the process 3500 requests the local manager 425 to detach the logical port 3336, as illustrated in
The process 3600 begins at 3605 by receiving an instruction from the compute manager 3405 to register a new VM, for attachment to the logical segment 3330 that spans the source site and the destination site. The register command is illustrated in
At 3610, the process 3600 requests a port attachment for the new VM, from the local manager 430 at the destination site. The port attachment request is illustrated in
At 3615, the process 3600 powers on the requested VM, as indicated in
At 3620, the process 3600 receives saved state data from the source host 3415. The saved state data includes the global port identifier (e.g., LP-1) in some embodiments, which was saved by the original VM on the source host 3415, as described above with reference to process 3500 of
In some embodiments, the transfer 3458 of state data from the source host 3415 is a lengthy process, including copying disks, memory, and other data. This process takes several hours in some embodiments, and the source VM is still active on the source host 3415 during this time. Packets received by the source host are also included in the state transfer, so even though there are no policies applied to the destination host during the state transfer, it does not matter, since the traffic is being correctly processed by the source VM. The destination VM does not receive any direct data traffic during the state transfer. Therefore, in some embodiments, the pending flag is used to ensure that the destination host does not send a success message to the local manager until the state transfer is complete. Otherwise, the local manager would report the success to the global manager and traffic would begin to arrive prematurely at the destination VM. The pending state is tracked in some embodiments by both the destination host 3410 and the destination local manager 430.
At 3625, the process 3600 overwrites the configuration of the logical port 3337 with the received state data, as indicated in
In some embodiments, when the destination local manager 430 identifies that the full replication is complete (memory, disks, etc), it performs a fast suspend and resume (FSR) operation at the destination VM. This is coordinated with the source local manager 425 in some embodiments (e.g., via the compute manager 3405), to also perform the same FSR operation at the source VM. The FSR operation flips active status of the source VM off and enables active status for the destination VM, so that the destination VM may begin receiving traffic. The pending state of the destination port is also removed in some embodiments at this stage.
At 3630, the process 3600 sends a success message to the local manager 430 after attaching the port. In some embodiments, the process 3600 also reports the global port identifier (e.g., LP-1), and also any associated tags, to the local manager 430. The provision of the success message, global port identifier, and tags to the local manager 430 is illustrated in
Upon receiving the global port identifier, in some embodiments the local manager 430 updates the segment 3330 to associate the global port identifier (e.g., LP-1) with the local port identifier (e.g., LP-B) for port 3337. The local manager 430 then uses the global port identifier and the tags to request the desired state for the port 3337, from the global manager 420, as illustrated in
At 3635, the process 3600 receives and applies the desired state to the logical port 3337, as illustrated in
Many of the above-described features and applications are implemented as software processes that are specified as a set of instructions recorded on a computer readable storage medium (also referred to as computer readable medium). When these instructions are executed by one or more processing unit(s) (e.g., one or more processors, cores of processors, or other processing units), they cause the processing unit(s) to perform the actions indicated in the instructions. Examples of computer readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc. The computer readable media does not include carrier waves and electronic signals passing wirelessly or over wired connections.
In this specification, the term “software” is meant to include firmware residing in read-only memory or applications stored in magnetic storage, which can be read into memory for processing by a processor. Also, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while remaining distinct software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that together implement a software invention described here is within the scope of the invention. In some embodiments, the software programs, when installed to operate on one or more electronic systems, define one or more specific machine implementations that execute and perform the operations of the software programs.
The bus 3705 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the electronic system 3700. For instance, the bus 3705 communicatively connects the processing unit(s) 3710 with the read-only memory 3730, the system memory 3725, and the permanent storage device 3735.
From these various memory units, the processing unit(s) 3710 retrieve instructions to execute and data to process in order to execute the processes of the invention. The processing unit(s) may be a single processor or a multi-core processor in different embodiments.
The read-only-memory (ROM) 3730 stores static data and instructions that are needed by the processing unit(s) 3710 and other modules of the electronic system. The permanent storage device 3735, on the other hand, is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when the electronic system 3700 is off. Some embodiments of the invention use a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) as the permanent storage device 3735.
Other embodiments use a removable storage device (such as a floppy disk, flash drive, etc.) as the permanent storage device. Like the permanent storage device 3735, the system memory 3725 is a read-and-write memory device. However, unlike storage device 3735, the system memory is a volatile read-and-write memory, such a random-access memory. The system memory stores some of the instructions and data that the processor needs at runtime. In some embodiments, the invention's processes are stored in the system memory 3725, the permanent storage device 3735, and/or the read-only memory 3730. From these various memory units, the processing unit(s) 3710 retrieve instructions to execute and data to process in order to execute the processes of some embodiments.
The bus 3705 also connects to the input and output devices 3740 and 3745. The input devices enable the user to communicate information and select commands to the electronic system. The input devices 3740 include alphanumeric keyboards and pointing devices (also called “cursor control devices”). The output devices 3745 display images generated by the electronic system. The output devices include printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD). Some embodiments include devices such as a touchscreen that function as both input and output devices.
Finally, as shown in
Some embodiments include electronic components, such as microprocessors, storage and memory that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a variety of recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and/or solid state hard drives, read-only and recordable Blu-Ray® discs, ultra-density optical discs, any other optical or magnetic media, and floppy disks. The computer-readable media may store a computer program that is executable by at least one processing unit and includes sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as is produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter.
While the above discussion primarily refers to microprocessor or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions that are stored on the circuit itself.
As used in this specification, the terms “computer”, “server”, “processor”, and “memory” all refer to electronic or other technological devices. These terms exclude people or groups of people. For the purposes of the specification, the terms display or displaying means displaying on an electronic device. As used in this specification, the terms “computer readable medium,” “computer readable media,” and “machine readable medium” are entirely restricted to tangible, physical objects that store information in a form that is readable by a computer. These terms exclude any wireless signals, wired download signals, and any other ephemeral signals.
This specification refers throughout to computational and network environments that include virtual machines (VMs). However, virtual machines are merely one example of data compute nodes (DCNs) or data compute end nodes, also referred to as addressable nodes. DCNs may include non-virtualized physical hosts, virtual machines, containers that run on top of a host operating system without the need for a hypervisor or separate operating system, and hypervisor kernel network interface modules.
VMs, in some embodiments, operate with their own guest operating systems on a host using resources of the host virtualized by virtualization software (e.g., a hypervisor, virtual machine monitor, etc.). The tenant (i.e., the owner of the VM) can choose which applications to operate on top of the guest operating system. Some containers, on the other hand, are constructs that run on top of a host operating system without the need for a hypervisor or separate guest operating system. In some embodiments, the host operating system uses name spaces to isolate the containers from each other and therefore provides operating-system level segregation of the different groups of applications that operate within different containers. This segregation is akin to the VM segregation that is offered in hypervisor-virtualized environments that virtualize system hardware, and thus can be viewed as a form of virtualization that isolates different groups of applications that operate in different containers. Such containers are more lightweight than VMs.
Hypervisor kernel network interface modules, in some embodiments, is a non-VM DCN that includes a network stack with a hypervisor kernel network interface and receive/transmit threads. One example of a hypervisor kernel network interface module is the vmknic module that is part of the ESXi™ hypervisor of VMware, Inc.
It should be understood that while the specification refers to VMs, the examples given could be any type of DCNs, including physical hosts, VMs, non-VM containers, and hypervisor kernel network interface modules. In fact, the example networks could include combinations of different types of DCNs in some embodiments.
While the invention has been described with reference to numerous specific details, one of ordinary skill in the art will recognize that the invention can be embodied in other specific forms without departing from the spirit of the invention. In addition, a number of the figures (including
| Number | Date | Country | Kind |
|---|---|---|---|
| 202041015134 | Apr 2020 | IN | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 6219699 | McCloghrie et al. | Apr 2001 | B1 |
| 6347374 | Drake et al. | Feb 2002 | B1 |
| 7502884 | Shah et al. | Mar 2009 | B1 |
| 7539745 | Wang et al. | May 2009 | B1 |
| 7802000 | Huang et al. | Sep 2010 | B1 |
| 8479275 | Naseh | Jul 2013 | B1 |
| 8611351 | Gooch et al. | Dec 2013 | B2 |
| 8625616 | Vobbilisetty et al. | Jan 2014 | B2 |
| 8660129 | Brendel et al. | Feb 2014 | B1 |
| 8707417 | Liang et al. | Apr 2014 | B1 |
| 9330161 | D'Amato et al. | May 2016 | B2 |
| 9432215 | Stabile et al. | Aug 2016 | B2 |
| 9602312 | Koponen et al. | Mar 2017 | B2 |
| 9755965 | Yadav et al. | Sep 2017 | B1 |
| 9825851 | Agarwal et al. | Nov 2017 | B2 |
| 9847938 | Chanda et al. | Dec 2017 | B2 |
| 9906560 | Jain et al. | Feb 2018 | B2 |
| 9923811 | Agarwal et al. | Mar 2018 | B2 |
| 10091028 | Koponen et al. | Oct 2018 | B2 |
| 10110417 | Hankins et al. | Oct 2018 | B1 |
| 10135675 | Yu et al. | Nov 2018 | B2 |
| 10142127 | Cherian et al. | Nov 2018 | B2 |
| 10187302 | Chu et al. | Jan 2019 | B2 |
| 10243846 | Jiang et al. | Mar 2019 | B2 |
| 10243848 | Agarwal et al. | Mar 2019 | B2 |
| 10339123 | Venkatesh et al. | Jul 2019 | B2 |
| 10382529 | Wan et al. | Aug 2019 | B2 |
| 10560343 | Cartsonis et al. | Feb 2020 | B1 |
| 10601705 | Hira et al. | Mar 2020 | B2 |
| 10637800 | Wang et al. | Apr 2020 | B2 |
| 10673752 | Agarwal et al. | Jun 2020 | B2 |
| 10693833 | Mathew et al. | Jun 2020 | B2 |
| 10862753 | Hira et al. | Dec 2020 | B2 |
| 10880158 | Lambeth et al. | Dec 2020 | B2 |
| 10880170 | Wang et al. | Dec 2020 | B2 |
| 10897420 | Pianigiani et al. | Jan 2021 | B1 |
| 10999154 | Ahrenholz et al. | May 2021 | B1 |
| 11057275 | Arunachalam et al. | Jul 2021 | B1 |
| 20020029270 | Szczepanek | Mar 2002 | A1 |
| 20020093952 | Gonda | Jul 2002 | A1 |
| 20020131414 | Hadzic | Sep 2002 | A1 |
| 20030167333 | Kumar et al. | Sep 2003 | A1 |
| 20030185151 | Kurosawa et al. | Oct 2003 | A1 |
| 20030185152 | Nederveen et al. | Oct 2003 | A1 |
| 20030188114 | Lubbers et al. | Oct 2003 | A1 |
| 20030188218 | Lubbers et al. | Oct 2003 | A1 |
| 20040052257 | Abdo et al. | Mar 2004 | A1 |
| 20050190757 | Sajassi | Sep 2005 | A1 |
| 20050235352 | Staats et al. | Oct 2005 | A1 |
| 20050288040 | Charpentier et al. | Dec 2005 | A1 |
| 20060092976 | Lakshman et al. | May 2006 | A1 |
| 20060179243 | Fields et al. | Aug 2006 | A1 |
| 20060179245 | Fields et al. | Aug 2006 | A1 |
| 20060221720 | Reuter | Oct 2006 | A1 |
| 20060251120 | Arimilli et al. | Nov 2006 | A1 |
| 20060287842 | Kim | Dec 2006 | A1 |
| 20070058631 | Mortier et al. | Mar 2007 | A1 |
| 20070130295 | Rastogi et al. | Jun 2007 | A1 |
| 20070217419 | Vasseur | Sep 2007 | A1 |
| 20070239987 | Hoole et al. | Oct 2007 | A1 |
| 20080013474 | Nagarajan et al. | Jan 2008 | A1 |
| 20080049646 | Lu | Feb 2008 | A1 |
| 20080104302 | Carpio | May 2008 | A1 |
| 20080133729 | Fridman et al. | Jun 2008 | A1 |
| 20080268847 | Mukherjee et al. | Oct 2008 | A1 |
| 20080301379 | Pong | Dec 2008 | A1 |
| 20090037367 | Wein | Feb 2009 | A1 |
| 20090070337 | Romem et al. | Mar 2009 | A1 |
| 20090193297 | Williams et al. | Jul 2009 | A1 |
| 20090241192 | Thomas | Sep 2009 | A1 |
| 20090279536 | Unbehagen et al. | Nov 2009 | A1 |
| 20090279545 | Moonen | Nov 2009 | A1 |
| 20090296726 | Snively et al. | Dec 2009 | A1 |
| 20100250784 | Henry et al. | Sep 2010 | A1 |
| 20100257263 | Casado et al. | Oct 2010 | A1 |
| 20100275199 | Smith et al. | Oct 2010 | A1 |
| 20100322255 | Hao et al. | Dec 2010 | A1 |
| 20110032898 | Kazmi et al. | Feb 2011 | A1 |
| 20110047218 | Nojima et al. | Feb 2011 | A1 |
| 20110051714 | Somes | Mar 2011 | A1 |
| 20110085569 | Gnanasekaran et al. | Apr 2011 | A1 |
| 20110188509 | Kern et al. | Aug 2011 | A1 |
| 20110231602 | Woods et al. | Sep 2011 | A1 |
| 20110299413 | Chatwani et al. | Dec 2011 | A1 |
| 20120084406 | Kumbalimutt | Apr 2012 | A1 |
| 20120120964 | Koponen et al. | May 2012 | A1 |
| 20120147898 | Koponen et al. | Jun 2012 | A1 |
| 20120275328 | Iwata et al. | Nov 2012 | A1 |
| 20130024579 | Zhang et al. | Jan 2013 | A1 |
| 20130042242 | Kagan | Feb 2013 | A1 |
| 20130044636 | Koponen et al. | Feb 2013 | A1 |
| 20130044641 | Koponen et al. | Feb 2013 | A1 |
| 20130044761 | Koponen et al. | Feb 2013 | A1 |
| 20130058250 | Casado et al. | Mar 2013 | A1 |
| 20130058335 | Koponen et al. | Mar 2013 | A1 |
| 20130058350 | Fulton | Mar 2013 | A1 |
| 20130058354 | Casado et al. | Mar 2013 | A1 |
| 20130058358 | Fulton et al. | Mar 2013 | A1 |
| 20130060819 | Lambeth et al. | Mar 2013 | A1 |
| 20130074065 | McNeeney et al. | Mar 2013 | A1 |
| 20130103817 | Koponen et al. | Apr 2013 | A1 |
| 20130132533 | Padmanabhan | May 2013 | A1 |
| 20130144992 | Barabash et al. | Jun 2013 | A1 |
| 20130159637 | Forgette et al. | Jun 2013 | A1 |
| 20130212212 | Addepalli et al. | Aug 2013 | A1 |
| 20130215769 | Beheshti-Zavareh et al. | Aug 2013 | A1 |
| 20130254328 | Inoue et al. | Sep 2013 | A1 |
| 20130286833 | Torres et al. | Oct 2013 | A1 |
| 20130287026 | Davie | Oct 2013 | A1 |
| 20130301425 | Udutha et al. | Nov 2013 | A1 |
| 20130301501 | Olvera-Hernandez et al. | Nov 2013 | A1 |
| 20130308641 | Ackley | Nov 2013 | A1 |
| 20140006465 | Davis et al. | Jan 2014 | A1 |
| 20140064104 | Nataraja et al. | Mar 2014 | A1 |
| 20140075002 | Pradhan et al. | Mar 2014 | A1 |
| 20140136908 | Maggiari et al. | May 2014 | A1 |
| 20140146817 | Zhang | May 2014 | A1 |
| 20140172740 | McCormick et al. | Jun 2014 | A1 |
| 20140172939 | McSherry et al. | Jun 2014 | A1 |
| 20140208150 | Abuelsaad et al. | Jul 2014 | A1 |
| 20140211661 | Gorkemli et al. | Jul 2014 | A1 |
| 20140241356 | Zhang et al. | Aug 2014 | A1 |
| 20140250220 | Kapadia et al. | Sep 2014 | A1 |
| 20140269435 | McConnell et al. | Sep 2014 | A1 |
| 20140301391 | Krishnan et al. | Oct 2014 | A1 |
| 20140304355 | Kamath et al. | Oct 2014 | A1 |
| 20140307744 | Dunbar et al. | Oct 2014 | A1 |
| 20150009808 | Bejerano et al. | Jan 2015 | A1 |
| 20150016276 | Decusatis et al. | Jan 2015 | A1 |
| 20150085862 | Song | Mar 2015 | A1 |
| 20150100704 | Davie et al. | Apr 2015 | A1 |
| 20150103842 | Chandrashekhar et al. | Apr 2015 | A1 |
| 20150103843 | Chandrashekhar et al. | Apr 2015 | A1 |
| 20150106804 | Chandrashekhar et al. | Apr 2015 | A1 |
| 20150117216 | Anand et al. | Apr 2015 | A1 |
| 20150154330 | Yachide et al. | Jun 2015 | A1 |
| 20150195126 | Vasseur et al. | Jul 2015 | A1 |
| 20150229641 | Sun et al. | Aug 2015 | A1 |
| 20150326467 | Fullbright et al. | Nov 2015 | A1 |
| 20160094396 | Chandrashekhar | Mar 2016 | A1 |
| 20160094661 | Jain et al. | Mar 2016 | A1 |
| 20160134528 | Lin et al. | May 2016 | A1 |
| 20160173338 | Wolting | Jun 2016 | A1 |
| 20160226700 | Zhang et al. | Aug 2016 | A1 |
| 20160226762 | Zhang et al. | Aug 2016 | A1 |
| 20160226822 | Zhang et al. | Aug 2016 | A1 |
| 20160226959 | Zhang et al. | Aug 2016 | A1 |
| 20160359705 | Parandehgheibi et al. | Dec 2016 | A1 |
| 20160380815 | Agarwal et al. | Dec 2016 | A1 |
| 20160380891 | Agarwal et al. | Dec 2016 | A1 |
| 20160380925 | Agarwal et al. | Dec 2016 | A1 |
| 20160380973 | Sullenberger et al. | Dec 2016 | A1 |
| 20170034052 | Chanda et al. | Feb 2017 | A1 |
| 20170048130 | Goliya | Feb 2017 | A1 |
| 20170063633 | Goliya et al. | Mar 2017 | A1 |
| 20170063794 | Jain | Mar 2017 | A1 |
| 20170093636 | Chanda et al. | Mar 2017 | A1 |
| 20170104720 | Bansal | Apr 2017 | A1 |
| 20170163442 | Shen et al. | Jun 2017 | A1 |
| 20170163532 | Tubaltsev | Jun 2017 | A1 |
| 20170222873 | Lee et al. | Aug 2017 | A1 |
| 20170250912 | Chu et al. | Aug 2017 | A1 |
| 20170288981 | Hong et al. | Oct 2017 | A1 |
| 20170289033 | Singh et al. | Oct 2017 | A1 |
| 20180062881 | Chandrashekhar et al. | Mar 2018 | A1 |
| 20180063036 | Chandrashekhar et al. | Mar 2018 | A1 |
| 20180234337 | Goliya et al. | Aug 2018 | A1 |
| 20180234459 | Kung et al. | Aug 2018 | A1 |
| 20190014040 | Yerrapureddy et al. | Jan 2019 | A1 |
| 20190158537 | Miriyala | May 2019 | A1 |
| 20190207847 | Agarwal et al. | Jul 2019 | A1 |
| 20190260610 | Dubey et al. | Aug 2019 | A1 |
| 20190334765 | Jain et al. | Oct 2019 | A1 |
| 20190363975 | Djernaes | Nov 2019 | A1 |
| 20200007392 | Goyal | Jan 2020 | A1 |
| 20200021541 | Chanda | Jan 2020 | A1 |
| 20200057669 | Hutcheson et al. | Feb 2020 | A1 |
| 20200076684 | Naveen et al. | Mar 2020 | A1 |
| 20200162325 | Desai et al. | May 2020 | A1 |
| 20200169496 | Goliya et al. | May 2020 | A1 |
| 20200195607 | Wang et al. | Jun 2020 | A1 |
| 20200257549 | Vlaznev et al. | Aug 2020 | A1 |
| 20200296035 | Agarwal et al. | Sep 2020 | A1 |
| 20200358693 | Rawlins | Nov 2020 | A1 |
| 20200409563 | Parasnis et al. | Dec 2020 | A1 |
| 20210067556 | Tahan | Mar 2021 | A1 |
| 20210117908 | Coles et al. | Apr 2021 | A1 |
| 20210168197 | Jones et al. | Jun 2021 | A1 |
| 20210194729 | Semwal et al. | Jun 2021 | A1 |
| Number | Date | Country |
|---|---|---|
| 102124456 | Jul 2011 | CN |
| 102986172 | Mar 2013 | CN |
| 103650433 | Mar 2014 | CN |
| 103890751 | Jun 2014 | CN |
| 1154601 | Nov 2001 | EP |
| 1635506 | Mar 2006 | EP |
| 1868318 | Dec 2007 | EP |
| 3016331 | May 2016 | EP |
| 3314831 | May 2018 | EP |
| 2010028364 | Mar 2010 | WO |
| 2011140028 | Nov 2011 | WO |
| 2012113444 | Aug 2012 | WO |
| 2013152716 | Oct 2013 | WO |
| 2015054671 | Apr 2015 | WO |
| 2017003881 | Jan 2017 | WO |
| 2018044341 | Mar 2018 | WO |
| Entry |
|---|
| Author Unknown, “Apache Cassandra™ 1.2 Documentation,” Jan. 13, 2013, 201 pages, DataStax. |
| Author Unknown, “OpenFlow Switch Specification, Version 1.1.0 Implemented (Wire Protocol 0x02),” Feb. 28, 2011, 56 pages, Open Networking Foundation. |
| Berde, Pankaj, et al., “ONOS Open Network Operating System An Open-Source Distributed SDN OS,” Dec. 19, 2013, 34 pages. |
| Guo, Yingya, et al., “Traffic Engineering in SDN/OSPF Hybrid Network,” The 22nd IEEE International Conference on Network Protocols (ICNP 2014), Oct. 21-24, 2014, 6 pages, IEEE, The Research Triangle, North Carolina, USA. |
| Hanna, Jeremy, “How ZooKeeper Handles Failure Scenarios,” http//.apache.org/hadoop/Zookeeper/Failurescenarios. Dec. 9, 2010, 1 page. |
| Heller, Brandon, et al., “The Controller Placement Problem,” Hot Topics in Software Defined Networks, Aug. 13, 2012, 6 pages, Helsinki, Finland. |
| Jin, Xin, et al., “Dynamic Scheduling of Network Updates,” SIGCOMM'14, Aug. 17-22, 2014, 12 pages, ACM, Chicago, IL, USA. |
| Krishnaswamy, Umesh, et al., “ONOS Open Network Operating System—An Experimental Open-Source Distributed SDN OS,” Apr. 16, 2013, 24 pages. |
| Lebresne, Sylvain, “[Release] Apache Cassandra 1.2 released,” Jan. 2, 2013, 1 page. |
| Mahalingham, Mallik, et al., “VXLAN: A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks,” draft-mahalingham-dutt-dcops-vxlan-00.txt Internet Draft, Aug. 26, 2011, 20 pages, Internet Engineering Task Force. |
| Mahalingham, Mallik, et al., “VXLAN: A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks,” draft-mahalingham-dutt-dcops-vxlan-02.txt Internet Draft, Aug. 22, 2012, 20 pages, Internet Engineering Task Force. |
| Non-Published Commonly Owned U.S. Appl. No. 16/888,851, filed Jun. 1, 2020, 72 pages, Nicira, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,889, filed Jun. 19, 2020, 125 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,891, filed Jun. 19, 2020, 125 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,893, filed Jun. 19, 2020, 126 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,901, filed Jun. 19, 2020, 125 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,902, filed Jun. 19, 2020, 126 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,905, filed Jun. 19, 2020, 126 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,908, filed Jun. 19, 2020, 125 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,913, filed Jun. 19, 2020, 124 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,921, filed Jun. 19, 2020, 43 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,925, filed Jun. 19, 2020, 111 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,929, filed Jun. 19, 2020, 113 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,934, filed Jun. 19, 2020, 112 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,935, filed Jun. 19, 2020, 114 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,942, filed Jun. 19, 2020, 127 pages, VMware, Inc. |
| Non-Published Commonly Owned Related U.S. Appl. No. 16/906,944 with similar specification, filed Jun. 19, 2020, 128 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,955, filed Jun. 19, 2020, 127 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,960, filed Jun. 19, 2020, 128 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,964, filed Jun. 19, 2020, 126 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,966, filed Jun. 19, 2020, 128 pages, VMware, Inc. |
| Number | Date | Country | |
|---|---|---|---|
| 20210314226 A1 | Oct 2021 | US |
