Patent Grant
11343283
As more networks move to the cloud (public, private, or hybrid), it is common for application developers within a corporation or other entity to want to deploy applications that may span these sites. This presents various challenges for the network administrators (e.g., the IT team for a corporation). In current systems, an application developer either has to be made aware of the security and networking configurations in order to deploy the application themselves, or work with the network admins to have the application deployed within the network. Various solutions are required to solve these issues.
Some embodiments of the invention provide a network management system for managing a virtualization infrastructure having one or more logical networks defined across one or more sites (e.g., datacenters). The network management system allows for a top-level user of the virtual infrastructure (e.g., a cloud provider, an enterprise's network administrators) to define a provider logical network and to define one or more second-level users (e.g., tenants, enterprise organizational units) of the virtualization infrastructure. These second-level users can in turn define accounts for different users. Specifically, a second-level user can define security and networking configuration (e.g., logical networking constructs, security zones) that is opaque to third-level users (e.g., application developers) as well as other second-level users. Application developers can specify their application requirements, without any knowledge of the security and networking configuration, and the network management system of some embodiments automatically deploys the application according to the requirements while ensuring that the security and networking policies are adhered to.
As mentioned, the top-level user defines a provider logical network (this logical network will be referred to herein as a provider logical network irrespective of whether the top-level user is a cloud provider, enterprise network admin, etc.). In some embodiments, this provider logical network is defined using a particular data model, such as a hierarchical policy tree with nodes and connections between the nodes. Through the data model, the top-level user can define security policies and security groups as well as logical network elements (e.g., logical routers and/or logical switches). In addition, in some embodiments, the provider data model includes a definition of the physical infrastructure onto which the virtualization infrastructure is deployed. This physical infrastructure of some embodiments may specify the multiple federated sites across which the virtualization infrastructure is deployed, as well as groups of physical elements within that site (e.g., groups of edge devices that physically connect to external networks in each site, zones of physical servers and/or forwarding elements within each site).
The second-level users (referred to herein as tenants), in some embodiments, are granted the ability to define their own security and networking configuration, as well as to use certain entities exposed by the provider network. Though these provider network entities may be exposed to tenant users, the tenant users are restricted from viewing or editing the configuration of the provider network (including these entities). In addition, to provide isolation between users, one tenant user is restricted from viewing (or modifying) the configuration of any other tenant users. That is, each tenant is provided with the ability to view and modify only their own configuration, and not that of any other tenants or the higher-level provider user. Within a tenant's configuration, sub-users can be created that can define their own isolated configuration (which may be linked to the tenant or provider configurations, in some embodiments). Each of these isolated configurations may be referred to as a virtual hybrid cloud (VHC); as described further below, each VHC may be isolated to a single site or spread across multiple physical sites (e.g., a combination of enterprise on-prem datacenters, branch offices, and public clouds).
In some embodiments, the data model for the tenant configuration is similar to that for the provider configuration. That is, the tenant configuration is also represented (e.g., in storage of the network management system) as a hierarchical policy tree with similar nodes and connections. One difference, however, is that in some embodiments only the provider user policy tree includes the physical infrastructure nodes defining sites and groups of physical computing devices within a site. That is, some embodiments prevent the tenant users from both viewing or configuring the provider physical infrastructure nodes and from defining their own physical infrastructure policy.
In addition, the API is similar for the tenant user and for the provider user when defining (e.g., creating, modifying) the same type of entity. For instance, the tenant administrator may define security policies and security groups as well as logical network elements. In some embodiments, tenant users (and/or sub-users of the tenant) can define a logical network including logical switches to which data compute nodes (e.g., virtual machines (VMs), containers, etc.) connect and logical routers that connect these logical switches to each other. As described further below, the data compute nodes (DCNs) may implement applications deployed in the tenant logical network in some embodiments.
In some embodiments, logical routers can be either tier-0 (T0) logical routers that provide a connection to external networks or tier-1 (T1) logical routers that (i) connect logical switches to each other, (ii) provide services (e.g., firewall, load balancing, etc.), and (iii) enable connections to T0 logical routers. Some embodiments only allow the provider user to define T0 logical routers, so that the provider user (e.g., the enterprise IT team) can oversee all of the rules regarding external connections to the virtualization infrastructure (e.g., routing protocols for route exchange with external routers, virtual private network (VPN) connections, etc.). The provider user can define multiple virtual routing and forwarding tables (VRFs) for a single T0 router in some embodiments. The tenant user, on the other hand, can only define T1 logical routers in some embodiments. In other embodiments, when a tenant user is created by the provider, an isolated VHC is automatically created with a T0 VRF specific to the tenant. The tenant user can manage this T0 VRF and define services such as routing protocols (e.g., BGP, OSPF) and VPNs, while isolating this configuration from any sub-users.
All types of users (providers, tenants, sub-users) may define security policy for their respective VHCs in some embodiments. This security policy may include, in some embodiments, security rules (e.g., distributed firewall rules and/or gateway firewall rules), security group definitions, and domains. Domains, in some embodiments, are logical groupings of one or more sites (e.g., within a similar geographic regions) that serve as an envelope to group different logical entities together (e.g., for security purposes). For example, firewall policies or other policy micro-segmentation applied to the domain will automatically be applied to all groups of logical endpoints defined within the domain in some embodiments. Security groups, in some embodiments, are groups of logical network endpoints that share one or more attributes (e.g., operating system, region, application tier, etc.). The user (provider or tenant) can apply security rules to a security group, and the network management and control system translates these security rules into rules applied to each DCN in the security group. In some embodiments, the security rules and groups are defined within domains.
While the tenant users cannot configure or view the configuration of entities defined within the provider VHC, some embodiments allow the provider user to expose some of these entities for use by the tenant users. In some embodiments, the provider user exposes these entities to one or more tenants via labels, which allow the tenant user to identify the type of entity and any specific properties of import, but not to view or modify the full configuration of the entity. For instance, a provider user might expose a T0 logical router or simply a VRF of a T0 logical router to which the tenant user (or its sub-users) can connect their T1 logical routers. The label might indicate the sites spanned by the T0 logical router, but not the VLAN segments connected to the T0 logical router or its routing protocol configuration. Other types of entities that a provider user might expose via labels include security groups that the tenant user can use to define firewall rules (with the tenant user aware of the attributes that cause a DCN to belong to the security group, but not the full membership of the group outside of the tenant VHC), edge clusters (a physical infrastructure grouping of devices for implementing centralized routing components of logical routers) that the tenant user can use for T1 logical routers, as well as services such as load balancers.
As mentioned, in addition to being provided with the ability to configure their own isolated logical network and security policy in a VHC, each tenant user can define additional sub-users, who in turn can define their own isolated logical network and security policy within a VHC. For instance, if a tenant is a line of business within an enterprise, then sub-users may be defined for different organizational units within that business line, etc. Similarly, if a tenant is a public cloud tenant, then the sub-users might be different business lines within the scope of the tenant. In addition, tenant users (or sub-users) may define accounts for application developers. In some embodiments, application developer users may not view or configure their own logical networking and security rules (i.e., they are not granted their own VHC), but rather can define applications within the scope of a particular VHC. The implementation of these applications within a VHC is described further below.
Tenant users and their sub-users can also define security zones in some embodiments. In some embodiments, security zones are defined to govern the connectivity (or other security policies) of applications. That is, DCNs implementing an application are assigned to security zones, and the connectivity and/or other security policies of the security zone are applied to those DCNs. In some embodiments, the user defining a security zone defines the span of that security zone in terms of sites (of the provider's physical infrastructure), thereby restricting applications (or tiers of applications) assigned to the security zone to those sites. Examples of common security zones include a first zone that allows connections from external endpoints (often referred to as a DMZ), as well as a second zone that does not allow connections from external endpoints but does allow incoming connections from the first zone (often referred to as a production zone, and with which back-end servers or application tiers may be associated).
As mentioned, application developer users can be provided with the ability to define an application within a VHC, without the ability to view or edit the networking and security configurations for the VHC. In some embodiments, application developers specify a set of connectivity requirements for an application. Based on these requirements, the network management and control system automatically assigns the DCNs that implement the application to the security zones defined for the VHC.
The applications are defined as sets of application tiers in some embodiments. For instance, applications are commonly specified as 3-tier applications, with a web tier of web servers that accept incoming connections from external endpoints, an application tier, and a back-end database tier. The network management and control system assigns each of these defined application tiers to one of the security zones defined for the VHC based on the requirements (e.g., connectivity requirements). For instance, a web tier that requires incoming connections from external client endpoints would be assigned to the DMZ, while a back-end database tier that requires incoming connections from the web tier but not from external client endpoints would be assigned to the production zone. Applications might also specify connection requirements to other non-application DCNs within the tenant network, such as backup servers. So long as these non-application DCNs are assigned to the appropriate security zone (e.g., the production zone), the connections from application DCNs are allowed (even if these non-application DCNs are in a different VHC than the application).
Each VHC spans a set of the sites of the physical infrastructure, and part of the application specification in some embodiments is a set of the sites of the application's VHC across which to deploy the application. As described above, each application tier is assigned to a security zone, and that security zone spans a set of sites. Thus, each application tier is assigned to a subset of the sites designated for the application that is the intersection of (i) the sites designated for the application and (ii) the sites to which the security zone for that application tier is restricted. That is, the DCNs implementing the application tier are deployed within that subset of sites. If there is no overlap between the sites designated for an application and the sites to which a security zone for a tier of the application is restricted, some embodiments generate an error and alert the application developer user of the error so that the application developer can designate a different set of sites for the application.
As part of assigning application tiers to security zones, the network management and control system of some embodiments also automatically populates various sets of firewall rules to implement the security zones for the application. For instance, for internal connections (e.g., for connections from the DMZ to the production zone), some embodiments automatically populate distributed firewall rules to allow these specific connections. These distributed firewall rules are implemented by the internal physical infrastructure (e.g., managed forwarding elements and/or firewall modules in virtualization software of host computers), as configured by the network management and control system.
In addition, if the application has connectivity to external networks (e.g., if the application requires connections from external endpoint clients), some embodiments auto-populate gateway (edge) firewall rules associated with a centralized routing component of a logical router. In some embodiments, these gateway firewall rules are arranged in sections. The firewall rules have relative priorities within a section, and the sections have different priorities as well, so that the highest priority rule in the highest priority section is the highest priority rule overall. The primary (highest priority) section in some embodiments is created and managed by the user that governs the logical router—e.g., the provider for a T0 logical router, or a tenant user for a T1 logical router.
For applications that require incoming connections, the firewall rules should be configured to allow connections in for that application, but not such that all incoming traffic is allowed at the gateway. Thus, some embodiments have a default rule denying incoming connections, but include a higher priority rule that specifies to skip (jump) to a different firewall rule section specific to the application if the incoming connection is directed to the application (e.g., based on a destination address and/or transport layer port in the packet, or a determination that the packet is directed to a particular logical port of the logical router). The application-specific firewall rule section has rules configured specifically for the application, typically along with another default rule denying connections that do not meet the specific required characteristics for the application. For instance, the application firewall rules might allow any incoming connections addressed correctly to the application, only allow incoming connections from certain addresses, etc.
In addition to assigning application tiers to security zones and defining firewall rules for the applications, some embodiments automatically generate logical networking constructs based on the application definition. Specifically, some embodiments allow the user that defines a VHC to define application association policies for the VHC. An application association policy, in some embodiments, specifies how an application with tiers is translated into logical network entities. For instance, an application association policy might automatically define a separate logical switch for each application tier, along with a T1 logical router that connects these new logical switches. If the T1 logical router has centralized services requiring the use of an edge gateway, then some embodiments automatically use an edge device located in an edge cluster (part of the physical infrastructure nodes of the provider policy tree) associated with the VHC in which the application is defined. If the application requires external connections (e.g., from external client endpoints or to internal backup servers located in a different VHC), then some embodiments automatically connect the T1 logical router to a particular T0 logical router that is exposed to the VHC. Rather than defining a separate logical switch for each tier, an application association policy could assign all of the tiers (i.e., all of the DCNs implementing the tiers) to the same logical switch. In this case, some embodiments differentiate the tiers using compute expressions (e.g., assigning tags or other metadata to the DCNs implementing the different tiers).
The preceding Summary is intended to serve as a brief introduction to some embodiments of the invention. It is not meant to be an introduction or overview of all inventive subject matter disclosed in this document. The Detailed Description that follows and the Drawings that are referred to in the Detailed Description will further describe the embodiments described in the Summary as well as other embodiments. Accordingly, to understand all the embodiments described by this document, a full review of the Summary, Detailed Description, and the Drawings is needed. Moreover, the claimed subject matters are not to be limited by the illustrative details in the Summary, Detailed Description, and the Drawings, but rather are to be defined by the appended claims, because the claimed subject matters can be embodied in other specific forms without departing from the spirit of the subject matters.
The novel features of the invention are set forth in the appended claims. However, for purpose of explanation, several embodiments of the invention are set forth in the following figures.
In the following detailed description of the invention, numerous details, examples, and embodiments of the invention are set forth and described. However, it will be clear and apparent to one skilled in the art that the invention is not limited to the embodiments set forth and that the invention may be practiced without some of the specific details and examples discussed.
Some embodiments of the invention provide a network management system for managing a virtualization infrastructure having one or more logical networks defined across one or more sites (e.g., datacenters). The network management system allows for a top-level user of the virtual infrastructure (e.g., a cloud provider, an enterprise's network administrators) to define a provider logical network and to define one or more second-level users (e.g., tenants, enterprise organizational units) of the virtualization infrastructure. These second-level users can in turn define accounts for different users. Specifically, a second-level user can define security and networking configuration (e.g., logical networking constructs, security zones) that is opaque to third-level users (e.g., application developers) as well as other second-level users. Application developers can specify their application requirements, without any knowledge of the security and networking configuration, and the network management system of some embodiments automatically deploys the application according to the requirements while ensuring that the security and networking policies are adhered to.
These logical networks, in some embodiments, are conceptual network structures that the irrespective user accounts define through the network management system. As mentioned, the top-level user defines a provider logical network (this logical network will be referred to herein as a provider logical network irrespective of whether the top-level user is a cloud provider, enterprise network admin, etc.). In some embodiments, this provider logical network is defined using a particular data model, such as a hierarchical policy tree with nodes and connections between the nodes. Through the data model, the top-level user can define security policies and security groups as well as logical network elements (e.g., logical routers and/or logical switches). In addition, in some embodiments, the provider data model includes a definition of the physical infrastructure onto which the virtualization infrastructure is deployed. This physical infrastructure of some embodiments may specify the multiple federated sites across which the virtualization infrastructure is deployed, as well as groups of physical elements within that site (e.g., groups of edge devices that physically connect to external networks in each site, zones of physical servers and/or forwarding elements within each site).
The logical network elements represented in the policy tree include logical forwarding elements (e.g. logical routers, logical switches, etc.). For example, in
The node for the T0 logical router 105 has a number of child nodes, including static route definitions 130 and locale services 135 and 140 referencing different physical sites A and B. This means that the T0 logical router 105 spans these two physical sites and has centralized routing components defined within these sites, as described further below by reference to the portion of the policy tree for these sites. Grand-child nodes are defined for interfaces 120 and 125 (e.g., uplink interfaces for external connections, service interfaces for third-party services) at each of these sites as well as an IPsec VPN 115 at site B.
The node for the T1 logical router 110 has a child node for the logical switch 145. In addition, this T1 logical router node 110 references the T0 logical router node 105 (the dashed arrow), indicating that the T1 logical router 110 connects to the T0 logical router 105 in order for logical network endpoints connected to the logical switch 145 to communicate with external endpoints (i.e., endpoints of other logical networks that also connect to the T0 logical router or endpoints completely external to the virtualization infrastructure). Though not shown for the sake of simplicity, many of the entities in the policy tree 100 have various additional characteristics that define the configuration of the represented logical network entities, which may be defined by the user or inherited from a parent entity. For instance, services such as DHCP may be defined for different logical forwarding elements, logical switches are assigned IP address ranges, etc.
In addition to the logical forwarding elements, the provider hierarchical policy tree 100 include nodes for physical entities, including nodes for each physical site (e.g., each datacenter in the virtualization infrastructure). For example, in
In the illustrated example, the edge clusters 151 and 152 have respective incoming references from locale services 135 and 140 attached to the T0 logical router 105. In some embodiments, edge clusters also have children corresponding to specific edge nodes (e.g., the edge node 153). In some embodiments, the network management system identifies the physical structure of a site (and therefore the structure of the child nodes for the site in the policy tree 100) via auto-discovery, when the site is added to the overall provider network represented by the global root node 102.
The provider network policy tree 100 also includes nodes for security-related entities. In some embodiments, security policy is at least partially organized by defining domains, which are logical groupings of one or more sites (e.g., geographic regions) that serve as an envelope to group different logical entities together (e.g., for security purposes). Domains are defined and represented as nodes in the policy tree 100 beneath the global root 102. In this case, a single domain node 170 is shown in the figure. Unlike sites (described below), which represent a physical construct, domains are a logical construct, which serve as an envelope to group different logical entities together (e.g., for security purposes). For example, firewall policies or other policy micro-segmentation applied to the domain will automatically be applied to all groups of logical endpoints defined within the domain in some embodiments.
Domains can represent a single physical site or multiple sites. In this example, the domain 170 corresponds to both of the sites A and B, and thus the domain enforcement point node 172 references both of the site enforcement point nodes 161 and 178. Some embodiments place restrictions on how sites can be organized into logical domains. For instance, each physical site may be restricted to membership in (i) a single domain containing only that site and (ii) a single domain containing more than one site. In addition, domains are only created as top-level nodes beneath the global root node 102 (i.e., domains are not children of other domains) in some embodiments. The provider user defines domains within the provider policy tree by selecting the sites that belong to the domain.
In some embodiments, logical network endpoints at each site are logically organized into security groups which can span multiple sites. These security groups are groups of logical network endpoints that share one or more attributes (e.g., operating system, region, etc.). Service machines as well as the managed network elements executing on host computers (the transport nodes) apply logical network policies (such as network policy 173) to the data messages exchanged between endpoints belonging to the security groups in some embodiments, based on policy rules that are defined in terms of these groups. In some embodiments, security groups and network policies are represented in the global policy tree 100 as child nodes of domains, and accordingly inherit their parent domain's span. In some embodiments, the span of a network policy is defined not only by its parent domain, but also by sites and/or domains which are referenced by the policy. The logical network policies in some embodiments include forwarding policies, service policies, and security policies, and are applied in some embodiments to govern the behavior of the logical forwarding elements. The policies can be child nodes of a logical network element node in some embodiments (e.g., static routing policy configuration for a logical router).
As noted,
The provider user can also create one or more second-level (tenant) users, providing initial information and constraints for these users, and exposing certain entities from the provider logical network to the tenant user. For instance, if the provider user has multiple T0 logical routers, the provider user can associate a specific one (or more) of these T0 logical routers (or T0 virtual routing and forwarding tables (VRFs), as described below) with the tenant. The tenant can use these T0 logical routers or T0 VRFs to provide external connectivity for the tenant logical network(s). The provider also associates one or more edge clusters (of the physical infrastructure) with the tenant; this may be a manual or automatic association in different embodiments. The tenant can use these edge clusters to host T1 logical router centralized routing components, centralized services (e.g., stateful services), etc. In addition, the provider user may place certain constraints on the tenant user, such as limiting the number of certain types of elements (e.g., a maximum number of T1 logical routers) or limiting the tenant logical network to a subset of sites of the overall physical infrastructure.
The tenant user can create their own configuration 210 through the network management and control system, defining logical networking and security policies. Unlike the provider user configuration 205, the tenant user configurations 210 (or 215) do not include any physical infrastructure; otherwise these user configurations are similar in structure to the provider user configuration. As shown in the figure, the tenant user is not able to view or modify the provider user configuration 205 via the network management system. As described in more detail below, however, certain logical and/or physical entities (e.g., one or more T0 logical routers, one or more edge clusters) of the provider user configuration 205 are exposed by the provider to the tenant user configuration 210. In addition, the tenant user configurations 210 and 215 are fully isolated from each other; that is, one tenant user has no ability to view, modify, or even necessarily be aware of any other tenant user configuration, even if they share physical infrastructure resources and/or the use of provider logical network entities.
Furthermore, as shown, tenant users can create additional sub-users. As shown in
In some embodiments, these sub-user accounts are organized hierarchically. That is, each tenant can create one or more sub-users (referred to as organizational units in some embodiments), and these sub-users can also create further sub-user accounts. Each of the sub-user accounts at any level can create their own isolated security and networking configuration, referred to in some embodiments as a virtual hybrid cloud (VHC). In other embodiments, the tenant and organizational units are containers for lower-level accounts, and only the accounts can create separate VHCs. As an example organization, the provider user might be an IT team for an enterprise, with the sites over which the virtualization infrastructure is implemented being a set of main and branch offices of the enterprise. In this case, a tenant user might be the network admin(s) for a line of business within the enterprise, and sub-users are defined for different organizational units within that business line, etc. Similarly, if the provider user is a public cloud provider with multiple datacenters and a tenant is a public cloud tenant (e.g., an enterprise), then the sub-users might be different business lines within the scope of the tenant.
In addition, some embodiments allow the tenants or sub-users to create application developer users. These users, as described in greater detail below, are not able to view or modify any logical networking configuration (neither the configuration of the user that created the application developer user nor of any higher-level users) but can define requirements for applications to be deployed in a particular VHC. As shown in
In some embodiments, the data model for the tenant configuration (or for other sub-user configurations) is similar to that for the provider configuration. That is, the tenant configuration is also represented (e.g., in storage of the network management system) as a hierarchical policy tree with similar nodes and connections.
On the logical networking side, the policy tree 300 includes a T1 logical router node 310 with three connected logical switch nodes 315-325. This T1 logical router node 310 is also shown as connecting to a T0 logical router node 105 that is part of the provider configuration. This T0 logical router 105 may be exposed to the tenant user (and thus its sub-users), enabling the user creating the VHC to connect to the T0 logical router 105 without viewing or modifying its full configuration. If the T1 logical router represented by node 310 has centralized components (e.g., for providing stateful services) rather than only having a distributed component, then the node 310 in some embodiments would also be associated with edge clusters in one or more sites (e.g., via locale service sub-nodes, as is the case for the T0 logical router 105 shown in
The policy tree 300 also includes security-related nodes, with a domain node 330 defining a domain. For simplicity, the links to one or more site enforcement points (in the provider policy tree) are not shown in this figure, but some embodiments do allow tenants and other sub-users to define domains across one or more sites (though the sites for a domain are limited by the sites to which the VHC is restricted).
All types of users may define security policy for their respective VHCs in some embodiments. This security policy may include, in some embodiments, security rules (e.g., distributed firewall rules and/or gateway firewall rules), security group definitions, etc.
For simplicity, the links to one or more site enforcement points (in the provider policy tree) are not shown in this figure, but some embodiments do allow tenants and other sub-users to define domains across one or more sites (though the sites for a domain are limited by the sites to which the VHC is restricted). Within this domain 330, the user has defined two security groups 335 and 340, as well as a security policy 345 that includes a rule 350 referencing the two security groups 335 and 340 (e.g., a rule allowing or denying traffic between Group A and Group B).
Each VHC in a virtualization infrastructure (i.e., a set of datacenters) managed by a network management system has a similar data model.
The provider creates a tenant user 410 as described, and directly within the domain of this tenant user 410, one organizational unit 415 is created and one account 420 is created. In addition, within the organization unit 415, that user creates an account 425. In some embodiments, any type of user may define a VHC, while in other embodiments only “account” users may define a VHC (as is the case in this example). In this case, the account user 425 defines two separate isolated VHCs 430 and 435, while the account 420 defines a single VHC 440 that is isolated from the first two VHCs 430 and 435 (but shares the same data model with both them and the provider policy tree 405). That is, a single account may define multiple VHCs (e.g., to deploy different applications).
The isolation between VHCs requires that entities in one VHC may not be associated with entities in other VHCs, in some embodiments. That is, there should be no connections between the policy tree nodes for one VHC and those of another VHC (except that they may both be associated with the same exposed entity from the provider logical network. While the VHCs are isolated, some embodiments allow some communication between endpoint DCNs in different VHCs (e.g., so that a DCN implementing an application in one VHC can access backup servers in a different VHC of the same tenant or sub-user). In some embodiments, when a user creates a VHC, before (or in addition to) defining the logical network, the user defines capabilities of the VHC (e.g., services offered within the VHC) as well as the capacity of the VHC (e.g., the sites to which the VHC is restricted, as well as various requirements related to the size, scale, latency, throughput, etc.). In accordance with having a similar data model for the respective configurations, the API is also similar for the provider user and for the various sub-users when defining (e.g., creating, modifying) the same type of entity.
As mentioned above, some embodiments only allow the provider user to create and configure T0 logical routers, and particular T0 logical routers or T0 VRFs can be associated with different tenants and their VHCs. This allows the provider user (e.g., the enterprise IT team, a cloud provider network admin) to oversee all of the rules regarding external connections to the virtualization infrastructure (e.g., routing protocols for route exchange with external routers, virtual private network (VPN) connections, etc.). The provider user can define multiple virtual routing and forwarding tables (VRFs) for a single T0 router in some embodiments. The tenant user, on the other hand, can only define T1 logical routers in some embodiments.
The types of logical routers in some embodiments will be quickly described in greater detail. In some embodiments, logical routers can be either T0 logical routers that provide a connection to external networks or T1 logical routers that (i) connect logical switches to each other, (ii) provide services (e.g., firewall, load balancing, etc.), and (iii) enable connections to T0 logical routers. T0 logical routers are generally defined by the network management and control system to have a distributed routing component (DR) that is implemented in a distributed manner and one or more centralized routing components (service routers, or SRs) that are implemented in a centralized manner by edge devices. T1 logical routers may be entirely distributed (e.g., if no centralized stateful services are defined for the T1 logical router) or may similarly have both a DR as well as one or more SRs. The details of these types of logical routers and logical routing components are further described in U.S. patent application Ser. No. 16/906,905, filed Jun. 19, 2020, now published as U.S. Patent Publication 2021/0314257, which is incorporated herein by reference.
The provider can expose external connectivity via a T0 logical router to the tenant VHCs in a variety of different ways in some embodiments. As noted above, some embodiments allow for one or more VRFs (i.e., independent routing tables to which packets may be assigned) to be defined for a particular T0 logical router and for these VRFs to be exposed to the tenant users (or the T0 VRFs can be managed by the tenant user), rather than exposing the T0 logical router itself. In some embodiments, the SR for a T0 logical router is configured to implement all of the VRFs for that T0 logical router, whether those VRFs are managed by the provider user or the tenant user. When a particular T1 logical router is connected to a particular T0 VRF, outgoing packets processed by the particular T1 logical router are sent to the T0 logical router in some embodiments with indicators (e.g., within an encapsulation header) that the packets are from the particular T1 logical router (e.g., based on a logical switch defined internally to connect the two logical routers). Based on this indicator, the T0 SR applies the correct VRF to the packets.
Finally,
While the tenant users cannot configure or view the configuration of entities defined within the provider logical network, as indicated, some embodiments allow the provider user to expose some of these entities for use by the tenant users. In some embodiments, the provider user exposes these entities to one or more tenants via labels, which allow the tenant user to identify the type of entity and any specific properties of import, but not to view or modify the full configuration of the entity.
The figure also illustrates that a number of these entities are exposed to tenants via labels. Specifically, the T0 logical router 1005 is exposed via a label 1040, the T0 VRF 1010 is exposed via a label 1045, the edge cluster 1035 is exposed via a label 1050, and the security group 1020 is exposed via a label 1055. The use of labels allows the provider to expose the entities without exposing the configuration of the entity (i.e., the tenant cannot view the configuration of the exposed entity at all). The T0 logical router label 1040 or T0 VRF label 1045 enables a tenant user to which these labels are exposed (or their sub-users) to connect their T1 logical routers to these entities without viewing the logical router configuration. Such labels might indicate the sites spanned by the T0 logical router, but not the VLAN segments connected to the T0 logical router or its routing protocol configuration. The edge cluster label 1050 might indicate the site at which the edge cluster is located, but not any details about the edge cluster that are available to the provider (e.g., the type or number of edge devices in the cluster). Finally, the security group label 1055 might indicate the attributes that cause a DCN to belong to the security group, but not the full membership of the group outside of the tenant VHC. It should be noted that other types of entities may be exposed via labels in some embodiments (e.g., services such as load balancers).
In addition to the types of users described above, in some embodiments tenant users or sub-users may define user accounts for application developers. In some embodiments, these application developer users are associated with a VHC, but may not view or configure the logical networking and security rules of this VHC (i.e., these users are not granted the ability to create a new VHC or view the configuration of the VHC with which they are associated). Rather these users can define applications within the scope of the associated VHC, and various policies defined for the VHC enable the automated implementation of these applications without additional intervention by the user that owns the VHC.
As shown, the process 1200 begins by receiving (at 1205) a definition of an application to be deployed in a particular VHC. The application definition, in some embodiments, specifies requirements for deployment of a real-world application. These applications may include widely used applications such as SharePoint, Epic, or SAP, as well as single- or multi-tier proprietary applications specific to a business's needs. The requirements specified by the application definition may include, e.g., connectivity requirements (e.g., whether external connectivity is required), service requirements (e.g., load balancers, etc.), and the set of available sites in which the application may be deployed (e.g., if the application does not comply with Europe's general data protection regulation (GDPR), it may only be deployed in North American sites), among other requirements. In some embodiments, as described further below, the application is defined in terms of a set of tiers (e.g., web tier, app tier, database tier), and the connectivity and other requirements for each tier.
The process 1200 then determines (at 1210) security zones into which the application is deployed. In some embodiments, the tenant users and their sub-users (i.e., the owner of a VHC) can define security zones within the VHC. In some embodiments, security zones are defined to govern the connectivity (or other security policies) of applications. That is, DCNs implementing an application or application tier are assigned to security zones, and the connectivity and/or other security policies of the security zone are applied to those DCNs. Examples of common security zones include a first zone that allows connections from external endpoints (often referred to as a DMZ) as well as a second zone that does not allow connections from external endpoints but does allow incoming connections from the first zone (often referred to as a production zone, and with which back-end servers or application tiers may be associated). Some embodiments may also include a testing security zone for applications in their testing phase (with minimal or no external traffic required at this point).
Next, the process 1200 determines (at 1215) the sites at which to deploy the application. In some embodiments, the user that defines a security zone also defines the span of that security zone in terms of sites (of the provider's physical infrastructure), thereby restricting applications (or tiers of applications) assigned to the security zone to those sites. The sites to which a security zone is restricted can include any of the sites to which the containing VHC is restricted, in some embodiments, but cannot include other sites that are not available for the VHC. An application can be deployed in any of the sites that are (i) specified for the application and (ii) available for the security zone(s) to which the application is assigned. In some embodiments, this determination is made on a tier-by-tier basis.
The process 1200 also determines (at 1220) the logical networking setup for the application. Some embodiments automatically generate logical networking constructs based on the application definition. Specifically, some embodiments allow the user that defines a VHC to define application association policies for the VHC. An application association policy, in some embodiments, specifies how an application is translated into logical network entities. For an application defined with multiple tiers (which may also be referred to as application subnets, in some cases), an application association policy might automatically define a separate logical switch for each application tier, along with a T1 logical router that connects these new logical switches. Rather than defining a separate logical switch for each tier, an application association policy could also assign all of the tiers (i.e., all of the DCNs implementing the tiers) to the same logical switch. In this case, some embodiments differentiate the tiers using compute expressions (e.g., assigning tags or other metadata to the DCNs implementing the different tiers).
The process 1200 also generates (at 1225) firewall rules for traffic to and from the application. In some embodiments, these firewall rules implement the security zones to which the application is assigned for the traffic to and from the application. For instance, for connections internal to the virtualization infrastructure (e.g., for connections between DCNs assigned to the DMZ and DCNs assigned to the production zone), some embodiments automatically populate distributed firewall rules to allow these specific connections.
In addition, if the application has connectivity to external networks, some embodiments auto-populate gateway (edge) firewall rules to ensure that these connections are allowed. In some embodiments, these gateway firewall rules are arranged in sections. The firewall rules have relative priorities within a section, and the sections have different priorities as well, so that the highest priority rule in the highest priority section is the highest priority rule overall. For applications that require incoming connections, the firewall rules are configured to allow connections in for that application, but not such that all incoming traffic is allowed at the gateway. Thus, some embodiments have a default rule denying incoming connections, but include a higher priority rule that specifies to skip (jump) to a different firewall rule section specific to the application if the incoming connection is directed to the application. The application-specific firewall rule section has rules configured specifically for the application, typically along with another default rule denying connections that do not meet the specific required characteristics for the application.
Next, the process 1200 deploys (at 1230) DCNs implementing the application at the determined sites. In some embodiments, this operation is actually performed by a compute manager that is separate but works in conjunction with the network management system. The DCNs may be VMs, containers, or other types of DCNs in different embodiments, and are deployed with software required to implement the application (or a specific tier of the application). The deployment, in some embodiments, also enables these DCNs to attach to the appropriate logical ports of the logical switches (or other logical forwarding elements) to which the DCNs are connected according to the determined logical networking constructs for the application.
Finally, the process 1200 configures (at 1235) managed network elements to implement the logical networking and firewall rules (and any other security or other services required for the application). These managed network elements, as described above, include various elements executing in the hypervisors of the nodes to perform logical switching, logical routing, implementation of distributed firewall rules, etc. The managed network elements also include the edge devices that implement the T0 and T1 SRs and implement gateway firewall rules in some embodiments. The process 1200 then ends.
As noted, tenant users and their sub-users are able to define security zones through the network management system in some embodiments. These security zones govern the connectivity or other security policies of applications, such that the connectivity and/or other security policies of a security zone are applied to the DCNs assigned to that security zone.
It should be noted that, in this example, the security zones are defined at the same level for a particular VHC. In some embodiments, the network management system allows a user (e.g., a tenant user or sub-user, or even a provider user) to define security zones that are available across all VHCs underneath that particular user. That is, a provider user could define the DMZ security zone for all VHCs in the entire virtualization infrastructure. Similarly, a tenant could define a DMZ security zone (if the provider does not) for all VHCs belonging to the tenant and its sub-users.
While the security zones applicable to a VHC are defined by the VHC owner or higher-level users, the applications are defined by an application developer user. These applications are defined as sets of application tiers in some embodiments, with requirements specified for each tier. For instance, applications are commonly specified as 3-tier applications, with a web tier of web servers that accept incoming connections from external endpoints, an application tier, and a back-end database tier.
In some embodiments, certain application service constructs are associated with the VHC, and the application developer can choose from these services when specifying the requirements for an application to be deployed in the VHC. These application service constructs may be exposed to the application developer as templates that indicate the configuration for a particular service (e.g., load balancing, etc.) with a combination of (i) pre-populated configuration elements that do not depend on the application specification, (ii) configuration elements that are populated based on application properties, and (iii) configuration elements that are populated based on explicit application developer input.
Upon receiving an application definition specifying one or more application tiers for a particular VHC, the network management and control system assigns each of these defined application tiers to one of the security zones defined for the particular VHC based on the requirements of the application (e.g., the connectivity requirements).
Each VHC is defined to span a set of the sites of the physical infrastructure, and part of the application specification in some embodiments is a set of the sites of the application's VHC across which the application can be deployed. In addition, the VHC owner (or other higher-level user) that defines a security zone defines the span of that security zone in terms of a set of sites to which the security zone is restricted, thereby restricting application tiers assigned to the security zone to those sites. Thus, each application tier is assigned to a subset of the sites designated for the application that is the intersection of (i) the sites designated for the application and (ii) the sites to which the security zone for that application tier is restricted. That is, the DCNs implementing the application tier are deployed within the subset of sites to which the application tier is assigned. If there is no overlap between the sites designated for an application and the sites to which a security zone for a tier of the application is restricted, some embodiments generate an error and alert the application developer user of the error so that the application developer can designate a different set of sites for the application
Specifically,
As shown, the process 1600 begins by receiving (at 1605) a definition of an application to be deployed in a particular VHC. The application definition, in some embodiments, specifies the tiers of the application, requirements for each tier of the application, and a set of sites across which the application can be deployed. In some embodiments, the sites specified for an application may only include sites with which the particular VHC is associated (e.g., in the case of
Returning to
The process 1600 assigns (at 1615) the selected tier to a security zone based on the requirements of the tier. In both of the examples, as in
Next, to determine the sites for the selected tier, the process 1600 identifies (at 1620) the intersection of (i) the sites designated for the application and (ii) the sites to which the assigned security zone for the selected application tier is restricted. That is, each application tier is restricted by two factors: the DCNs implementing that tier can only be deployed in sites that are both (i) specified for the application by the application developer and (ii) specified for the security zone to which the tier is assigned by the user that defined the security zone (e.g., the user that manages the VHC).
Based on this identification, the process 1600 determines (at 1625) whether there are any sites included in the intersection (i.e., whether or not the intersection is the null set). If there are no sites that satisfy both of these criteria, then the process 1600 generates (at 1630) an error. In
On the other hand, if at least one site satisfies both of the criteria for the selected tier, then the process 1600 assigns (at 1635) the workloads (e.g., the DCNs) for the tier to the identified site(s).
Next, the process 1600 determines (at 1640) whether additional tiers of the application remain to be evaluated. If additional tiers remain, the process returns to 1610 to select the next tier. As noted previously, the process 1600 is a conceptual process, and some embodiments actually evaluate all of the tiers at the same time.
Once all of the tiers have been evaluated (and no errors are generated), the process 1600 deploys (at 1645) DCNs implementing each tier at the sites to which the tier workloads are assigned. In some embodiments, this deployment operation is actually performed by a compute manager that is separate but works in conjunction with the network management system. The DCNs may be VMs, containers, or other types of DCNs in different embodiments, and are deployed with software required to implement the respective application tier. The deployment, in some embodiments, also enables these DCNs to attach to the appropriate logical ports of the logical switches (or other logical forwarding elements) to which the DCNs are connected according to the determined logical networking constructs for the application, which is described in more detail below.
As part of the process of assigning application tiers to security zones, the network management and control system of some embodiments automatically generates various sets of firewall rules in order to implement the security zones for the application. These firewall rules enforce the required ability or inability of DCNs in different security zones to communicate with each other and/or with external sources. In some embodiments, the firewall rules defined to implement security zones for an application include both distributed firewall rules (enforced by managed forwarding elements and/or firewall modules executing in virtualization software of host computers) and gateway firewall rules (enforced by the edge devices or firewalls connected to the edge devices).
As shown, the process 2000 begins by receiving (at 2005) a definition of an application specifying the tiers of the application and the connectivity (and other) requirements for each of the tiers. As mentioned, some of these requirements are specified in terms of application services that will be consumed by the application. In some embodiments, the application developer is required to provide values for certain services according to application service templates
The process 2000 then assigns (at 2010) the tiers to security zones, as described above (e.g., by reference to
The process 2000 defines (at 2015) a domain for the application in the policy tree of the VHC in which the application is defined. As described above, a domain in some embodiments is a logical construct that serves as an envelope for various security groups and policies. While the process 2000 described herein defines one domain for the application (with all of the security groups relating to the application), some embodiments define a separate domain for each of the services specified by the application.
In this case, within the domain, the process 2000 defines (at 2020) security groups for (i) the application and (ii) each tier of the application. A given DCN may belong to numerous different security groups. For instance, a DCN that is part of the web tier of an application could belong to a first security group for any DCN implementing the application, a second security group for DCNs implementing the web tier of the application, a third security group including any DCNs executing a particular operating system, a fourth security group for DCNs assigned to the DMZ (assuming that the web tier is assigned to this security zone), etc.
The process 2000 also defines (at 2025) distributed firewall rules to allow communication (i) between different application tiers that need to be able to communicate with each other and (ii) between application tiers and other internal DCNs if this communication is required. In some embodiments, the distributed firewall rules specify to either allow or deny specific traffic (e.g., from one or more source security groups to one or more destination security groups). Specifically, some embodiments use a whitelisting scheme, in which all traffic is assumed to be blocked unless specifically allowed by a firewall rule (e.g., by having a default rule denying traffic that does not meet any other rule). As an example of such a rule, traffic from the web tier of an application to the database tier of that application can be allowed with a distributed firewall specifying to allow traffic with the web tier security group as the source and the database tier security group as the destination. In addition, certain applications may require connections with DCNs internal to the virtualization infrastructure. For instance, a particular tenant might setup backup servers for use by multiple applications in a VHC or a set of VHCs. In some embodiments, these backup servers (or other services with which multiple applications communicate) are created as a separate application, and distributed firewall rules are defined so as to allow traffic between the backup servers and the application tiers that require connectivity with the backup servers. In some embodiments, each connectivity requirement is specified in terms of an application service that in turn specifies one or more distributed (or gateway) firewall rules based on the application specifics.
In addition, the process 2000 determines (at 2030) whether connectivity with external sources is required. In some embodiments, this connectivity is specified as an application service requirement for a specific tier of the application and invokes the creation of specific firewall rules. Because this traffic passes through the T0 logical router SR (i.e., the gateway at an edge device), some embodiments generate gateway (edge) firewall rules (i.e., firewall rules associated with the SR) to allow certain traffic to (and from) the application.
If connectivity is required with external sources, the process defines (at 2035) a firewall rule application section (i.e., section of firewall rules specific to the application) for allowing specific traffic from external sources to the application. In some embodiments, the gateway firewall rules (and, separately, the distributed firewall rules) are arranged in sections. The firewall rules have relative priorities within a section, and the sections have different priorities as well, so that the highest priority rule in the highest priority section is the highest priority rule overall. The primary (highest priority) section in some embodiments is created and managed by the user that governs the logical router—e.g., the provider for a T0 logical router, or a tenant user for a T1 logical router. In addition, some embodiments generate a section for each security zone (e.g., a section for the DMZ, a section for the production zone, etc.), as well as a separate (lower-priority) section for the application. Within the application section, the process 2000 adds a rule allowing specific traffic from external sources (e.g., from the Internet, from any IP address, etc.) directed to the application. In certain cases, this rule might use a public IP address used by the application, rather than the actual IP addresses of the web tier of the application, as this public IP address will be used in traffic processed by the T0 gateway (before performing network address translation on the traffic). The rule allowing certain traffic might also be configured to only allow connections from certain addresses, depending on the specified connectivity requirement for the application. In addition, the application section typically includes a default rule specifying that any other traffic that does not meet the specified criteria is to be denied.
The process 2000 also defines (at 2040) a gateway firewall rule in the primary section that jumps to the application section for traffic from external sources directed to the application. This primary section might be the highest-priority application section managed by the provider as part of the T0 gateway or could be a section associated with the specific security zone that receives incoming connections (e.g., the DMZ). That is, this primary firewall rule section should not be configured to allow all incoming traffic received at the gateway, and therefore has a default rule denying traffic. Rather than specifically allowing traffic at a higher priority, a rule with a new type of action skipping the rest of the section (and jumping to the application section) is generated so that data traffic directed to the application is referred to the lower-priority application firewall rule section. The details of whether to allow or deny such incoming traffic are therefore handled by the rules generated for that section (at 2035). This new rule might identify the incoming traffic for the application based on a destination address and/or transport layer port in the packet, or a determination that the packet is directed to a particular logical port of the logical router associated with the application.
After generating the required firewall rules for the application, the process 2000 configures (at 2045) managed network elements to implement the defined firewall rules. As noted previously, the distributed firewall rules are implemented by the internal physical infrastructure (e.g., managed forwarding elements and/or firewall modules in virtualization software of host computers) of some embodiments, and the network management and control system configures these elements to use the newly generated firewall rules. Similarly, the edge devices that implement the T0 SRs also implement the gateway firewall rules in some embodiments, and the network management and control system also configures these edge devices and/or their associated edge firewalls.
The gateway firewall 2105 includes firewall rules in at least two sections. The primary section 2110 includes a first rule specifying that if the destination IP address of a data message is “App.Public.IP” (i.e., the public IP address used by a particular application), then the action to be taken is to skip to the application section 2115 (without making a final decision on whether to allow or deny the packet). The primary section 2110 also includes a second rule that denies any other incoming data messages. The application section 2115 includes a first rule specifying that if the destination IP address of a data message is “App.Public.IP” and the destination transport layer port is “K”, then the data message should be allowed. That is, the application connectivity requirements, as translated to firewall rules, require that data messages directed to the application also specify the correct transport layer port K. The section 2115 also includes a default deny rule.
In
In
In addition to assigning application tiers to security zones and defining firewall rules for the applications, some embodiments automatically generate logical networking constructs based on the application definition. Specifically, some embodiments allow the user that manages a VHC (or a higher-level user) to define application association policies for the VHC. These application association policies, in some embodiments, specify how an application definition with one or more tiers is translated into logical networking constructs (e.g., logical routers, logical switches, etc.).
As shown, the process 2300 begins by receiving (at 2305) a definition of an application to be deployed in a VHC specifying one or more tiers and connectivity requirements (as well as other types of requirements) for each of the tiers. As described above, the application definition of some embodiments specifies the number of tiers, connectivity and other service requirements for each tier, and other information. In some embodiments, if the application is a common application type (e.g., Epic, SharePoint, etc.), the application developer user may use a template provided by the network management system for the application that uses a standard set of tiers and requirements for the application type, with minimal customization. If the application is a proprietary application, then some embodiments require additional customization (e.g., the number of tiers, specific connectivity requirements) from the application developer.
Based on the application definition, the process 2300 determines (at 2310) an appropriate application association policy of the VHC in which the application is to be deployed to use for generating logical networking constructs for the application. In some embodiments, the manager of the VHC affiliates one or more application association policies (which may be pre-defined for the system or generated by the user that manages the VHC) with the VHC. Different application association policies may specify different logical networking constructs based on the number of tiers in the application (e.g., different policies for 2-tier and 3-tier applications), whether the application is a specific type of application (e.g., there might be a first policy for SharePoint applications, a second policy for blog applications, etc.), or based on other factors.
The first application association policy 2405 (for Type 1 applications) specifies to create a separate segment (e.g., logical switch) for each tier of the application and to use IP address ranges available to the VHC to assign subnets for each of these segments. That is, the VHC will have been assigned a set of private IP addresses (e.g., by the tenant user that creates the VHC or creates the user that creates the VHC) to be used for the DCNs in the VHC for communication internal to the virtualization infrastructure. The application association policy specifies to select a subnet from within this set of IP addresses to associate with the logical switch; DCNs added to the logical switch will be assigned an IP address from within the subnet. The application association policy 2405 also specifies to create a T1 logical router to which each of the newly created segments attaches. Assuming that at least one edge cluster in the physical infrastructure has been associated with the VHC, the application association policy 2405 specifies for this edge cluster to be selected for hosting the SR(s) for the T1 logical router. Finally, the application association policy 2405 specifies for the T1 logical router to be connected to the T0 logical router or T0 VRF (e.g., of the provider network) that is associated with the VHC in which the application is deployed.
The second application association policy 2410 (for Type 2 applications, which in this case do not require external connectivity and may also have other specific properties) specifies to create a single segment for all tiers of the application and to use an IP address range available to the VHC to assign a subnet for the segment. That is, for these applications, all of the DCNs implementing any of the application tiers are connected to the same logical switch. In this case, some embodiments differentiate the tiers using compute expressions associated with the DCNs, so that the DCNs can still be easily assigned to different security groups for the different tiers. For instance, different embodiments have the compute manager assign tags or other metadata to the DCNs that indicate to which tier each DCN belongs. Other embodiments use specific IP addresses within the logical switch subnet to differentiate the tiers. As shown, a T1 logical router is still created according to the second application association policy 2410. In addition, the logical switch is still connected to this T1 logical router and an edge cluster associated with the VHC is selected to host the SR(s) for the T1 logical router. The T1 logical router is not connected to any T0 logical router or VRF in this case, as external connectivity is presumably not needed. This application association policy 2410 could be used for applications in a testing phase in some embodiments.
Returning to
The process 2300 also creates (at 2320) a T1 logical router to connect the one or more segments created for the application tiers and assigns one or more edge clusters for the T1 gateway (i.e., the T1 SR). It should be noted that if no centralized services are required for the T1 logical router (e.g., if the only purpose of the T1 logical router is to connect the logical switches), then the T1 logical router may be entirely distributed and no edge cluster assignment is required.
The process 2300 then determines (at 2325) whether connectivity with external sources is required. It should be noted that the process 2300 is a conceptual process, and in some embodiments this determination is part of the determination as to the appropriate application association policy. That is, whether or not connectivity is required goes into determining the application association policy, and the logical networking constructs to be generated are simply specified by that policy.
When external connectivity is required for the application (e.g., for a particular tier of the application), the process 2300 connects (at 2330) the T1 logical router created for the application to the T0 logical router or T0 VRF that is assigned to the VHC in which the application will be deployed. Whether the T0 logical router or one of multiple VRFs associated with the T0 logical router is assigned to the VHC depends on which entities are exposed to the tenant by the provider (and, in turn, which entities the tenant assigns to the sub-user that manages the VHC if the VHC is not directly managed by the tenant user).
Based on the application association policy 2405, the network management system defines a set of logical network structures that are shown in the policy tree 2500. It should be understood that this policy tree is not complete, as it does not show either (i) the security constructs defined for the application or (ii) any other logical networking constructs defined within the VHC (e.g., for other applications). Underneath its root node 2525, the policy tree 2500 includes a T1 logical router node 2530 and three logical switch nodes 2535-2545, with each logical switch corresponding to a different tier of the application. In addition, some embodiments define a DHCP service 2550 associated with the T1 logical router for the application. This DHCP service operates (e.g., at the edge cluster with which the T1 logical router 2530 is associated) to assign IP addresses to the DCNs implementing the application tiers that connect to any of the logical switches 2535-2545. Lastly, the T1 logical router is connected to a T0 logical router represented by node 2555, which is part of the provider network but exposed to the VHC represented by the policy tree 2500.
Returning to
The VHC 2610 is managed by a VHC administrator (e.g., a tenant, a sub-user of the tenant, etc.), and is defined to span all of sites 1-4. In addition to the DMZ security zone 2605 associated with the VHC, a VHC security administrator (which may be the same as the VHC administrator or a specific security admin user created by the VHC admin) defines a production security zone 2615 which is restricted to sites 2-4. Unlike the DMZ security zone 2605, the production security zone 2615 is specific to the VHC 2610.
Within the VHC, two different applications 2620 and 2625 have been created by two different application owners. The first application 2620 is a three-tier application with a web tier (assigned to the DMZ security zone 2605, as shown by the dash-dot border), an app tier, and a database tier (both of which are assigned to the production security zone 2615, as shown by the long-dash border). Three workloads (e.g., VMs, containers, etc.) implement the web tier and load balancing is used to distribute traffic between these workloads, two workloads implement the app tier, and one workload implements the database tier. The second application 2625 is a two-tier blog application with a web tier (assigned to the DMZ security zone 2605) and a backend tier (assigned to the production security zone 2615). Three workloads implement the web tier and load balancing is again used to distribute traffic between these workloads, while a single workload implements the backend tier.
The bottom portion of the diagram 2600 shows the security realization 2630 and connectivity realization 2635 for the three-tier application 2620. In this example, security groups are defined for (i) the database tier of the application, (ii) the app tier of the application, (iii) the web tier of the application, (iv) the application itself (a group that would include all workloads in any of the application tiers), (v) the DMZ security zone (i.e., any workloads assigned to this security zone), and (vi) the production security zone. The first four groups are relevant only to this application, and corresponding security groups would be created for the second application 2625 (not shown), while the last two groups corresponding to the security zones are applicable to any other applications deployed in the VHC.
In addition, the security policies are broken down into different sections, shown in order of priority. The highest-priority section is the DMZ section (which includes a rule for each of the applications to skip to the application-specific sections for data traffic between external endpoints and the web tier workloads for those applications), followed by the production section, and then the two application-specific sections of security policies.
The connectivity realization 2635 shows the logical networking configured for the three-tier application 2620. In this case, the three workloads of the web tier are attached to a first segment (logical switch), the two workloads of the app tier are attached to a second segment, and the single workload of the database tier are attached to a third segment. Each of these segments is connected to a T1 logical router defined for the three-tier application, which in turn connects to a T0 logical router (presumably managed by the provider).
Finally,
The provider user also creates the tenant user 2715, which manages the tenant section 2710 and specifies the physical sites that the tenant may use. It should be understood that any number of tenant users, each managing their own sections, may be created by the provider user. The tenant may specify any number of sub-users, in this case organizational units. Either the tenant directly or these organizational units may specify account sub-users (e.g., the account 2720), and the creator of this account specifies the physical sites that the account may use for its one or more VHCs. The tenant user can also specify one or more shared security zones (e.g., a DMZ) that are shared across all organization units and/or accounts created underneath the tenant, as well as an application service template for connectivity, load balancing, or other services. As described, the application service templates define the configuration (possibly according to details from an application developer) for a service consumed by an application tier, such as external connectivity, communication with backup servers, load balancing, IDS/IPS, etc.
The account user 2720 can also define a VHC 2725, which may be configured by the account user 2720 or a separate VHC admin. Within a VHC 2725, the account user 2720 or the VHC admin can define security zones (in addition to the shared security zones defined at the tenant level) and restrict these security zones to certain sites (so long as those sites are within the purview of the VHC). The account user 2720 or VHC admin can also define application association policies for the VHC that specify how to turn application definitions into logical networking constructs.
An application owner (application developer) governs their application definition 2730, which is under the purview of the VHC 2725. The application owner specifies the application, indicating the sites at which the application can be deployed, as well as one or more application tiers. For each application tier, application services may be invoked for consumption by the tier by reference to application service templates defined by the tenant. Each application tier is automatically associated with a security zone, either one that is defined within the VHC (as in this case) or that is shared between VHCs. In addition, an application association policy is used to generate logical networking constructs (e.g., T1 logical routers, segments) under the root node (infra) for the VHC policy tree definition. This policy tree also includes IP assignment information (IP pools and IP blocks) as well as the security information (domains, security policies with rules, and security groups) either specified by the VHC admin (or other higher-level user) or automatically generated based on the definition of an application for the VHC. The security policy rules in the VHC can refer to services and/or security groups defined within the VHC and/or defined within the provider policy tree. Similarly, security rules within the provider network can refer to services and/or security groups that are defined within the provider policy tree and/or a VHC. These rules can specify the security groups as sources and/or destinations of traffic and indicate whether or not such traffic should be allowed.
The bus 2805 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the electronic system 2800. For instance, the bus 2805 communicatively connects the processing unit(s) 2810 with the read-only memory 2830, the system memory 2825, and the permanent storage device 2835.
From these various memory units, the processing unit(s) 2810 retrieve instructions to execute and data to process in order to execute the processes of the invention. The processing unit(s) may be a single processor or a multi-core processor in different embodiments.
The read-only-memory (ROM) 2830 stores static data and instructions that are needed by the processing unit(s) 2810 and other modules of the electronic system 2800. The permanent storage device 2835, on the other hand, is a read-and-write memory device. This device 2835 is a non-volatile memory unit that stores instructions and data even when the electronic system 2800 is off. Some embodiments of the invention use a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) as the permanent storage device 2835.
Other embodiments use a removable storage device (such as a floppy disk, flash drive, etc.) as the permanent storage device 2835. Like the permanent storage device 2835, the system memory 2825 is a read-and-write memory device. However, unlike storage device 2835, the system memory 2825 is a volatile read-and-write memory, such as random-access memory. The system memory 2825 stores some of the instructions and data that the processor needs at runtime. In some embodiments, the invention's processes are stored in the system memory 2825, the permanent storage device 2835, and/or the read-only memory 2830. From these various memory units, the processing unit(s) 2810 retrieve instructions to execute and data to process in order to execute the processes of some embodiments.
The bus 2805 also connects to the input and output devices 2840 and 2845. The input devices 2840 enable the user to communicate information and select commands to the electronic system 2800. The input devices 2840 include alphanumeric keyboards and pointing devices (also called “cursor control devices”). The output devices 2845 display images generated by the electronic system. The output devices 2845 include printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD). Some embodiments include devices such as a touchscreen that function as both input and output devices.
Finally, as shown in
Some embodiments include electronic components, such as microprocessors, storage and memory that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a variety of recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and/or solid state hard drives, read-only and recordable Blu-Ray® discs, ultra-density optical discs, any other optical or magnetic media, and floppy disks. The computer-readable media may store a computer program that is executable by at least one processing unit and includes sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as is produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter.
While the above discussion primarily refers to microprocessor or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions that are stored on the circuit itself.
As used in this specification, the terms “computer”, “server”, “processor”, and “memory” all refer to electronic or other technological devices. These terms exclude people or groups of people. For the purposes of the specification, the terms display or displaying means displaying on an electronic device. As used in this specification, the terms “computer-readable medium,” “computer-readable media,” and “machine-readable medium” are entirely restricted to tangible, physical objects that store information in a form that is readable by a computer. These terms exclude any wireless signals, wired download signals, and any other ephemeral signals.
This specification refers throughout to computational and network environments that include virtual machines (VMs). However, virtual machines are merely one example of data compute nodes (DCNs) or data compute end nodes, also referred to as addressable nodes. DCNs may include non-virtualized physical hosts, virtual machines, containers that run on top of a host operating system without the need for a hypervisor or separate operating system, and hypervisor kernel network interface modules.
VMs, in some embodiments, operate with their own guest operating systems on a host using resources of the host virtualized by virtualization software (e.g., a hypervisor, virtual machine monitor, etc.). The tenant (i.e., the owner of the VM) can choose which applications to operate on top of the guest operating system. Some containers, on the other hand, are constructs that run on top of a host operating system without the need for a hypervisor or separate guest operating system. In some embodiments, the host operating system uses name spaces to isolate the containers from each other and therefore provides operating-system level segregation of the different groups of applications that operate within different containers. This segregation is akin to the VM segregation that is offered in hypervisor-virtualized environments that virtualize system hardware, and thus can be viewed as a form of virtualization that isolates different groups of applications that operate in different containers. Such containers are more lightweight than VMs.
Hypervisor kernel network interface modules, in some embodiments, are non-VM DCNs that include a network stack with a hypervisor kernel network interface and receive/transmit threads. One example of a hypervisor kernel network interface module is the vmknic module that is part of the ESXi™ hypervisor of VMware, Inc.
It should be understood that while the specification refers to VMs, the examples given could be any type of DCNs, including physical hosts, VMs, non-VM containers, and hypervisor kernel network interface modules. In fact, the example networks could include combinations of different types of DCNs in some embodiments.
While the invention has been described with reference to numerous specific details, one of ordinary skill in the art will recognize that the invention can be embodied in other specific forms without departing from the spirit of the invention. In addition, a number of the figures (including
| Number | Date | Country | Kind |
|---|---|---|---|
| 202041042167 | Sep 2020 | IN | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 6219699 | McCloghrie et al. | Apr 2001 | B1 |
| 6347374 | Drake et al. | Feb 2002 | B1 |
| 7502884 | Shah et al. | Mar 2009 | B1 |
| 7539745 | Wang et al. | May 2009 | B1 |
| 7802000 | Huang et al. | Sep 2010 | B1 |
| 8479275 | Naseh | Jul 2013 | B1 |
| 8611351 | Gooch et al. | Dec 2013 | B2 |
| 8625616 | Vobbilisetty et al. | Jan 2014 | B2 |
| 8660129 | Brendel et al. | Feb 2014 | B1 |
| 8707417 | Liang et al. | Apr 2014 | B1 |
| 8805971 | Roth et al. | Aug 2014 | B1 |
| 8856077 | Roth et al. | Oct 2014 | B1 |
| 9311122 | Guay et al. | Apr 2016 | B2 |
| 9330161 | D'Amato et al. | May 2016 | B2 |
| 9432215 | Stabile et al. | Aug 2016 | B2 |
| 9602312 | Koponen et al. | Mar 2017 | B2 |
| 9672054 | Gupta et al. | Jun 2017 | B1 |
| 9672060 | Behere et al. | Jun 2017 | B2 |
| 9755965 | Yadav et al. | Sep 2017 | B1 |
| 9825851 | Agarwal et al. | Nov 2017 | B2 |
| 9847938 | Chanda et al. | Dec 2017 | B2 |
| 9858559 | Raleigh et al. | Jan 2018 | B2 |
| 9882968 | Holgers et al. | Jan 2018 | B1 |
| 9912616 | Shen et al. | Mar 2018 | B2 |
| 9923811 | Agarwal et al. | Mar 2018 | B2 |
| 9977688 | Nipane et al. | May 2018 | B2 |
| 10091028 | Koponen et al. | Oct 2018 | B2 |
| 10110417 | Hankins et al. | Oct 2018 | B1 |
| 10120668 | Palavalli et al. | Nov 2018 | B2 |
| 10135675 | Yu et al. | Nov 2018 | B2 |
| 10142127 | Cherian et al. | Nov 2018 | B2 |
| 10162656 | Palavalli et al. | Dec 2018 | B2 |
| 10187302 | Chu et al. | Jan 2019 | B2 |
| 10205771 | Palavalli et al. | Feb 2019 | B2 |
| 10241820 | Lambeth et al. | Mar 2019 | B2 |
| 10243797 | Lambeth et al. | Mar 2019 | B2 |
| 10243834 | Shekhar et al. | Mar 2019 | B1 |
| 10243846 | Jiang et al. | Mar 2019 | B2 |
| 10243848 | Agarwal et al. | Mar 2019 | B2 |
| 10257049 | Fried et al. | Apr 2019 | B2 |
| 10333959 | Katrekar et al. | Jun 2019 | B2 |
| 10339123 | Venkatesh et al. | Jul 2019 | B2 |
| 10382529 | Wan et al. | Aug 2019 | B2 |
| 10560343 | Cartsonis et al. | Feb 2020 | B1 |
| 10579945 | Gaurav et al. | Mar 2020 | B2 |
| 10581755 | Jain et al. | Mar 2020 | B2 |
| 10601705 | Hira et al. | Mar 2020 | B2 |
| 10637800 | Wang et al. | Apr 2020 | B2 |
| 10673752 | Agarwal et al. | Jun 2020 | B2 |
| 10693833 | Mathew et al. | Jun 2020 | B2 |
| 10832224 | Palavalli et al. | Nov 2020 | B2 |
| 10862753 | Hira et al. | Dec 2020 | B2 |
| 10880158 | Lambeth et al. | Dec 2020 | B2 |
| 10880170 | Wang et al. | Dec 2020 | B2 |
| 10897420 | Pianigiani et al. | Jan 2021 | B1 |
| 10908938 | Palavalli et al. | Feb 2021 | B2 |
| 10942788 | Palavalli et al. | Mar 2021 | B2 |
| 10999154 | Ahrenholz et al. | May 2021 | B1 |
| 11057275 | Arunachalam et al. | Jul 2021 | B1 |
| 11088902 | Palavalli et al. | Aug 2021 | B1 |
| 11088916 | Chandrashekhar et al. | Aug 2021 | B1 |
| 11088919 | Chandrashekhar et al. | Aug 2021 | B1 |
| 11115301 | Margarian et al. | Sep 2021 | B1 |
| 11153170 | Chandrashekhar et al. | Oct 2021 | B1 |
| 20020029270 | Szczepanek | Mar 2002 | A1 |
| 20020093952 | Gonda | Jul 2002 | A1 |
| 20020131414 | Hadzic | Sep 2002 | A1 |
| 20030167333 | Kumar et al. | Sep 2003 | A1 |
| 20030185151 | Kurosawa et al. | Oct 2003 | A1 |
| 20030185152 | Nederveen et al. | Oct 2003 | A1 |
| 20030188114 | Lubbers et al. | Oct 2003 | A1 |
| 20030188218 | Lubbers et al. | Oct 2003 | A1 |
| 20040052257 | Abdo et al. | Mar 2004 | A1 |
| 20040098447 | Verbeke et al. | May 2004 | A1 |
| 20050114862 | Bisdikian et al. | May 2005 | A1 |
| 20050190757 | Sajassi | Sep 2005 | A1 |
| 20050235352 | Staats et al. | Oct 2005 | A1 |
| 20050288040 | Charpentier et al. | Dec 2005 | A1 |
| 20060092976 | Lakshman et al. | May 2006 | A1 |
| 20060179243 | Fields et al. | Aug 2006 | A1 |
| 20060179245 | Fields et al. | Aug 2006 | A1 |
| 20060193252 | Naseh et al. | Aug 2006 | A1 |
| 20060221720 | Reuter | Oct 2006 | A1 |
| 20060251120 | Arimilli et al. | Nov 2006 | A1 |
| 20060287842 | Kim | Dec 2006 | A1 |
| 20070008884 | Tang | Jan 2007 | A1 |
| 20070058631 | Mortier et al. | Mar 2007 | A1 |
| 20070130295 | Rastogi et al. | Jun 2007 | A1 |
| 20070171921 | Wookey et al. | Jul 2007 | A1 |
| 20070217419 | Vasseur | Sep 2007 | A1 |
| 20070219653 | Martin | Sep 2007 | A1 |
| 20070239987 | Hoole et al. | Oct 2007 | A1 |
| 20080013474 | Nagarajan et al. | Jan 2008 | A1 |
| 20080049646 | Lu | Feb 2008 | A1 |
| 20080104302 | Carpio | May 2008 | A1 |
| 20080133729 | Fridman et al. | Jun 2008 | A1 |
| 20080268847 | Mukherjee et al. | Oct 2008 | A1 |
| 20080301379 | Pong | Dec 2008 | A1 |
| 20090037367 | Wein | Feb 2009 | A1 |
| 20090070337 | Romem et al. | Mar 2009 | A1 |
| 20090193297 | Williams et al. | Jul 2009 | A1 |
| 20090241192 | Thomas | Sep 2009 | A1 |
| 20090279536 | Unbehagen et al. | Nov 2009 | A1 |
| 20090279545 | Moonen | Nov 2009 | A1 |
| 20090292858 | Lambeth et al. | Nov 2009 | A1 |
| 20090296726 | Snively et al. | Dec 2009 | A1 |
| 20100250784 | Henry et al. | Sep 2010 | A1 |
| 20100257263 | Casado et al. | Oct 2010 | A1 |
| 20100275199 | Smith et al. | Oct 2010 | A1 |
| 20100322255 | Hao et al. | Dec 2010 | A1 |
| 20110032898 | Kazmi et al. | Feb 2011 | A1 |
| 20110047218 | Nojima et al. | Feb 2011 | A1 |
| 20110051714 | Somes | Mar 2011 | A1 |
| 20110085569 | Gnanasekaran et al. | Apr 2011 | A1 |
| 20110164752 | Wainner et al. | Jul 2011 | A1 |
| 20110188509 | Kern et al. | Aug 2011 | A1 |
| 20110225293 | Rathod | Sep 2011 | A1 |
| 20110231602 | Woods et al. | Sep 2011 | A1 |
| 20110299413 | Chatwani et al. | Dec 2011 | A1 |
| 20120084406 | Kumbalimutt | Apr 2012 | A1 |
| 20120089845 | Raleigh | Apr 2012 | A1 |
| 20120120964 | Koponen et al. | May 2012 | A1 |
| 20120147898 | Koponen et al. | Jun 2012 | A1 |
| 20120185913 | Martinez et al. | Jul 2012 | A1 |
| 20120275328 | Iwata et al. | Nov 2012 | A1 |
| 20130018947 | Archer et al. | Jan 2013 | A1 |
| 20130024579 | Zhang et al. | Jan 2013 | A1 |
| 20130042242 | Kagan | Feb 2013 | A1 |
| 20130044636 | Koponen et al. | Feb 2013 | A1 |
| 20130044641 | Koponen et al. | Feb 2013 | A1 |
| 20130044761 | Koponen | Feb 2013 | A1 |
| 20130058250 | Casado et al. | Mar 2013 | A1 |
| 20130058335 | Koponen et al. | Mar 2013 | A1 |
| 20130058350 | Fulton | Mar 2013 | A1 |
| 20130058354 | Casado et al. | Mar 2013 | A1 |
| 20130058358 | Fulton et al. | Mar 2013 | A1 |
| 20130060819 | Lambeth et al. | Mar 2013 | A1 |
| 20130060940 | Koponen et al. | Mar 2013 | A1 |
| 20130060945 | Allam et al. | Mar 2013 | A1 |
| 20130074065 | McNeeney et al. | Mar 2013 | A1 |
| 20130103817 | Koponen et al. | Apr 2013 | A1 |
| 20130125120 | Zhang et al. | May 2013 | A1 |
| 20130132533 | Padmanabhan et al. | May 2013 | A1 |
| 20130132561 | Pasala et al. | May 2013 | A1 |
| 20130144992 | Barabash et al. | Jun 2013 | A1 |
| 20130159637 | Forgette et al. | Jun 2013 | A1 |
| 20130212212 | Addepalli et al. | Aug 2013 | A1 |
| 20130215769 | Beheshti-Zavareh et al. | Aug 2013 | A1 |
| 20130232480 | Winterfeldt et al. | Sep 2013 | A1 |
| 20130254328 | Inoue et al. | Sep 2013 | A1 |
| 20130262685 | Shelton et al. | Oct 2013 | A1 |
| 20130282994 | Wires et al. | Oct 2013 | A1 |
| 20130286833 | Torres et al. | Oct 2013 | A1 |
| 20130287026 | Davie | Oct 2013 | A1 |
| 20130301425 | Udutha et al. | Nov 2013 | A1 |
| 20130301501 | Olvera-Hernandez et al. | Nov 2013 | A1 |
| 20130304616 | Raleigh et al. | Nov 2013 | A1 |
| 20130308641 | Ackley | Nov 2013 | A1 |
| 20140006465 | Davis et al. | Jan 2014 | A1 |
| 20140059544 | Koganty et al. | Feb 2014 | A1 |
| 20140064104 | Nataraja et al. | Mar 2014 | A1 |
| 20140075002 | Pradhan et al. | Mar 2014 | A1 |
| 20140098671 | Raleigh et al. | Apr 2014 | A1 |
| 20140129719 | Weber et al. | May 2014 | A1 |
| 20140136908 | Maggiari et al. | May 2014 | A1 |
| 20140146817 | Zhang | May 2014 | A1 |
| 20140172740 | McCormick et al. | Jun 2014 | A1 |
| 20140172939 | McSherry et al. | Jun 2014 | A1 |
| 20140201218 | Catalano et al. | Jul 2014 | A1 |
| 20140208150 | Abuelsaad | Jul 2014 | A1 |
| 20140211661 | Gorkemli et al. | Jul 2014 | A1 |
| 20140241356 | Zhang et al. | Aug 2014 | A1 |
| 20140250220 | Kapadia et al. | Sep 2014 | A1 |
| 20140269435 | McConnell et al. | Sep 2014 | A1 |
| 20140301391 | Krishnan et al. | Oct 2014 | A1 |
| 20140304355 | Kamath et al. | Oct 2014 | A1 |
| 20140337500 | Lee | Nov 2014 | A1 |
| 20140351396 | Stabile et al. | Nov 2014 | A1 |
| 20140372533 | Fu et al. | Dec 2014 | A1 |
| 20150009797 | Koponen et al. | Jan 2015 | A1 |
| 20150009808 | Bejerano et al. | Jan 2015 | A1 |
| 20150016276 | Decusatis et al. | Jan 2015 | A1 |
| 20150063364 | Thakkar et al. | Mar 2015 | A1 |
| 20150067171 | Yum et al. | Mar 2015 | A1 |
| 20150085862 | Song | Mar 2015 | A1 |
| 20150100704 | Davie et al. | Apr 2015 | A1 |
| 20150103842 | Chandrashekhar et al. | Apr 2015 | A1 |
| 20150103843 | Chandrashekhar et al. | Apr 2015 | A1 |
| 20150106804 | Chandrashekhar et al. | Apr 2015 | A1 |
| 20150117216 | Anand et al. | Apr 2015 | A1 |
| 20150117256 | Sabaa et al. | Apr 2015 | A1 |
| 20150154330 | Yachide et al. | Jun 2015 | A1 |
| 20150195126 | Vasseur et al. | Jul 2015 | A1 |
| 20150229641 | Sun et al. | Aug 2015 | A1 |
| 20150237014 | Bansal et al. | Aug 2015 | A1 |
| 20150263946 | Tubaltsev et al. | Sep 2015 | A1 |
| 20150312116 | Taheri et al. | Oct 2015 | A1 |
| 20150312326 | Archer et al. | Oct 2015 | A1 |
| 20150326467 | Fullbright et al. | Nov 2015 | A1 |
| 20160057014 | Thakkar et al. | Feb 2016 | A1 |
| 20160094396 | Chandrashekhar et al. | Mar 2016 | A1 |
| 20160134528 | Lin et al. | May 2016 | A1 |
| 20160173338 | Wolting | Jun 2016 | A1 |
| 20160210209 | Verkaik et al. | Jul 2016 | A1 |
| 20160218918 | Chu et al. | Jul 2016 | A1 |
| 20160226700 | Zhang et al. | Aug 2016 | A1 |
| 20160226762 | Zhang et al. | Aug 2016 | A1 |
| 20160226822 | Zhang et al. | Aug 2016 | A1 |
| 20160294728 | Jain et al. | Oct 2016 | A1 |
| 20160352588 | Subbarayan et al. | Dec 2016 | A1 |
| 20160359705 | Parandehgheibi et al. | Dec 2016 | A1 |
| 20160373530 | Duda | Dec 2016 | A1 |
| 20160380815 | Agarwal et al. | Dec 2016 | A1 |
| 20160380891 | Agarwal et al. | Dec 2016 | A1 |
| 20160380925 | Agarwal et al. | Dec 2016 | A1 |
| 20160380973 | Sullenberger et al. | Dec 2016 | A1 |
| 20170034052 | Chanda et al. | Feb 2017 | A1 |
| 20170034129 | Sawant et al. | Feb 2017 | A1 |
| 20170041347 | Nagaratnam et al. | Feb 2017 | A1 |
| 20170048110 | Wu et al. | Feb 2017 | A1 |
| 20170048130 | Goliya et al. | Feb 2017 | A1 |
| 20170063633 | Goliya et al. | Mar 2017 | A1 |
| 20170063794 | Jain et al. | Mar 2017 | A1 |
| 20170063822 | Jain et al. | Mar 2017 | A1 |
| 20170093636 | Chanda et al. | Mar 2017 | A1 |
| 20170104720 | Bansal et al. | Apr 2017 | A1 |
| 20170126551 | Pfaff et al. | May 2017 | A1 |
| 20170163532 | Tubaltsev et al. | Jun 2017 | A1 |
| 20170163598 | Shen et al. | Jun 2017 | A1 |
| 20170163599 | Shen et al. | Jun 2017 | A1 |
| 20170171055 | Wang et al. | Jun 2017 | A1 |
| 20170222873 | Lee et al. | Aug 2017 | A1 |
| 20170249195 | Sadana et al. | Aug 2017 | A1 |
| 20170250912 | Chu et al. | Aug 2017 | A1 |
| 20170257432 | Fu et al. | Sep 2017 | A1 |
| 20170264483 | Lambeth et al. | Sep 2017 | A1 |
| 20170288981 | Hong et al. | Oct 2017 | A1 |
| 20170310641 | Jiang et al. | Oct 2017 | A1 |
| 20170317954 | Masurekar et al. | Nov 2017 | A1 |
| 20170317969 | Masurekar et al. | Nov 2017 | A1 |
| 20170317971 | Dubey et al. | Nov 2017 | A1 |
| 20170324645 | Johnsen et al. | Nov 2017 | A1 |
| 20170331711 | Duda | Nov 2017 | A1 |
| 20170344444 | Costa-Roberts et al. | Nov 2017 | A1 |
| 20180062880 | Yu et al. | Mar 2018 | A1 |
| 20180062881 | Chandrashekhar et al. | Mar 2018 | A1 |
| 20180062923 | Katrekar et al. | Mar 2018 | A1 |
| 20180062944 | Altman et al. | Mar 2018 | A1 |
| 20180063036 | Chandrashekhar et al. | Mar 2018 | A1 |
| 20180063195 | Nimmagadda et al. | Mar 2018 | A1 |
| 20180123877 | Saxena et al. | May 2018 | A1 |
| 20180123903 | Holla et al. | May 2018 | A1 |
| 20180157537 | Chen et al. | Jun 2018 | A1 |
| 20180234337 | Goliya et al. | Aug 2018 | A1 |
| 20180234459 | Kung et al. | Aug 2018 | A1 |
| 20180309718 | Zuo | Oct 2018 | A1 |
| 20190014040 | Yerrapureddy et al. | Jan 2019 | A1 |
| 20190158537 | Miriyala | May 2019 | A1 |
| 20190190780 | Wang et al. | Jun 2019 | A1 |
| 20190207847 | Agarwal et al. | Jul 2019 | A1 |
| 20190245888 | Martinez et al. | Aug 2019 | A1 |
| 20190260610 | Dubey et al. | Aug 2019 | A1 |
| 20190260630 | Stabile et al. | Aug 2019 | A1 |
| 20190303326 | Desai et al. | Oct 2019 | A1 |
| 20190334765 | Jain et al. | Oct 2019 | A1 |
| 20190342175 | Wan et al. | Nov 2019 | A1 |
| 20190363975 | Djernaes | Nov 2019 | A1 |
| 20190379731 | Johnsen et al. | Dec 2019 | A1 |
| 20200007392 | Goyal | Jan 2020 | A1 |
| 20200021541 | Chanda | Jan 2020 | A1 |
| 20200057669 | Hutcheson et al. | Feb 2020 | A1 |
| 20200076684 | Naveen et al. | Mar 2020 | A1 |
| 20200106744 | Miriyala et al. | Apr 2020 | A1 |
| 20200162325 | Desai et al. | May 2020 | A1 |
| 20200169496 | Goliya et al. | May 2020 | A1 |
| 20200177518 | Jain et al. | Jun 2020 | A1 |
| 20200195607 | Wang et al. | Jun 2020 | A1 |
| 20200257549 | Vlaznev et al. | Aug 2020 | A1 |
| 20200296035 | Agarwal et al. | Sep 2020 | A1 |
| 20200358693 | Rawlins | Nov 2020 | A1 |
| 20200366741 | Kancherla et al. | Nov 2020 | A1 |
| 20200409563 | Parasnis et al. | Dec 2020 | A1 |
| 20210067556 | Tahan | Mar 2021 | A1 |
| 20210117908 | Coles et al. | Apr 2021 | A1 |
| 20210168197 | Jones et al. | Jun 2021 | A1 |
| 20210194729 | Semwal et al. | Jun 2021 | A1 |
| 20210311960 | Rogozinsky et al. | Oct 2021 | A1 |
| 20210314192 | Chandrashekhar et al. | Oct 2021 | A1 |
| 20210314193 | Chandrashekhar et al. | Oct 2021 | A1 |
| 20210314212 | Chandrashekhar et al. | Oct 2021 | A1 |
| 20210314215 | Manzanilla et al. | Oct 2021 | A1 |
| 20210314219 | Gujar et al. | Oct 2021 | A1 |
| 20210314225 | Chandrashekhar et al. | Oct 2021 | A1 |
| 20210314226 | Chandrashekhar et al. | Oct 2021 | A1 |
| 20210314227 | Chandrashekhar et al. | Oct 2021 | A1 |
| 20210314228 | Chandrashekhar et al. | Oct 2021 | A1 |
| 20210314235 | Chandrashekhar et al. | Oct 2021 | A1 |
| 20210314251 | Dubey et al. | Oct 2021 | A1 |
| 20210314256 | Chandrashekhar et al. | Oct 2021 | A1 |
| 20210314257 | Chandrashekhar et al. | Oct 2021 | A1 |
| 20210314258 | Chandrashekhar et al. | Oct 2021 | A1 |
| 20210314289 | Chandrashekhar et al. | Oct 2021 | A1 |
| 20210314291 | Chandrashekhar et al. | Oct 2021 | A1 |
| Number | Date | Country |
|---|---|---|
| 102124456 | Jul 2011 | CN |
| 102681899 | Sep 2012 | CN |
| 103064744 | Apr 2013 | CN |
| 103650433 | Mar 2014 | CN |
| 103747059 | Apr 2014 | CN |
| 103890751 | Jun 2014 | CN |
| 104123189 | Oct 2014 | CN |
| 110061899 | Jul 2019 | CN |
| 1154601 | Nov 2001 | EP |
| 1290856 | Mar 2003 | EP |
| 1635506 | Mar 2006 | EP |
| 1868318 | Dec 2007 | EP |
| 3016331 | May 2016 | EP |
| 3032412 | Jun 2016 | EP |
| 3314831 | May 2018 | EP |
| 3485610 | May 2019 | EP |
| 2010028364 | Mar 2010 | WO |
| 2012113444 | Aug 2012 | WO |
| 2013152716 | Oct 2013 | WO |
| 2014066820 | May 2014 | WO |
| 2015054671 | Apr 2015 | WO |
| 2016161394 | Oct 2016 | WO |
| 2017003881 | Jan 2017 | WO |
| 2018044341 | Mar 2018 | WO |
| 2021206785 | Oct 2021 | WO |
| 2021206786 | Oct 2021 | WO |
| 2021206790 | Oct 2021 | WO |
| Entry |
|---|
| Author Unknown, “Apache Cassandra™ 1.2 Documentation,” Jan. 13, 2013, 201 pages, DataStax. |
| Author Unknown, “OpenFlow Switch Specification, Version 1.1.0 Implemented (Wire Protocol 0x02),” Feb. 28, 2011, 56 pages, Open Networking Foundation. |
| Berde, Pankaj, et al., “ONOS Open Network Operating System An Open-Source Distributed SDN OS,” Dec. 19, 2013, 34 pages. |
| Guo, Yingya, et al., “Traffic Engineering in SDN/OSPF Hybrid Network,” The 22nd IEEE International Conference on Network Protocols (ICNP 2014), Oct. 21-24, 2014, 6 pages, IEEE, The Research Triangle, North Carolina, USA. |
| Hanna, Jeremy, “How ZooKeeper Handles Failure Scenarios,” http://apache.org/hadoop/Zookeeper/Failurescenarios. Dec. 9, 2010,1 page. |
| Heller, Brandon, et al., “The Controller Placement Problem,” Hot Topics in Software Defined Networks, Aug. 13, 2012, 6 pages, Helsinki, Finland. |
| Jin, Xin, et al., “Dynamic Scheduling of Network Updates,” SIGCOMM'14, Aug. 17-22, 2014, 12 pages, ACM, Chicago, IL, USA. |
| Krishnaswamy, Umesh, et al., “ONOS Open Network Operating System—An Experimental Open-Source Distributed SDN OS,” Apr. 16, 2013, 24 pages. |
| Lebresne, Sylvain, “[Release] Apache Cassandra 1.2 released,” Jan. 2, 2013, 1 page. |
| Mahalingham, Mallik, et al., “VXLAN: A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks,” draft-mahalingham-dutt-dcops-vxlan-00.txt Internet Draft, Aug. 26, 2011, 20 pages, Internet Engineering Task Force. |
| Mahalingham, Mallik, et al., “VXLAN: A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks,” draft-mahalingham-dutt-dcops-vxlan-02.txt Internet Draft, Aug. 22, 2012, 20 pages, Internet Engineering Task Force. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,889, filed Jun. 19, 2020, 125 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,891, filed Jun. 19, 2020, 125 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,893, filed Jun. 19, 2020, 126 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,901, filed Jun. 19, 2020, 125 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,902, filed Jun. 19, 2020, 126 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,905, filed Jun. 19, 2020, 126 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,908, filed Jun. 19, 2020, 125 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,913, filed Jun. 19, 2020, 124 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,921, filed Jun. 19, 2020, 43 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,925, filed Jun. 19, 2020, 111 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,929, filed Jun. 19, 2020, 113 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,934, filed Jun. 19, 2020, 112 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,935, filed Jun. 19, 2020, 114 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,942, filed Jun. 19, 2020, 127 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,944, filed Jun. 19, 2020, 128 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,950, filed Jun. 19, 2020, 128 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,955, filed Jun. 19, 2020, 127 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,960, filed Jun. 19, 2020, 128 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,964, filed Jun. 19, 2020, 126 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 16/906,966, filed Jun. 19, 2020, 128 pages, VMware, Inc. |
| Non-Published Commonly Owned Related U.S. Appl. No. 17/103,700 with similar specification, filed Nov. 24, 2020, 80 pages, VMware, Inc. |
| Non-Published Commonly Owned Related U.S. Appl. No. 17/103,704 with similar specification, filed Nov. 24, 2020, 80 pages, VMware, Inc. |
| Non-Published Commonly Owned Related U.S. Appl. No. 17/103,706 with similar specification, filed Nov. 24, 2020, 79 pages, VMware, Inc. |
| Non-Published Commonly Owned Related U.S. Appl. No. 17/103,708 with similar specification, filed Nov. 24, 2020, 79 pages, VMware, Inc. |
| Non-Published Commonly Owned Related International Patent Application PCT/US2021/042117 with similar specification, filed Jul. 17, 2021, 80 pages, VMware, Inc. |
| Non-Published Commonly Owned U.S. Appl. No. 17/391,917, filed Aug. 2, 2021, 51 pages, VMware, Inc. |
| PCT Invitation to Pay Additional Fees for Commonly Owned International Patent Application PCT/US2021/042117, dated Jan. 10, 2022, 10 pages, International Searching Authority (EPO). |
| Number | Date | Country | |
|---|---|---|---|
| 20220103429 A1 | Mar 2022 | US |
