Using CRDs to create externally routable addresses and route records for pods

Abstract

Some embodiments provide an automated method for defining externally routable Pods within a Kubernetes cluster. In some embodiments, the Pod operates in a guest cluster has its own VPC (virtual private cloud) network in a datacenter with several other guest clusters that have their own VPC networks and their own set of managers. In some embodiments, a Pod within a GC can be made externally routable so that it can be directly addressable from an external client outside of the Pod's network by using two new Kubernetes CRDs (custom resource definitions), which are an IPPool CRD and a RouteSet CRD. Examples of such external clients include VMs or Pods in another GC or a supervisor cluster connected to the particular GC through a gateway, or from a machine outside of the network of all of the GCs or SC.

Description

BACKGROUND

With the recent increase in cloud native applications, today there is more demand than ever for fast deployment of on-demand networking for connecting machines that are deployed in software defined datacenters (SDDC). It is desirable to provide auto-deployed networking solutions as many compute-cluster administrators do not have extensive knowledge of networking. However, for administrators who wish to adjust their system's networking, it is desirable to provide such administrators with the ability to configure and customize their network deployments.

BRIEF SUMMARY

Some embodiments provide an automated method for defining externally routable Pods within a Kubernetes cluster. Externally routable Pods are Pods that are directly addressable from an external client outside of the Pod's network. In some embodiments, the Pod (1) resides in a guest cluster that is managed by its own set of one or more managers, and (2) connects to the external network through one or more intervening gateway routers that are configured by a set of one or more controllers of a supervisor cluster. In some embodiments, the Pod's guest cluster has its own VPC (virtual private cloud) network in a datacenter with several other guest clusters that have their own VPC networks and their own set of managers.

In some embodiments, a Pod within a GC can be made externally routable so that it can be directly addressable from an external client outside of the Pod's network by using two new Kubernetes CRDs (custom resource definitions), which are an IPPool CRD and a RouteSet CRD. Examples of such external clients include VMs or Pods in another GC or a supervisor cluster connected to the particular GC through a gateway, or from a machine outside of the network of all of the GCs or SC.

At a control plane of a supervisor second cluster (SC), the method of some embodiments (1) receives, from a server, notification that an IPPool custom resource instance has been created in response to an addition of the worker node to the first cluster, (2) creates a set of one or more externally routable IP subnets for the IPPool, and (3) provides, to the server, the set of externally routable IP subnets for the server to use to update IPPool customer resource instance. The method then provides a status update to a manager in the Pod's GC, which this manager then uses to direct a Kubernetes manager to update a CIDR (Classless Inter-Domain Routing) of the Pod on the worker node.

Before the SC control plane receives the notification, the Kubernetes manager in some embodiments sends to the manager of the Pod's GC a notification that the worker node has been added, so that this manager can then send an order to the server to create an IPPool custom resource based on an IPPool CRD. The server also receives from the manager of the Pod's GC, a request to create a RouteSet custom resource based on the RouteSet CRD. This manager then creates the RouteSet CR and sends the SC control plane a notification that a RouteSet CR has been added.

The SC control plane, in turn, creates in response one or more static routes to deploy at one or more intervening gateway routers between the Pod in the first cluster of the VPC and a network external to a first network for the first VPC. As mentioned above, the external network can be the SC network, the network of another GC, or a network residing outside of the datacenter on the other side of its gateways.

The preceding Summary is intended to serve as a brief introduction to some embodiments of the invention. It is not meant to be an introduction or overview of all inventive subject matter disclosed in this document. The Detailed Description that follows and the Drawings that are referred to in the Detailed Description will further describe the embodiments described in the Summary as well as other embodiments. Accordingly, to understand all the embodiments described by this document, a full review of the Summary, Detailed Description, the Drawings and the Claims is needed. Moreover, the claimed subject matters are not to be limited by the illustrative details in the Summary, Detailed Description, and Drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appended claims. However, for purposes of explanation, several embodiments of the invention are set forth in the following figures.

FIG. 1 conceptually illustrates an example of externally routable Pods that are defined in a guest cluster.

FIG. 2 conceptually illustrates an example of an IPPool CRD and CR that is specified in some embodiments.

FIG. 3 conceptually illustrates a data flow diagram that illustrates how an IP pool of externally routable IP addresses is created for a worker node.

FIG. 4 conceptually illustrates an example of a routeset1 CR that is defined based on this CRD.

FIG. 5 conceptually illustrates a data flow diagram for creating a RouteSet CR for a worker node or a Pod that is instantiated.

FIG. 6 conceptually illustrates an example of an IPPool API.

FIG. 7 conceptually illustrates an example of an API to create a static route on a T1 router.

FIG. 8 conceptually illustrates a computer system with which some embodiments of the invention are implemented.

DETAILED DESCRIPTION

In the following detailed description of the invention, numerous details, examples, and embodiments of the invention are set forth and described. However, it will be clear and apparent to one skilled in the art that the invention is not limited to the embodiments set forth and that the invention may be practiced without some of the specific details and examples discussed.

Some embodiments provide an automated method for defining externally routable Pods within a Kubernetes cluster. Externally routable Pods are Pods that are directly addressable from an external client outside of the Pod's network. FIG. 1 illustrates an example of externally routable Pods that are defined in a guest cluster. This figure illustrates several guest clusters 105, a supervisor cluster 110 and two tiers (T0 and T1) of gateway routers 115 and 120 that connect these clusters. In this example, each GC has its own tier 1 (T1) router, the SC has its own T1 router and all the T1 routers connect to one T0 router.

In other embodiments, other gateway router architectures are used. For instance, in some embodiments, one tier 1 (T1) gateway router connects the different guest clusters (GCs) and supervisor cluster (SC). In still other embodiments, the supervisor cluster has one tier 1 router for each of its namespaces and each guest cluster connects to the SC T1 router for the namespace under with the guest cluster operates.

In some embodiments, each guest or supervisor cluster has its own compute and network managers/controllers. In addition, the supervisor cluster has managers and controllers for configuring the gateways between the GCs and between the GCs/SC and external networks on the other side of the Tier 0 gateway router 120. In some embodiments, the SC 110 also has responsibility of assigning for a worker node in a GC a set of IP addresses (e.g., an IP CIDR) that allows Pods on this worker node to be externally routable from machines outside of the GC's network (e.g., from machines outside of the GC's virtual private cloud, VPC).

In some embodiments, a Pod within a GC can be made externally routable so that it can be directly addressable from an external client outside of the Pod's network by using two new Kubernetes CRDs, which are an IPPool CRD and a RouteSet CRD. Examples of such external clients include VMs or Pods in another GC or a supervisor cluster connected to the particular GC through a gateway, or from a machine outside of the network of all of the GCs or SC.

In some embodiments, the scope that supports routable Pod CIDRs for a guest cluster includes (1) inside the supervisor cluster (T0 router domain), such that the Pod traffic is routable within the SC TO domain, and can be routed across SC Namespaces, and (2) global scope across SC, such that the Pod traffic can be routed to an external network outside of the SC TO.

To allow a GC Pod be routable from outside of the GC network, the SC controllers as well as the SC-adapter controllers have to be configured as follows. To allow traffic to be sent to and received from the externally routable Pod, the SC controller sets one or more static routes for the Pod network are added in some embodiments on one or more intervening gateways (e.g., Tier-1 router that is connected with GC node's segment). Some embodiments define service rules that are enforced at the gateways with the static routes (e.g., SC T1 router) on the packets that are sent to or received from the externally routable Pod.

Also, for different routable scope, static route advertisement on Tier-1 router and Tier-0 router are different in some embodiments. For instance, inside Supervisor Cluster TO domain, some embodiments enable static route advertisement on T1 router and enable route re-distribution (Tier-1 static routes) on T0 router, but do not advertise static route to T0 BGP neighbors. For the global scope (outside of TO domain), some embodiments enable static route advertisement on T1 and enable route re-distribution (Tier-1 static routes) on T0 router, and advertise static route to T0 BGP neighbors.

In some embodiments, the GC control plane does not talk directly with the SC control plane. Instead, in these embodiments, a network container plugin (referred to as NCP) on the SC receives information and communicates with the SC control plane to initiate the configuration changes to the SC control plane. One example of NCP is used in VMware's vSphere system. In this system, NCP is a control plane module that run as a Kubernetes pod in the Supervisor cluster control plane. NCP watches for network resources added through the Kubernetes API and orchestrates the creation of corresponding objects in NSX environment of vSphere.

The overall workflow for using the IPPool and RouteSet CRDs is as follows. In some embodiments, the SC has a Network Profile that includes a Namespace level routable CIDR configuration, which could be used to assign IP addresses to a GC Pod. In some embodiments, there is a NamespaceNetwork CR (custom resource, which is an instance of CRD) created based on the Network Profile configurations. The NCP will allocate subnets from NamespaceCIDR in NamespaceNetwork CR.

During the Guest Cluster creation, some embodiments add routing properties in a KubernetesCluster CRD to configure routable Pods. Also, to add externally routable Pod to a GC, some embodiments use the two new CRDs, RouteSet CRD and IPPool CRD, to assign one or more IP subnets to the worker node and to define statistic routes for the Pod. In some embodiments, the GCCP service creates the IPPool CRD to request the GC Node's Pod CIDR. The NCP will then allocate one or more subnets from the SC Namespace CIDR.

The RouteSet CRD is then used to configure static routes on the SC Namespace network for the GC Pod CIDRs. The GC cloud provider (GCCP) service in some embodiments creates a RouteSet CR for each GC Node in the Supervisor Namespace, with the node.PodCIDR. The NCP will create a static route on the SC Namespace T1. The GCCP service is a Kubernetes control plane service that interacts with cloud provider layer (e.g., Amazon Web Services, Google Cloud, vSphere, etc.) to configure cloud services (e.g., load balancer, route table, routable IP allocation, storage volume, etc.) for a Kubernetes cluster. When the SC is operated by vSphere, the cloud provider layer is the SC API server. The SC API server in some embodiments is the WCP supervisor Kubernetes API server.

In some embodiments, the GCCP service will create a single IPPool CR for each guest cluster. After the NCP allocates subnets from the SC Namespace IPPool, the GCCP service will set it to Node.podCIDR. The GCCP service will set the Node name and subnet prefix and IP family in IPPool CR, while the NCP will update the CIDR to IPPool CR. For GC Node scale up/down in runtime, the GCCP service will create/delete IPPool CR accordingly, and then update the Node.podCIDR.

FIG. 2 illustrates an example of an IPPool CRD 200 and CR 205 that is specified in some embodiments. As shown, the CRD has a scope and name fields, identifying the namespace scope and several names associated with the CRD. The CR 205 has a metadata attribute containing its name, plus spec and status attributes. As shown, the subnet attributes in this example specify two subnets each with a 24 value prefix length and IPv4 type in this instance. The status attribute specifies two subnets, 192.168.100.0/24 and 192.168.101.0/24, as well as several condition values.

FIG. 3 presents a data flow diagram that illustrates how an IP pool of externally routable IP addresses is created for a worker node, so that IP addresses from this pool can be assigned to Pods operating on the worker node. As shown, the process starts when a GCCP service 304 receives notification from a GC API server 302 (e.g., a Kubernetes API server) that a worker node has been added to a guest cluster.

The GCCP service 304 then directs the SC API server 306 to create an IPPool custom resource. The SC API server 306 creates this resource from the IPPool CRD, and then notifies the NCP 308 that the IPPool CR has been added. The NCP then directs the SC controller 310 (e.g., an NSX-T controller cluster) to create one or more IP subnets that are specified in the created CR's specification. The NCP 308 then collects the realized state for the created IP subnet(s) from the SC controller 310. In some embodiments, the realized state includes the status values, such as the status values shown in FIG. 2, which include the created subnet values as well as the condition attributes.

The NCP 308 then provides the collected realized state information to the SC API Server 306, which then passes the creation of the IPPool CR and its state to the GCCP service 304. This manager then uses the received information to update its records, and to also direct the GC API Server 302 to update the Node.podCIDR. In this manner, the GCCP service 304 in some embodiments creates a single IPPool CR for each guest cluster, and after the NCP 308 allocates subnets from the SC Namespace IPPool for this CR, the GCCP service will set it to Node.podCIDR. The GCCP service also sets the Node name and subnet prefix and IP family in IPPool CR, while the NCP defines CIDR for IPPool CR.

When the Pod is created on a worker node and this Pod needs to have an externally routable IP address, an IP address is assigned to this Pod from the externally-routable IP pool that was created for this Pod through the IPPool CRD. To enable packets to reach this Pod from an external network (e.g., from a SC machine, another GC machine or a machine on the other side of the T0 router in FIG. 1), one or more static routes need to be defined for this Pod in the intervening routers.

To define such static routes, some embodiments use the RouteSet CRD 405 that is illustrated in FIG. 4. This figure also illustrates an example of a routeset1 CR 410 that is defined based on this CRD. As shown, the CRD 405 includes kind, metadata and spec attributes. The spec attribute includes group, version, scope and name attributes. The routeset1 CR 410 includes kind, metadata, spec and status attributes. The spec attribute specifies that each route has a destination that is the podCIDR, a target that is the IP addresses of the pod's worker node (i.e., nodeIPAddress) and a route name. The status attribute then provides the values for the destination, target and name attributes (e.g., as specified by the SC controller). The destination and target IP parameters are specified because when a Pod is created on a worker node, for a packet to reach the Pod, the packet first needs to be routed the worker node and then the Pod.

FIG. 5 presents a data flow diagram for creating a RouteSet CR for a worker node or a Pod that is instantiated. As shown, the process starts when the GCCP service 304 receives notification from the GC API Server 302 that a worker node with a routable Pod has been added to a GC managed by the GCCP service 304. In some embodiments, this process also starts when the GCCP service receives notification that a new routable Pod has been added to a previously deployed worker node.

The GCCP service 304 then directs the SC API Server 306 to create a RouteSet CR. The API server 306 creates this CR, and also informs the NCP 308 of the creation of this CR. The NCP 308 then directs the SC controller 310 to create for the RouteSet CR instance one or more static routes, such as one or more static routes for the intervening TO or T1 routers between the externally routable Pod's GC and the external machines that needs to reach the Pod (e.g., the machines in the other GCs, SC, or the machines in the external networks on the other side of the T0 router). The SC controller 310 in some embodiments also defines middlebox service rules and distributes these service rules to the gateways with the static routes (e.g., SC T1 router) and/or to middlebox service nodes (e.g., service VMs, service engines or service appliances) associated with these gateways, so that these gateways and/or service nodes can enforce these middlebox service rules on the packets that are sent to or received from the externally routable Pod.

The NCP 308 retrieves the status information for the RouteSet CR from the SC controller 310, and then provides this information to the SC API server 306, which then updates its created RouteSet CR status attributes to reflect this state information. Based on the route data that the SC controller receives from the NCP and provides to the NCP, the SC controller then creates one or more static routes in the intervening T0 and T1 routers between the externally routable Pod's GC and the external machines that needs to reach the Pod.

In some embodiments, the IPPool CR and RouteSet CR can be created only if routable-pod is enabled for the GC (e.g., for a GuestCluster CRD). As each GC has an SC ServiceAccount, the GCCP service in some embodiments creates a Role with permission to get/list/watch/create/delete/update the IPPool CRD and RouteSet CRD, and then bind the ServiceAccount with the Role. The GCCP service then authenticates to list/create/delete the IPPool CR and RouteSet CR in SC by using this ServiceAccount.

In some embodiments, the routes interface of a route controller in an upstream Kubernetes CP manager has the following structure

// Route is a representation of an advanced routing rule.
type Route struct {
// Name is the name of the routing rule in the cloud-provider.
Name string
// TargetNode is the NodeName of the target instance.
TargetNode types.NodeName
// DestinationCIDR is the CIDR format IP range that this routing rule
// applies to.
DestinationCIDR string
// Blackhole is set to true if this is a blackhole route
// The node controller will delete the route if it is in the managed range.
Blackhole bool
}
// Routes is an abstract, pluggable interface for advanced routing rules.
type Routes interface {
// ListRoutes lists all managed routes that belong to the specified clusterName
ListRoutes(ctx context.Context, clusterName string) ([ ]*Route, error)
// CreateRoute creates the described managed route
// route.Name will be ignored, although the cloud-provider may use nameHint
// to create a more user-meaningful name.
CreateRoute(ctx context.Context, clusterName string, nameHint string, route
*Route) error
// DeleteRoute deletes the specified managed route
// Route should be as returned by ListRoutes
DeleteRoute(ctx context.Context, clusterName string, route *Route) error
}

In some embodiments, the GCCP service gets RouteSet CR list from SC namespace, filters RouteSet CR that belong to this cluster, and only sends a return to route controller if CRD status “ready” is true. Also, the RouteSet CRD definition in some embodiments is added in GCCP service, so the GCCP service can create the RouteSet CR through API client interfaces. For NotReady RouteSet CR, the GCCP service in some embodiments would not re-create them but wait for the NCP to realize. In some embodiments, the GCCP service in some embodiments will request deletion of the RouteSet CR when there is a Node deleted, while the NC will request deletion of static routes for a GC when the GC is deleted.

The table below provides the mapping between RouteSet CR and Route struct in route controller in some embodiments.

Route struct	RouteSet CR	Notes
name	metadata.name	nameHint is node UID.
		We could set <nameHint>_<podCIDR> as the CR
		name
TargetNode	name	TargetNode is name of route
	target	node IP address
		IP family of node address and podCIDR should be the
		same.
		There are 3 types of address: Hostname, ExternalIP or
		InternalIP, will choose node InternalIP address first,
		then ExternalIP address.
DestinationCIDR	destination	node.spec.podCIDR (single stack) or
		node.spec.podCIDRs (dual stack)
		RouteSet CR is per Kubernetes cluster Node

In some embodiments, NCP 308 creates IP blocks based on a NamespaceNetwork CR. After the NCP gets an IPPool CR event, the NCP creates the IP pool from the IP blocks. When the NamespaceNetwork CIDR does not exist, the NCP will create the IP pool from the SC Pod CIDR. FIG. 6 presents an example of an IP Pool API.

In some embodiments, the NCP 308 watches RouteSet CR events, and creates static routes on GC's T1 router. The GCCP service 304 in some embodiments creates RouteSet CR with a label. The NCP will check if route destination belongs to the SC namespace subnetCIDR or cluster container IP block. To manage static routes, the NCP 308 creates a RouteSet CR watcher, creates a RouteSet CR controller to process events, and creates RouteSet CR store and SC static route store.

Listed below are examples of APIs used in some embodiments to create, check, delete, and list static routes, while FIG. 7 illustrates an example of an API to create a static route on a T1 router.

// Create static route
PATCH /policy/api/v1/infra/tier-1s/<tier-1-id>/static-routes/<route-id>
// Check static route realization status
GET /policy/api/v1/infra/realized-state/realized-entities\?intent_path\=/infra/tier-
1s/<tier-1-id>/static-routes
/<route-id>
// Delete static route
DELETE /policy/api/v1/infra/tier-1s/<tier-1-id>/static-routes/<route-id>
// List static routes by searching vnet-uid tag
GET
/policy/api/v1/search/query\?query\=resource_type:StaticRoutes+AND+tags.scope:ncp/vnet_uid
+AND+tags.tag:
<vnet_uid>

The above-described approach of using CRDs to deploy externally routable Pods has several advantages. It allows cloud providers and network administrators a very simple way of deploying and supporting Pods with externally routable IP Addresses. Such Pods can be directly addressed from a client external to the Guest Cluster. Also, the Pod IPs are preserved for traffic to external network, so external network services and servers can identify the source Pods and apply policies based on the Pod IPs. When Pods reach out to shared services, the source IP address is going to be the Pod IP address instead of the Nat'ed IP address.

Moreover, the GC data plane will not require encapsulation and the overhead associated with encapsulation protocols for routing to and from these Pods from and to the external networks. The Pod traffic will also be transparent to the SC fabric enabling the potential to deploy advanced SC services on the traffic. This approach also allows administrators to audit and trace egress requests from the Pods to common shared services. In addition to audit and traceability, there are deployment that have legacy applications that do not support connections over network addressed translated addresses.

Many of the above-described features and applications are implemented as software processes that are specified as a set of instructions recorded on a computer readable storage medium (also referred to as computer readable medium). When these instructions are executed by one or more processing unit(s) (e.g., one or more processors, cores of processors, or other processing units), they cause the processing unit(s) to perform the actions indicated in the instructions. Examples of computer readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc. The computer readable media does not include carrier waves and electronic signals passing wirelessly or over wired connections.

In this specification, the term “software” is meant to include firmware residing in read-only memory or applications stored in magnetic storage, which can be read into memory for processing by a processor. Also, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while remaining distinct software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that together implement a software invention described here is within the scope of the invention. In some embodiments, the software programs, when installed to operate on one or more electronic systems, define one or more specific machine implementations that execute and perform the operations of the software programs.

FIG. 8 conceptually illustrates a computer system 800 with which some embodiments of the invention are implemented. The computer system 800 can be used to implement any of the above-described computers and servers. As such, it can be used to execute any of the above described processes. This computer system includes various types of non-transitory machine readable media and interfaces for various other types of machine readable media. Computer system 800 includes a bus 805, processing unit(s) 810, a system memory 825, a read-only memory 830, a permanent storage device 835, input devices 840, and output devices 845.

The bus 805 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the computer system 800. For instance, the bus 805 communicatively connects the processing unit(s) 810 with the read-only memory 830, the system memory 825, and the permanent storage device 835.

From these various memory units, the processing unit(s) 810 retrieve instructions to execute and data to process in order to execute the processes of the invention. The processing unit(s) may be a single processor or a multi-core processor in different embodiments. The read-only-memory (ROM) 830 stores static data and instructions that are needed by the processing unit(s) 810 and other modules of the computer system. The permanent storage device 835, on the other hand, is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when the computer system 800 is off. Some embodiments of the invention use a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) as the permanent storage device 835.

Other embodiments use a removable storage device (such as a flash drive, etc.) as the permanent storage device. Like the permanent storage device 835, the system memory 825 is a read-and-write memory device. However, unlike storage device 835, the system memory is a volatile read-and-write memory, such a random access memory. The system memory stores some of the instructions and data that the processor needs at runtime. In some embodiments, the invention's processes are stored in the system memory 825, the permanent storage device 835, and/or the read-only memory 830. From these various memory units, the processing unit(s) 810 retrieve instructions to execute and data to process in order to execute the processes of some embodiments.

The bus 805 also connects to the input and output devices 840 and 845. The input devices enable the user to communicate information and select commands to the computer system. The input devices 840 include alphanumeric keyboards and pointing devices (also called “cursor control devices”). The output devices 845 display images generated by the computer system. The output devices include printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD). Some embodiments include devices such as a touchscreen that function as both input and output devices.

Finally, as shown in FIG. 8, bus 805 also couples computer system 800 to a network 865 through a network adapter (not shown). In this manner, the computer can be a part of a network of computers (such as a local area network (“LAN”), a wide area network (“WAN”), or an Intranet, or a network of networks, such as the Internet. Any or all components of computer system 800 may be used in conjunction with the invention.

Some embodiments include electronic components, such as microprocessors, storage and memory that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a variety of recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and/or solid state hard drives, read-only and recordable Blu-Ray® discs, ultra-density optical discs, and any other optical or magnetic media. The computer-readable media may store a computer program that is executable by at least one processing unit and includes sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as is produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter.

While the above discussion primarily refers to microprocessor or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions that are stored on the circuit itself.

As used in this specification, the terms “computer”, “server”, “processor”, and “memory” all refer to electronic or other technological devices. These terms exclude people or groups of people. For the purposes of the specification, the terms display or displaying means displaying on an electronic device. As used in this specification, the terms “computer readable medium,” “computer readable media,” and “machine readable medium” are entirely restricted to tangible, physical objects that store information in a form that is readable by a computer. These terms exclude any wireless signals, wired download signals, and any other ephemeral or transitory signals.

While the invention has been described with reference to numerous specific details, one of ordinary skill in the art will recognize that the invention can be embodied in other specific forms without departing from the spirit of the invention. Thus, one of ordinary skill in the art would understand that the invention is not to be limited by the foregoing illustrative details, but rather is to be defined by the appended claims.

Claims

1. A method of deploying an externally routable Pod on a worker node in a first cluster deployed in a first virtual private cloud, the method comprising: at a control plane of a supervisor second cluster (SC): receiving, from a server, notification that an IP Pool custom resource instance has been created in response to an addition of the worker node to the first cluster;creating a set of one or more externally routable IP subnets for the IP pool; andproviding, to the server, the set of externally routable IP subnets for the server to use to update IP Pool customer resource instance.

2. The method of claim 1 further comprising: providing a status update to a manager of the first cluster for the manager to direct a Kubernetes manager to update a routing record for the Pod on the worker node.

3. The method of claim 2, wherein the routing record is a CIDR (Classless Inter-Domain Routing) record.

4. The method of claim 2 further comprising before the notification is received: from the Kubernetes manager, receiving, at the first cluster manager, a notification that the worker node has been added; andfrom the first cluster manager, receiving, at the server, a request to create an IP Pool custom resource based on an IP Pool CRD (custom resource definition).

5. The method of claim 4 further comprising: from the first cluster manager, receiving, at the server, a request to create a RouteSet custom resource based on a RouteSet CRD;from the first cluster manager, receiving, at the SC control plane, a notification that a RouteSet CR has been added; andcreating, at the SC control plane, one or more static routes to deploy at one or more intervening gateway routers between the Pod in the first cluster of the VPC and a network external to a first network for the first VPC.

6. The method of claim 5, wherein the external network is the network of the supervisor cluster.

7. The method of claim 5, wherein the external network is the network of a second VPC that connects to the first VPC and the SC through one or more gateway routers.

8. The method of claim 7, wherein each VPC has a set of managers to manage compute and network resources in each VPC, and the SC has a set of managers to manage compute and network resources in the SC as well as configure the gateway routers to allow traffic to reach said Pod from outside of the first network of the first VPC.

9. The method of claim 5 further comprising: specifying a middlebox service rule to perform a middlebox service operation on packets sent to the externally routable Pod from outside of the Pod's network; anddistributing the middlebox service rule to a service node associated with an intervening gateway, said service node enforcing the service rule on said packets.

10. The method of claim 5 further comprising: specifying a middlebox service rule to perform a middlebox service operation on packets sent by the externally routable Pod to a machine outside of the Pod's network; anddistributing the middlebox service rule to a service node associated with an intervening gateway, said service node enforcing the service rule on said packets.

11. A non-transitory machine readable medium storing a program for execution by a set of processing units, the program for deploying an externally routable Pod on a worker node in a first cluster deployed in a first virtual private cloud, the program comprising sets of instructions for: at a control plane of a supervisor second cluster (SC): receiving, from a server, notification that an IP Pool custom resource instance has been created in response to an addition of the worker node to the first cluster;creating a set of one or more externally routable IP subnets for the IP pool; andproviding, to the server, the set of externally routable IP subnets for the server to use to update IP Pool customer resource instance.

12. The non-transitory machine readable medium of claim 11, the program further comprising a set of instructions for: providing a status update to a manager of the first cluster for the manager to direct a Kubernetes manager to update a routing record for the Pod on the worker node.

13. The non-transitory machine readable medium of claim 12, wherein the routing record is a CIDR (Classless Inter-Domain Routing) record.

14. The non-transitory machine readable medium of claim 12, the program further comprising, before the notification is received, sets of instructions for: from the Kubernetes manager, receiving, at the first cluster manager, a notification that the worker node has been added; andfrom the first cluster manager, receiving, at the server, a request to create an IP Pool custom resource based on an IP Pool CRD (custom resource definition).

15. The non-transitory machine readable medium of claim 14, the program further comprising sets of instructions for: from the first cluster manager, receiving, at the server, a request to create a RouteSet custom resource based on a RouteSet CRD;from the first cluster manager, receiving, at the SC control plane, a notification that a RouteSet CR has been added; andcreating, at the SC control plane, one or more static routes to deploy at one or more intervening gateway routers between the Pod in the first cluster of the VPC and a network external to a first network for the first VPC.

16. The non-transitory machine readable medium of claim 15, wherein the external network is the network of the supervisor cluster.

17. The non-transitory machine readable medium of claim 15, wherein the external network is the network of a second VPC that connects to the first VPC and the SC through one or more gateway routers.

18. The non-transitory machine readable medium of claim 7, wherein each VPC has a set of managers to manage compute and network resources in each VPC, and the SC has a set of managers to manage compute and network resources in the SC as well as configure the gateway routers to allow traffic to reach said Pod from outside of the first network of the first VPC.

19. The non-transitory machine readable medium of claim 15, the program further comprising sets of instructions for: specifying a middlebox service rule to perform a middlebox service operation on packets sent to the externally routable Pod from outside of the Pod's network; anddistributing the middlebox service rule to a service node associated with an intervening gateway, said service node enforcing the service rule on said packets.

20. The non-transitory machine readable medium of claim 15, the program further comprising sets of instructions for: specifying a middlebox service rule to perform a middlebox service operation on packets sent by the externally routable Pod to a machine outside of the Pod's network; anddistributing the middlebox service rule to a service node associated with an intervening gateway, said service node enforcing the service rule on said packets.