Patent Application
20220121543
Examples described herein relate to distributed software systems. Examples of cluster orchestration services that store key values with a higher software layer, such as a container orchestration service are described.
Distributed computing systems generally include multiple computing nodes which cooperate together to provide computing services. Frequently, multiple computing nodes may each run instances of a particular service—and the instances of the service cooperate together to perform the service as a whole.
Distributed systems may pose a variety of complexities. Coordination may be needed between the computing nodes to ensure services are accurately performed and that the system as a whole is resilient to failure of one or more of the computing nodes.
Moreover, software stacks may be utilized on computing nodes, including on computing nodes of distributed systems. Each layer of the software stack may perform designated functions, and may interact with other layers in the stack to provide inputs to a next layer, or receive outputs from a previous layer. In this manner, the layers may be somewhat modular.
Coordinating the interaction between layers of a software stack or services on a node as needed may accordingly be complex but important to ensuring efficient, accurate, and reliable operation of a distributed system.
Examples described herein relate to containerized clustered computing systems which may be used to provide a platform as a service (PaaS) environment. Examples of clustered computing systems described herein may manage metadata used to administer the cluster in a manner which improves the reliability and efficiency of cluster operations in some examples. In particular, key values (e.g., metadata) used by cluster orchestrators to manage computing node clusters may be stored in a different (e.g., higher) software layer than that of the cluster orchestrator. For example, a container orchestrator (e.g., Kubernetes) may be used to store and manage the storage of key values used to manage a cluster of computing nodes. The use of a higher software layer to manage this key value storage may provide a higher availability of the key value stores and greater resiliency to node failure.
A cluster orchestrator may be used to manage a cluster of computing nodes. A container orchestrator may be used to manage containers used on one or more of the computing nodes. The container orchestrator may be provided in a different software layer, a higher software layer, than the cluster orchestrator. The clustered system may be configured (e.g., setup or installed) by initiating a bootstrap node including a first cluster infrastructure orchestrator and first container orchestrator. Cluster metadata may be migrated (e.g., copied) to a key value store of the first container orchestrator (e.g., managed by the first container orchestrator). A second node in the cluster may be configured (e.g., setup or installed) by configuring a second cluster infrastructure orchestrator of the second node with a dependency on the first container orchestrator of the bootstrap node. The dependency may be created, for example, because the cluster infrastructure orchestrator may utilize metadata (which may be referred to as cluster metadata) stored by the container orchestrator in order to complete setup (e.g., for the second computing node to join the cluster).
Multi-cloud platform as a service systems (PaaS) may utilize computing clusters, including computing clusters utilizing containerized systems. Configuration, updates, and other tasks associated with such computing clusters may be complicated by installation of both cluster infrastructure orchestrators and container orchestrators at each node of the cluster. For example, where the cluster infrastructure orchestrator is located below a container orchestrator, configuration or upgrade of new nodes in the cluster may involve storing a key value store used by the cluster infrastructure orchestrator on each computing node at the cluster infrastructure orchestrator layer. Repeating the task of creating the key value store may increase complexity of cluster configuration and increase complexity of recovery of the cluster in case of node failure. For example, where a node of the cluster is rebooted, reconfigured, updated, or newly added to the cluster, a key value store is newly added to the cluster infrastructure orchestrator of the node and the key value store is updated at each of the cluster nodes. It may be cumbersome to manage this key value storage across the cluster.
Creating the key value store at a container orchestrator layer of a node of the cluster may reduce complexity of configuration, upgrades, and failure recovery of the cluster. The container orchestrator may, for example, provide resilient storage of data such that high availability of the key value store may be achieved across the cluster. In some examples, during initialization of a bootstrap node, a key value store may be moved from the cluster infrastructure orchestrator to the container orchestrator (which may include simply storing the key value store in the container orchestrator initially). As new nodes are initialized and added to the cluster, cluster infrastructure orchestrators on subsequent nodes may be configured with a dependency on the container orchestrator of the bootstrap node, such that instead of creating a key value store at each node, the cluster infrastructure orchestrators may reference (e.g., access) the key value store created at the bootstrap node. During rebooting of a node, expansion of the cluster, upgrades, or other tasks which may include accessing and/or updating the cluster key value store, complexity may be reduced by the ability to update a shared cluster key value store, improving operation and user experience for services provided using containerized computing clusters.
Various embodiments of the present disclosure will be explained below in detail with reference to the accompanying drawings. The detailed description includes sufficient detail to enable those skilled in the art to practice the embodiments of the disclosure. Other embodiments may be utilized, and structural, logical and electrical changes may be made without departing from the scope of the present disclosure. The various embodiments disclosed herein are not necessarily mutually exclusive, as some disclosed embodiments can be combined with one or more other disclosed embodiments to form new embodiments.
The network 110 may include any type of network capable of routing data transmissions from one network device (e.g., of the computing cluster service domains 112, the bare metal system service domain(s) 120, the central computing system 106, and/or the cloud computing system service domain(s) 130) to another. For example, the network 110 may include a local area network (LAN), wide area network (WAN), intranet, or a combination thereof. The network 110 may include a wired network, a wireless network, or a combination thereof.
Each of the computing cluster service domains 112 may be hosted on a respective computing cluster platform having multiple computing nodes (e.g., each with one or more processor units, volatile and/or non-volatile memory, communication or networking hardware, input/output devices, or any combination thereof) and may be configured to host a respective PaaS software stack 114. Each of the bare metal system service domain(s) 120 may be hosted on a respective bare metal computing platform (e.g., each with one or more processor units, volatile and/or non-volatile memory, communication or networking hardware, input/output devices, or any combination thereof) and may be configured to host a respective PaaS software stack 122. Each of the cloud computing system service domain(s) 130 may be hosted on a respective public or private cloud computing platform (e.g., each including one or more data centers with a plurality of computing nodes or servers having processor units, volatile and/or non-volatile memory, communication or networking hardware, input/output devices, or any combination thereof) and may be configured to host a respective PaaS software stack 132. “Computing platform” as used herein may refer to any one or more of a computing cluster platform, a bare metal system platform, or a cloud computing platform. “Service domain” as used herein may refer to any of the computing cluster service domains 112, the bare metal system service domain(s) 120, or the cloud computing system service domain(s) 130. The PaaS software stacks (e.g., any of the PaaS software stack 114, the PaaS software stack 122, and/or the Paas software stack 132) may include platform-specific software configured to operate on the respective system. The software may include instructions that are stored on a computer readable medium (e.g., memory, disks, etc.) that are executable by one or more processor units (e.g., central processor units (CPUs), graphic processor units (GPUs), tensor processing units (TPUs), hardware accelerators, video processing units (VPUs), etc.) to perform functions, methods, etc., described herein. In this manner, a service domain generally refers to a one or more services which may be installed on (e.g., hosted by) a particular computing platform. The service domain may contain abstracted versions of services that may be configured as needed to run on the particular computing platform on which the service domain will be installed. In this manner, service domains may be present on any of a variety of computing platforms and nonetheless form a distributed computing system across platforms—multiple service domains may, for example, include instances of a same service which may in some examples communicate with other instances of the service across computing platforms to provide a service by the system as a whole. In some examples, centralized management of multiple (e.g., all) instances of a service may be provided, even across the varied computing platforms on which the service domains are instantiated.
The data sources 118, 126, and 136 may each include one or more devices or repositories configured to receive, store, provide, generate, etc., respective source data. The data sources may include input/output devices (e.g., sensors (e.g., electrical, temperature, matter flow, movement, position, biometric data, or any other type of sensor), cameras, transducers, any type of RF receiver, or any other type of device configured to receive and/or generate source data), enterprise or custom databases, a data lake (e.g., a large capacity data storage system that holds raw data) or any other source of data consumed, retrieved, stored, or generated by the service domains. The service domain construct may allow a customer to deploy applications to locations proximate relevant data, in some examples. In some examples, the service domain construct may allow a customer to deploy applications to computing platforms that have a particular computing resource (e.g., hardware or software configuration) and/or based on computing resource capacity.
In some examples, various components of the system may need access to other cloud services 128. To facilitate communication with the other cloud services 128, the data pipelines of the PaaS software stacks may be configured to provide interfaces between applications hosted on one or more of the service domains 112, 120, or 130 and the other cloud services 128 via the network 110. In some examples, the data pipeline(s) 116, 124, and/or 134 hosted on any of the PaaS software stacks 114, 122, and/or 132, respectively, may be configured to provide data from the other cloud services 128 to applications hosted on one or more of the service domains 112, 120, or 130 to aggregate, transform, store, analyze, etc., the data.
Each of the PaaS software stacks may include one or more applications, data pipelines, ML models, containers, data services, etc., or any combination thereof (e.g., applications). The applications may be configured to receive, process/transform, and output data from and to other applications. The applications may be configured to process respective received data based on respective algorithms or functions to provide transformed data. At least some of the applications may be dependent on availability of supporting services to execute, such as communication services, runtime services, read-write data services, ML inference services, container management services, etc., or any combination thereof.
The data pipelines 116, 124, and/or 134 may provide a conduit through which data can be passed (e.g., provided and/or received) between applications hosted in the PaaS software stack, as well as a conduit through which data can be passed among the different service domains or to the other cloud services 128 via the network 110. Generally, a data pipeline of the data pipelines 116, 124, and/or 134 may include an input component to receive data from another data pipeline, any data source, or other service domain or other cloud services 128 (via the network 110); and at least one transform component configured to manipulate the input data to provide the output data.
The data pipelines 116, 124, and/or 134 can be constructed using computing primitives and building blocks, such as VMs, containers, processes, or any combination thereof. In some examples, the data pipelines 116, 124, and/or 134 may be constructed using a group of containers (e.g., a pod) that each perform various functions within the data pipeline (e.g., subscriber, data processor, publisher, connectors that transform data for consumption by another container within the application or pod, etc.) to consume, transform, and produce messages or data. In some examples, the definition of stages of a constructed data pipeline application may be described using a user interface or REST API, with data ingestion and movement handled by connector components built into the data pipeline. Thus, data may be passed between containers of a data pipeline using API calls.
In some examples, the PaaS software stacks may further include respective ML inference services that are configured to load and execute respective ML model applications. Thus, the ML inference services may be configured to receive a request for an inference or prediction using a ML model, and to load a ML model application that includes the requested ML model into an inference engine. The inference engine may be configured to select a runtime based on a hardware configuration of the edge system, and execute the ML model on input data to provide inference or prediction data. The inference engine may be configured to optimize the ML model for execution based on a hardware configuration. The ML inference service may provide the benefits of GPU abstraction, built-in frameworks for ML model execution, decoupling application development from hardware deployment, etc. In some examples, the PaaS manager 108 may be configured to access data from the one or more data lakes, train a ML model using the transformed data, and generate an ML model application based on the trained ML model.
The one or more applications of the PaaS software stacks may be implemented using a containerized architecture that is managed via a container orchestrator. The container orchestration managed by a PaaS infrastructure and application lifecycle manager (PaaS manager 108) under the service domain construct may handle (e.g., using middleware) underlying details of the PaaS related to containerized management complexity, orchestration, security, and isolation, thereby making it easier for a customer or user to focus on managing the applications. The management may be scalable via categories. In some examples, the service domains may be configured to support multi-tenant implementations, such that data is kept securely isolated between tenants. The applications communicate using application programming interface (API) calls, in some examples. In some examples, the supporting services may also be implemented in the containerized architecture.
The PaaS manager 108 hosted on the central computing system 106 may be configured to centrally manage the PaaS infrastructure (e.g., including the service domains) and manage lifecycles of deployed applications. The central computing system 106 may include one or more computing nodes configured to host the PaaS manager 108. The central computing system 106 may include a cloud computing system and the PaaS manager 108 may be hosted in the cloud computing system and/or may be delivered/distributed using a software as a service (SaaS) model, in some examples. In some examples, the PaaS manager 108 may be distributed across a cluster of nodes of the central computing system 106.
In some examples, an administrative computing system 102 may be configured to host a PaaS manager interface 104. The PaaS manager interface 104 may be configured to facilitate user or customer communication with the PaaS manager 108 to control operation of the PaaS manager 108. The PaaS manager interface 104 may include a graphical user interface (GUI), APIs, command line tools, etc., that are each configured to facilitate interaction between a user and the PaaS manager 108. The PaaS manager interface 104 may provide an interface that allows a user to develop template applications for deployment of the service domains, identify on which service domains to deploy applications, move applications from one service domain to another, remove an application from a service domain, update an application, service domain, or PaaS software stack (e.g., add or remove available services, update deployed services, etc.).
In some examples, the PaaS manager 108 may be configured to manage, for each of the computing platforms, creation and deployment of service domains, creation and deployment of application bundles to the PaaS software stacks, etc. For example, the PaaS manager 108 may be configured to create and deploy service domains on one or more of the computing platforms. The computing platforms may include different hardware and software architectures that may be leveraged to create and deploy a service domain. Thus, the PaaS manager 108 may be configured to manage detailed steps associated with generating a service domain in response to a received request.
The PaaS manager 108 may also be configured to build and deploy different types of applications to one or more of the service domains. A user may elect to deploy an application to a type of platform based on various criteria, such as type of and/or availability of a service, proximity to source data, available computing resources (e.g., both type and available capacity), platform cost, etc., physical location of the platform, or any combination thereof.
When an application is generated, successful execution may depend on availability of various additional supporting services, such as a read/write data services (e.g., publish/subscribe service, search services, etc.), ML inference services, container management services, runtime services, etc., or any combination thereof. The PaaS manager 108 may abstract deployment of the additional supporting services, as some of these may be platform-specific. Thus, a user may provide information directed to an application to be deployed to the PaaS manager 108 and identify one or more target service domains, and the PaaS manager 108 may deploy the application to the target service domains. The target service domains provide services to be used by the application, and accordingly, the application need not include services provided by the service domain. Moreover, the application need not take platform-specific actions which may be typically required for starting those services. The PaaS manager 108 may deploy the respective application to the corresponding one of the one or more identified target service domains.
The ability of the PaaS manager 108 to abstract platform-specific details for creating and deploying a service domain and creating and deploying an application or application bundle to run in a service domain may make deployment of applications to different service domains more efficient for a user, as well as may provide a customer with a wider selections of platforms than would otherwise be considered. Thus, the service domain construct may allow a customer to focus on core concerns with an application, while shifting consideration of supporting services to the PaaS manager 108 and the service domains. The service domain construct may also make applications more “light weight” and modular for more efficient deployment to different service domains. The PaaS manager interface 104 may provide a GUI interface for selecting a type of application to be deployed to one or more service domains, in accordance with an embodiment of the present disclosure.
The computing node 200 includes a communications fabric 222, which provides communication between one or more processor(s) 212, memory 214, local storage 202, communications unit 220, and I/O interface(s) 210. The communications fabric 222 can be implemented with any architecture designed for passing data and/or control information between processors (such as microprocessors, communications and network processors, etc.), system memory, peripheral devices, and any other hardware components within a system. For example, the communications fabric 222 can be implemented with one or more buses.
The memory 214 and the local storage 202 are computer-readable storage media. In this embodiment, the memory 214 includes random access memory RAM 216 and cache 218. In general, the memory 214 can include any suitable volatile or non-volatile computer-readable storage media. In an embodiment, the local storage 202 includes an SSD 204 and an HDD 206.
Various computer instructions, programs, files, images, etc., may be stored in local storage 202 for execution by one or more of the respective processor(s) 212 via one or more memories of memory 214. In some examples, local storage 202 includes a magnetic HDD 206. Alternatively, or in addition to a magnetic hard disk drive, local storage 202 can include the SSD 204, a semiconductor storage device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, or any other computer-readable storage media that is capable of storing program instructions or digital information.
The media used by local storage 202 may also be removable. For example, a removable hard drive may be used for local storage 202. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer onto another computer-readable storage medium that is also part of local storage 202.
Communications unit 220, in some examples, provides for communications with other data processing systems or devices. In these examples, communications unit 220 includes one or more network interface cards. Communications unit 220 may provide communications through the use of either or both physical and wireless communications links.
I/O interface(s) 210 allow for input and output of data with other devices that may be connected to a computing node 200. For example, I/O interface(s) 210 may provide a connection to external devices such as a keyboard, a keypad, a touch screen, and/or some other suitable input device. External devices can also include portable computer-readable storage media such as, for example, thumb drives, portable optical or magnetic disks, and memory cards. Software and data used to practice embodiments of the present disclosure can be stored on such portable computer-readable storage media and can be loaded onto local storage 202 via I/O interface(s) 210. I/O interface(s) 210 may also connect to a display 208.
Display 208 provides a mechanism to display data to a user a may be, for example, a computer monitor. In some examples, a GUI associated with the PaaS manager interface 104 may be presented on the display 208.
Examples described herein accordingly include one or more clusters, such as cluster 302 of
Examples of computing nodes in clusters described herein may have operating systems. For example, the nodes 306, 308, and 310 of the cluster 302 include an operating system (e.g., operating systems 326, 328, and 330) configured to interface with the physical layer of each respective computing node. Any type of operating system may be used in various examples.
Examples of computing nodes in clusters described herein may have one or more cluster infrastructure orchestrators, which may also be called a cluster orchestrator. For example,
Examples of computing nodes in clusters described herein may have a container orchestrator, such as container orchestrator 312, container orchestrator 314, and/or container orchestrator 316 of
Generally, container orchestrators may be dependent on (e.g., may be in a higher software layer than) cluster infrastructure orchestrators. That is, typically, the container orchestrators may require the cluster infrastructure orchestrator to be operational prior to starting up (e.g., operating) the container orchestrator.
In examples described herein, a container orchestrator, such as container orchestrator 314 of
In examples described herein, the container orchestrator may be present in a different software layer than the cluster orchestrator of a same or different node. For example, inputs and outputs for communication between layers may be defined. In some examples, the container orchestrator may be provided in a higher software layer than the cluster orchestrator, which may reside in a lower software layer. Typically, software in a lower software layer will be started up and/or configured prior to startup and/or configuration of software in a higher software layer. Software in a higher software layer may have dependencies on software in a lower software layer. In examples described herein, the container orchestrator(s) may have dependencies on the cluster infrastructure orchestrator(s). For example, for proper operation of the container orchestrator, the cluster infrastructure orchestrator on a same node should be operational. Accordingly, software in a lower software layer typically may not be expected to have dependencies on software from a higher software layer. However, in examples described herein, cluster infrastructure orchestrators, such as cluster infrastructure orchestrator 322 of
In examples described herein, aspects of the clustered nature of the computing environment may be utilized to increase and/or ensure resilient operation of the computing nodes despite the dependency of a lower level software layer (e.g., cluster orchestrators) higher level software layers (e.g., container orchestrators). For example, one or more nodes of the cluster may be a bootstrap node, such as bootstrap node 308 of
Each cluster infrastructure orchestrator layer is configured with a dependency on the container orchestrator 314 of the bootstrap node 308. Accordingly, each cluster infrastructure orchestrator 320, 322, and 324 can access the cluster key value store 318 to use cluster metadata stored at the cluster key value store 318. Accordingly, where the cluster is scaled (e.g., by adding an additional node), the cluster infrastructure orchestrator of the new node can be configured with a dependency on the container orchestrator 314 of the bootstrap node 308 to access the cluster key value store 318 instead of a separate cluster key value store being created at the new node. Additionally, updated cluster metadata (e.g., metadata reflecting the additional node) is added to the shared cluster key value store 318 such that each individual cluster infrastructure orchestrator does not need to be updated to reflect changed metadata. Similarly, in the event of node failure or node reboot, the shared cluster key value store 318 can be updated and the cluster infrastructure orchestrator of the rebooted node may be configured to reference the shared cluster key value store 318 at the bootstrap node 308.
When a cluster is initially setup, a cluster infrastructure orchestrator may be installed on a bootstrap node. The metadata used by the cluster orchestrator of the bootstrap node, e.g., cluster infrastructure orchestrator 322 of
As other nodes come up, their cluster orchestrators (e.g., cluster infrastructure orchestrator 320 and cluster infrastructure orchestrator 324) may access cluster key value store 318 from a different node than their own node (e.g., from bootstrap node 308) in order to access (e.g., obtain) the metadata used to manage the cluster. In this manner, cluster orchestrators on other nodes may be able to start up prior to startup of the container orchestrators on their own nodes, because they can access the key value store through the container orchestrator of the bootstrap node. For example, a second node may be added to the cluster (e.g., node 306 may be added to cluster 302 of
In various implementations, clusters may include multiple bootstrap nodes, each having a cluster key value store to further protect the cluster from node failure. For example, where one of the multiple bootstrap nodes fails, cluster infrastructure orchestrators on the non-bootstrap nodes may reference the cluster key value store on the remaining bootstrap node.
Installation of the software stack may first bring up (e.g., start up) the operating system 328 at the bootstrap node 308 to provide an interface between the hardware of the bootstrap node 308 and software running on the bootstrap node 308. Once the operating system 328 is installed and initiated, the cluster infrastructure orchestrator 322 is initiated, running on the operating system 328. At the time the cluster infrastructure orchestrator 322 is initiated, cluster metadata used by the cluster infrastructure orchestrator 322 may be stored at the cluster infrastructure orchestrator 322. A container orchestrator 314 may be initiated (e.g., installed) above the cluster infrastructure orchestrator 322 after initiation of the cluster infrastructure orchestrator 322. When the container orchestrator 314 is initiated (e.g., installed), the cluster metadata stored locally at the cluster infrastructure orchestrator 322 may be moved and/or copied to a cluster key value store 318 at the container orchestrator 314.
In block 404, a dependency of the cluster orchestrator is set to the container orchestrator on the bootstrap node. For example, after moving the cluster metadata from the cluster infrastructure orchestrator 322 to the cluster key value store 318, the cluster infrastructure orchestrator 322 may be configured with a dependency on the container orchestrator 314 to access the cluster metadata. In this manner, a dependency may be created such that the cluster orchestrator is dependent on the container orchestrator for proper operation. The cluster infrastructure orchestrator 322 may then access cluster metadata stored at the cluster key value store 318 to perform lifecycle management operations for the bootstrap node 308 of the cluster 302.
In block 406, additional nodes are added to the cluster with dependencies of the cluster orchestrator on the container orchestrator. For example, the node 306 may be configured as part of the cluster 302 by installing and initializing a PaaS software stack on the node 306. The operating system 326 may be installed and initialized first to provide a platform for other software on the node 306. When the cluster infrastructure orchestrator 320 is initialized at the node 306, the cluster infrastructure orchestrator 320 may be configured with a dependency on the container orchestrator 314 of the bootstrap node 308 for access to the cluster key value store 318. The container orchestrator 312 may then be initialized above the cluster infrastructure orchestrator 320.
Additional nodes may be added to the cluster 302 in a similar manner, where cluster infrastructure orchestrators of additional nodes are configured with dependencies on the container orchestrator 314 of the bootstrap node 308 for access to the cluster key value store 318. During the lifecycle of the cluster, cluster metadata at the cluster key value store 318 may be updated as nodes are added to or removed from the cluster or as the PaaS software stack is updated at the nodes of the cluster.
While certain components are shown in the figures and described throughout the specification, other additional, fewer, and/or alternative components may be included in the cluster 302 or other computing systems. Such additional, fewer, and/or alternative components are contemplated to be within the scope of this disclosure.
From the foregoing it will be appreciated that, although specific embodiments have been described herein for purposes of illustration, various modifications may be made while remaining with the scope of the claimed technology.
Examples described herein may refer to various components as “coupled” or signals as being “provided to” or “received from” certain components. It is to be understood that in some examples the components are directly coupled one to another, while in other examples the components are coupled with intervening components disposed between them. Similarly, signals may be provided directly to and/or received directly from the recited components without intervening components, but also may be provided to and/or received from the certain components through intervening components.
This application claims the benefit of the earlier filing date of U.S. Provisional Application 63/094,733, filed Oct. 21, 2020, which application is hereby incorporated by reference in its entirety for any purpose.
| Number | Date | Country | |
|---|---|---|---|
| 63094733 | Oct 2020 | US |
