Patent Application
20250219962
Several commercially available cloud platforms utilize Kubernetes® Engine for deploying, managing, and scaling software testing automation systems. These automation systems are highly valuable in performing tasks specified by a user. The current solutions are severely limited by requiring manual user intervention for provisioning. Manual provisioning of cloud resources has limitations on scaling automated quality tests which limit how many resources can process and which slow engineering processes. Manual testing also leads to more product defects, lower customer satisfaction, and higher support costs. Furthermore, manual provisioning of cloud resources has limitations in simulating complex production environments.
As mentioned above several commercially available software products attempt to automated interaction with Kubernetes® Engine. For example, Google's® Kubernetes® Engine (GKER) is an application programming interface (API) provided with Google's® Cloud Platform (GCP®) which automates various low-level interactions required of a user when using the API associated with Kubernetes® Engine. One such automation provided by the GKER API is node scaling, which is designed to dynamically allocate GCP® resources at any level dependent on a current need of a product. The node scaling feature of GKE® can determine the appropriate number of GCP® resources to provide dependent on the requests made for a given cluster of resources, a node pool of resources, or node of resources. Similarly, GKER automatically scales down GCP® resources and deletes resources to prevent customer billing associated with unused resources.
However, there are several limitations associated with GKE® API and other commercially available APIs that attempt to automate the low-level API interaction required by the API of Kubernetes® Engine. For example, GKER does not allow for multiple instances of node pools to be deployed at a same time. Thus, a node pool of 5 nodes can take approximately 120 seconds to deploy, when there are 5 node pools this means approximately 600 seconds or 10 minutes is required to deploy all 5 node pools. Furthermore, each deployment of each node pool requires manual user interaction with the GKER API even though the node pools are different instances of a same node pool. This is but one example of limitations associated with commercially available front-end APIs to Kubernetes® Engine.
The teachings provided herein solves the aforementioned issues and others associated with automating low-level interactions with Kubernetes® Engine. The teachings provide automation and corresponding features and function for configuring, managing, and controlling resources. In an embodiment, the novel features and function utilize existing third-party APIs to the Kubernetes® Engine. In an embodiment, the novel features and function utilize a new and novel API to the Kubernetes® Engine.
At least four features and corresponding functions are presented that all interact with one another. There are features and corresponding functions to 1) deploy the resources, 2) delete the resources; 3) get current resources in a cluster, pool, and/or node; and 4) control each of 1)-3). Initially, the user is authenticated for direct interaction and control of a cloud environment. Once a user account is authenticated, the user is permitted through a user interface to create a certain type of resource, such as a node pool or node. The user provides certain configurations and characteristics to the resources provided; for example, where to deploy the resource or the type of virtual machines (VMs) to use.
When this request is approved, a configurable number of resources can be made/reserved within the environment. For instance, a user can deploy 5 node pool instances using one node pool and corresponding resources definition provided by the user within the user interface (e.g., one user interface interaction or session with the user interface), these node pool instances can individually be deployed with, for example, 10 nodes within each of them. The status of the pool instances being deployed are monitored. The functions associated with creating and deploying resources are similarly used for the resource deletion feature of the API and user interface.
A loop-based function deploys the pool instances using a monitored constraint associated with a predefined reservation/creation time associated with the environment. The predefined reservation/creation time is the amount of hardcoded time for a resource to be reserved/created within the environment. The loop-based function combined with the known a length of the environment's time out effectively overrides the environment's hardcoded reservation/creation time and provides more time for multiple instances of resources or pool instances to be created within the environment without experiencing an environment timeout. This loop-based function also enhances the feature/function of third-party APIs that provide a high-level interface to the Kubernetes® API.
For example, using the example presented above if there are 5 node pool instances of 5 nodes having a known environment creation/reservation time out of 120 seconds, then the total expected elapsed time to create/reserve the 1 pool instance within the environment is 600 seconds (i.e., 5 resources in each pool instance, 5 multiplied by 120 seconds is 600 seconds). The monitored time constraint is equal to the square root of 600 seconds multiplied by 5 (i.e., the total number of pool instances), which equals approximately ⅕th the time required with the GKER API (i.e., 120 seconds per resource multiplied by 5 resources in each pool instance is equal to 600 seconds, the square root of 600 is approximately 24.5 seconds, which when multiplied by 5 pool instances is equal to approximately 125 seconds). The loop-based function effectively overrides the environment's time out for resource creation/reservation providing the user more time to deploy instances of resources within the environment. This is turn permits a feature that allows the user the capability to deploy multiple pool instances from a single session with a user interface, which is not currency available in the industry.
In an embodiment, the reservation/creation of current resources feature creates a text file used by the control feature as each resource is created, reserved, and deployed. This allows differentiation during resource deployment and deletion of different resource instances. The file includes all names of the resources. This also permits different instances of the resources to be processed in parallel in multiple environments using the text file to differentiate each of the resource instances in each of the environments.
The resources are auto scaled at every level by the control feature, which monitors the central processing unit (CPU) usage for every resource being deployed, deleted, and created/reserved. A predefined CPU usage percentage minimum and maximum causes either deployment of a new resource to keep up with the need for additional resources or causes deletion of a resource when it is being underutilized or unused.
As used herein, a “node” includes a physical machine and/or a VM. A “pool of nodes” includes a grouping of nodes with a same configuration of resources such that resources are shared within the pool to avoid unnecessary creation and deletion of resources. A “cluster” includes a collection of nodes that serve concurrent clients for failover, redundancy, and high availability (HA). A “pod” is the smallest deployable unit available in the Kubernetes® Engine and is a group of one or more containers, with shared storage and network resources along with a specification for how to process the container. A “resource” includes a node, a pool of nodes, a cluster, a pod, an application, a system, a collection of applications, a collection of systems, a workflow that includes one or more workloads, applications and/or systems, and/or a collection of workflows. A “resource type” identifies one of the aforementioned resources. An “instance of a resource” is a duplicated resource of an original defined resource, which processes in parallel or concurrently within a same processing environment or within multiple processing environments.
Furthermore, the various components (that are identified in system 100) are illustrated and the arrangement of the components are presented for purposes of illustration only. Notably, other arrangements with more or less components are possible without departing from the teachings of dynamic resource configuration, management, and control presented herein and below.
System 100 provides a user interface, an API, and functions that support resource creation/reservation features, resource deployment features, resource deletion features, resource monitoring features, and control features of resource creation/reservation, resource deployment, and resource deletion via resource monitoring within one or more processing environments. The user interface, API, features, and corresponding functions are now discussed with reference to the components of system 100 illustrated in
System 100 includes a cloud/server 110 (hereinafter just “cloud 110”), a plurality of distributed devices 120, a plurality of user-operated devices 130, and a plurality of enterprise servers 140. Cloud 110 includes at least one processor 111 and a non-transitory computer-readable storage medium (hereinafter just “medium”) 112, which includes instructions for a resource manager 113, an API 114, and a user interface 115. The instructions when provided to and executed by processor 111 cause processor 111 to perform processing, functions, and/or operations discussed herein and below with respect to 113-116.
Each distributed device 120 includes at least one processor 121 and a medium 122, which includes instructions for resources 123, a Kubernetes® API 124, optionally a third-party API 125, and a plurality of cloud services 126. The instructions when provided to and executed by processor 121 cause processor 121 to perform the processing, functions, and/or operations discussed herein and below with respect to 123-126.
Each user-operated device (hereinafter just “device”) 130 includes at least one processor 131 and a medium 132, which includes instructions for cloud/server (hereinafter just “cloud”) application (herein after just “app”) 133. The instructions when provided to and executed by processor 131 from medium 132 cause processor 131 to perform the processing, functions, and/or operations discussed herein and below with respect to 133.
Each enterprise server 140 includes at least one processor 141 and medium 143, which includes instructions for one or more cloud services 143 The instructions when provided to and executed by processor 141 from medium 142 cause processor 141 to perform the processing, functions, and/or operations discussed herein and below with respect to 143.
Resource manager 113 includes user interface 115 as a front-end interface, which is rendered via cloud/server app 133 during user sessions between a user operating device 130 and resource manager 113 of cloud 110. Resource manager 113 obtains configuration settings selected and/or entered by the user during the session through user interface screens or user interface 115. The configuration settings define constraints and requirements for resources 123. The resources 123 when processed within a given processing environment or multiple processing environments cooperate to provide cloud services 126. Cloud services 143 access cloud services 126 from enterprise servers 140.
In an embodiment, cloud services 126 provide a transaction system or transaction services via cloud service 126 to enterprise servers 140 and enterprise devices, such as transaction terminals. In an embodiment, the transaction system, provided via cloud services 126, includes a plurality of microservices for transaction processing, transaction security services, transaction monitoring services, transaction logging services, customer loyalty services, etc. In an embodiment, the cloud services 126 cooperate to provide any combination of a transaction system/transaction services, an inventory system/inventory services, a security system/security services, and a customer loyalty system/customer loyalty services, any combination of which is provided from the environment(s) to enterprise servers 140 servers via cloud service 143.
Resource manager 113 translates and converts the configuration settings into corresponding Kubernetes® API commands by processing operations and functions, described herein, and issuing Kubernetes® API calls using backend API 114 sent to the Kubernetes® API 124. The functions and operations provide for deploying resources 123, deleting resources, 123, getting/creating/reserving current resources 123, and controlling the deployment, the deletion, and creating/getting/obtaining of the resources 123 via environment management information.
In an embodiment, resource manager 113 translates and converts the configuration settings into a third-party API 125, such as a GKER API. The third-party API 125 then interacts with Kubernetes® API 124 to deploy the resources within the environment(s). In an embodiment, resource manager 113 translates configuration settings provided via user interface 115 into commands and calls associated with both the Kubernetes® API 124 and the third-party API 125.
Resource manager 113 provides options within the user interface 115, which is rendered to a user via cloud app 133, to deploy multiple instances of a same resource during a single user session. That is, in a single user session can define a resource for reservation and deployment within a single environment or multiple environments as multiple concurrent instances. This is not available via existing third-party APIs.
In an embodiment, resource manager 113 uses a loop-based function with a third-party API 125 associated with the environment(s) in response to a user option to deploy multiple instances of a resource 123 within the environment(s). the loop-based function overrides an environment resource reservation/creation time out to extend the time out, which in turn permits multiple instances of a resource to be created/reserved and deployed within the environment via a single user session with user interface 115. Example, pseudocode that the resource manager performs to provide the loop-based function is as follows:
In this way, the third-party API 125 is enhanced via the user interface option and resource manager 113 since resource manager 113 performs a feature or function not available with the third-party API 125 by effectively overriding a hardcoded resource creation/reservation/deployment time out and extending it in a manner that permits a user to use a single session and deploy multiple resource instances uses through a single session with resource manager 113 via user interface 115.
At the start of a session between the user and resource manager 113, resource manager 113 authenticates the user for access to the environment or environment(s) via user interface 115. Resource manager 113 initiates a session with third-party API 125 or environment APIs using the user's provided credentials. So, the session between the user and resource manager 113 is managed separately as a session between resource manager 113 and the environment(s). Once the user session with resource manager 113 is established and authenticated by the environment(s), resource manager 113 has access to the environment(s) for configuring, deploying, deleting, getting/creating/reserving, and controlling resources 123 for the user based on user defined configuration settings provided through user interface screens and environment management information.
The resource configuration settings include, by way of example only, resource types, hardware settings, management policies, environment identifiers, resource image types, resource identifiers, CPU types, boot disk sizes, memory requirements, concurrent session requirements, whether boot disk encryptions are enabled or not, provisioning models used, a total number of nodes being requested, whether autoscaling is on or off, any node zones, deployment schedules, and/or whether compact placement is being requested or not. Resource manager 113 translates or converts the configuration settings defined by the user into metadata recognized by Kubernetes® API 124 and/or third-party API 125. Notably, other configuration settings associated with customized management settings and/or policies can also be provided via field selections to the user via user interface screens. The customized managements settings and/or policies can include multiple low-level configuration settings packaged and identified via a name associated with the corresponding settings and/or policies.
When the user is reserving/initiating/getting multiple instances of a same resource 123 for deployment, resource manager 113 processes the loop-based function described above. As each resource instance is created, reserved, and deployed, resource manager 113 writes a resource name for each instance and/or its environment identifier into a text file once each instance creation, reservation, and deployment is confirmed by the environment. This ensures that a given instance is properly reserved/initiated before a next instance is sent to the environment via the third-party API 125 and/or Kubernetes® API 124. Essentially, a next instance of the resource 123 being reserved/initiated is blocked until confirmation is received that the prior instance of the resource 123 was reserved/initiated. Conventionally, the environment would time out once the hardcoded reservation time out is reached for the number of resources or resource instances being deployed. This time out period is extended by the loop-based function to be the square root of the product of the time out multiplied by the total number of resources being deployed. By extending the time out period of the environment, resource manager 113 provides a single user session with an ability or a feature to deploy multiple pools of resource instances, which has conventionally not been possible.
Resource manager 113 handles resource deletions similarly to how resource creation, reservation, and deployment are handled. The resource manager 113 uses the text file to manage deletion of instances of resources.
Resource manager 113 also is continuously obtaining management information for the deployed resources 123 from the environment using Kubernetes® API 124 and/or a third-party API 125. User set constraints are for maximums and minimums are monitored and the resources 123 are auto scaled. Thus, resource manager 113 will deploy a new resource 123 to keep up with demand for the resource being experienced by cloud services 126 (i.e., a maximum constraint is detected within the environment) and/delete resources 123 when demand for a resource by cloud services 126 wanes (i.e., a minimum constraint is detected within the environment).
System 100 permits dynamic cluster/node pool deployment, dynamic cluster/node pool management, dynamic cluster/node pool deletion, an ability to specify machine specifications, and an ability to assign taints/labels for simulating production environments. This is achieved with integration to the Kubernetes® API 124 and/or third-party APIs 125 that provide a high level interface to the Kubernetes® API 124. A user interface 115 permits a single user session with resource manager 113. Resource manager 113 performs resource deployment, deletion, creation/reservation, and control of deployment, deletion, and creation/reservation.
The above-referenced embodiments and other embodiments are now discussed with reference to
In an embodiment, the device that executes the resource manager is cloud 110. In an embodiment, the device that executes the resource manager is server 110. In an embodiment, at least a portion of the resource manager is executed by a combination of devices 110, 120, 130, and/or 140. In an embodiment, the resource manager is resource manager 113, API 114, user interface 115, resources 123, cloud services 126, cloud app 133, and/or cloud services 143.
At 210, the resource manager renders a user interface 115 to an environment that includes resources 123. In an embodiment, the resources 123 include any of the above-mentioned example resources 123 discussed with system 100 of
At 220, the resource manager receives resource configuration settings for the resources 123 from a user during a session. In an embodiment, the configuration settings include any of the above-mentioned example configuration settings discussed with system 100 of
In an embodiment, at 221, the resource manager identifies in the configuration settings a certain resource 125 that user is requesting during the session to be deployed as multiple instances of the certain resource 125 within the environment. That is, the configuration settings are provided during a single session by the user through the user interface 115 and identify that certain resource 123 is to be deployed as multiple duplicated and concurrent instances of certain resource 123 within the environment. In an embodiment, the certain resource 123 is a pool of nodes, the pool of nodes is identified in the configuration settings as requesting deployment of multiple instances of the pool of nodes within the environment.
At 230, the resource manager translates the resource configuration settings into API calls associated with the environment. In an embodiment of 221 and 230, at 231, the resource manager causes a resource time out for the environment to be overridden to permit additional time to deploy the instances of the certain resource 123.
In an embodiment, at 232, the resource manager manages the API calls being directed to a low-level associated with resource management of the environment. In an embodiment, the low-level is a Kubernetes® API 124.
In an embodiment, at 233, the resource manager manages the API calls being directed to a third-party API 125 that interacts with a low-level API associated with resource management of the environment. In an embodiment, the third-party API is the GKER API and the low-level API is the Kubernetes® API 124.
At 240, the resource manager controls deployment, deletion, and reservation of the resources 123 using the API calls. In an embodiment, at 241, the resource manager monitors utilization of the resources 123 after the reservation and deployment of the resources 123 using a low-level API associated with the environment.
In an embodiment of 241 and at 242, the resource manager reserves and deploys new resources 123 within the environment based on user defined maximums and minimums set for resource utilization and based on 241. In an embodiment of 242 and at 243, the resource manager deletes existing resources 123 within the environment based on the maximums and minimums and based on 241.
In an embodiment, at 244, the resource manager auto scales the resources 123 based on monitoring of the resources 123. That is, the utilization of the resources is monitored for CPU usage, memory usage, storage usage, network bandwidth, and/or concurrent network connections.
In an embodiment, at 245, the resource manager maintains a management file with resource names for the resources 123 as the resources 123 are deployed and deleted from the environment. In an embodiment of 245 and at 246, the resource manager maintains the management file as a text file to manage the resources 123 and instances of the resources 123.
In an embodiment, at 250, the resource manager (210-240) processes as a front-end interface to a low-level API. The low-level API provides resource management for the resources 123 of the environment. In an embodiment of 250 and at 251, the resource manager (210-240) processes as a further front-end interface to a third-party API 125. The third-party API 125 interacts with the low-level API and performs resource management for the resources 123 in the environment.
In an embodiment, the device that executes the distributed resource provisioner manager is cloud 110. In an embodiment, the device that executes the distributed resource provisioner manager is a server 110. In an embodiment, at least a portion of the distributed resource provisioner manager is executed by a combination of devices 110, 120, 130, and/or 140. In an embodiment, the distributed resource provisioner manager is resource manager 113, API 114, user interface 115, resources 123, cloud services 126, cloud app 133, and/or cloud services 143. The distributed resource provisioner manager presents another and, in some ways, enhanced processing perspective from that which were discussed above for system 100 and method 200.
At 310, the distributed resource provisioner manager provides a user interface 115 to an API that manages resources 123 of an environment. In an embodiment, the API is one or any combination of APIs 124 and/or 125.
At 320, the distributed resource provisioner manager extends capabilities of the API by providing options within the user interface 115 that are unavailable with the API alone. That is, the API does not provide and does not permit features/functions associated with the extended capabilities, which are identified by the options of the user interface 115.
In an embodiment, at 321, the distributed resource provisioner manager provides a certain capability that permits deployment of multiple instances of a certain resource 123 within the environment during the session. In an embodiment of 321 and at 322, the distributed resource provisioner manager processes a loop function to reserve and deploy the multiple instances of the resources 123 within the environment during the session. In an embodiment, the loop function performs the pseudocode presented above with system 100 of
In an embodiment of 322 and at 323, the distributed resource provisioner manager overrides, via the loop function, a resource reservation and deployment time out associated with one or more of the API and the environment. In an embodiment of 323 and at 324, the distributed resource provisioner manager maintains a management file to distinguish and to independently manage each of the multiple instances of the resources 123 within the environment.
At 330, the distributed resource provisioner manager controls deployment, deletion, and reservation/creation/initiation of the resources 123 within the environment based on configuration settings received from the user through the user interface 115 during a session with the user. The manner in which the deployment, deletion, reservation/creation/initiation is performed was discussed above with system 100 of
It should be appreciated that where software is described in a particular form (such as a component or module) this is merely to aid understanding and is not intended to limit how software that implements those functions may be architected or structured. For example, modules are illustrated as separate modules, but may be implemented as homogenous code, as individual components, some, but not all of these modules may be combined, or the functions may be implemented in software structured in any other convenient manner.
Furthermore, although the software modules are illustrated as executing on one piece of hardware, the software may be distributed over multiple processors or in any other convenient manner.
The above description is illustrative, and not restrictive. Many other embodiments will be apparent to those of skill in the art upon reviewing the above description. The scope of embodiments should therefore be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.
In the foregoing description of the embodiments, various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting that the claimed embodiments have more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Description of the Embodiments, with each claim standing on its own as a separate exemplary embodiment.
