Patent Grant
11226847
This invention relates to orchestration of roles in an application instantiated in a distributed storage and computation system.
Many computing tasks are complex and require many different application instances to be running and cooperating with one another. The installation and management of such an application is a complex task. One tool for performing such a task is KUBERNETES, which can both instantiate containers executing application instances, scale the number of application instances based on load, and replaced failed application instances.
The systems and methods disclosed herein provide an improved approach for using KUBERNETES to deploy and manage a multi-role application.
In order that the advantages of the invention will be readily understood, a more particular description of the invention briefly described above will be rendered by reference to specific embodiments illustrated in the appended drawings. Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered limiting of its scope, the invention will be described and explained with additional specificity and detail through use of the accompanying drawings, in which:
Referring to
One or more compute nodes 110 are also coupled to the network 104 and host user applications that generate read and write requests with respect to storage volumes managed by the storage manager 102 and stored within the memory devices 108 of the storage nodes 106.
In some embodiments, an orchestrator 112 may execute on a computer system of the network environment 100 and facilitate deployment of a multi-role application on the compute nodes 110. The orchestrator 112 may further instruct the storage manager 102 regarding the creation of storage volumes on the storage nodes 106 and the mounting of such storage volumes to particular compute nodes 110.
The orchestrator 112 may perform this function in cooperation with a KUBERNETES (hereinafter “Kubernetes”) master 114. In particular, the Kubernetes master 114 may manage the deployment of containers on compute nodes 110, monitor availability of the containers, and restore containers on compute nodes 110 upon failure of the container or the compute node 110 executing it.
The methods disclosed herein ascribe certain functions to the storage manager 102, storage nodes 106, compute node 110, orchestrator 112, and Kubernetes master 114. The methods disclosed herein are particularly useful for large scale deployment including large amounts of data distributed over many storage nodes 106 and accessed by many compute nodes 110. However, the methods disclosed herein may also be implemented using a single computer implementing the functions ascribed herein to some or all of the storage manager 102, storage nodes 106, and compute node 110.
Referring to
The orchestrator 112 may implement a bundled application 202 defining roles and relationships between roles as described in greater detail below. The bundled application 202 may include a manifest 204 that defines the roles of the bundled application 202, which may include identifiers of role instances, network addresses to be assigned to role instances, and possibly a number of instances for each role instance may be identified. The manifest 204 may define dynamic functions defining how the number of instances of particular role may grow or shrink depending on usage. The orchestrator 112 may then create or remove instances for a role as described below as indicated by usage and one or more functions for that role. The manifest 204 may define a topology of the bundled application 202, i.e. the relationship between roles, such as services of a role that are accessed by another role.
The bundled application 202 may include provisioning 206. The provisioning 206 defines the resources of storage nodes 106 and compute nodes 110 required to implement the bundle. The provisioning 206 may define resources for the bundle as a whole or for individual roles. Resources may include a number of processors (e.g., processing cores), an amount of memory (e.g., RAM (random access memory), an amount of storage (e.g., GB (gigabytes) on a HDD (Hard Disk Drive) or SSD (Solid State Drive)). These resources may be provisioned in a virtualized manner such that the bundled application 202 and individual roles 212 are not informed of the actual location or processing and storage resources and are relieved from any responsibility for managing such resources. In particular, storage resources may be virtualized by the storage manager 102 using the methods described above such that storage volumes are allocated and used without requiring the bundled application 202 or roles to manage the underlying storage nodes 106 and storage device 108 on which the data of the storage volumes is written.
Provisioning 206 may include static specification of resources and may also include dynamic provisioning functions that will invoke allocation of resources in response to usage of the bundled application. For example, as a database fills up, additional storage volumes may be allocated. As usage of a bundled application increases, additional processing cores and memory may be allocated to reduce latency.
A bundled application 202 may further include configuration parameters 208. Configuration parameters may include variables and settings for each role of the bundle. The configuration parameters are defined by the developer of the role and therefore may include any example of such parameters for any application known in the art. The configuration parameters may be dynamic or static. For example, some parameters may be dependent on resources such as an amount of memory, processing cores, or storage. Accordingly, these parameters may be defined as a function of these resources. The orchestrator 112 will then update such parameters according to the function in response to changes in provisioning of those resources that are inputs to the function. For example, CASSANDRA defines a variable Max_Heap_Size that is normally set to half the memory limit. Accordingly, as the memory provisioned for a CASSANDRA role increases, the value of Max_Heap_Size may be increased to half the increased memory.
The bundled application 202 may further include action hooks 210 for various actions that may be taken with respect to the bundled application and/or particular roles of the bundled applications. Actions may include some or all of stopping, starting, restarting, taking snapshots, cloning, and rolling back to a prior snapshot. For each action, one or more action hooks may be defined. A hook is a programmable routine that is executed by the orchestrator 112 when the corresponding action is invoked. A hook may specify a script of commands or configuration parameters input to one or more roles in a particular order. Hooks for an action may include a pre-action hook (executed prior to implementing an action), an action hook (executed to actually implement the action), and a post action hook (executed following implementation of the action).
The bundled application 202 may define a plurality of roles 212. Each role may include one or more provisioning constraints. As noted above, the bundled application 202 and roles 212 are not aware of the underlying storage nodes 106 and compute nodes 110 inasmuch as these are virtualized by the storage manager 102 and orchestrator 112. Accordingly, any constraints on allocation of hardware resources may be included in the provisioning constraints 214. As described in greater detail below, this may include constraints to create separate fault domains in order to implement redundancy and constraints on latency.
The role 212 may define a name space 216. A name space 216 may include variables, functions, services, and the like implemented by a role. In particular, interfaces and services exposed by a role may be included in the name space. The name space may be referenced through the orchestrator 112 by an addressing scheme, e.g. <Bundle ID>.<Role ID>.<Name>. In some embodiments, references to the name space 216 of another role may be formatted and processed according to the JINJA template engine or some other syntax. Accordingly, each role 212 may access the variables, functions, services, etc. in the name space 216 of another role 212 on order to implement a complex application topology. In some instances, credentials for authorizing access to a role 212 may be shared by accessing the namespace 216 of that role.
A role 212 may further include various configuration parameters 218 defined by the role, i.e. as defined by the developer that created the executable for the role. As noted above, these parameters 218 may be set by the orchestrator 112 according to the static or dynamic configuration parameters 208. Configuration parameters may also be referenced in the name space 216 and be accessible (for reading and/or writing) by other roles 212.
Each role 212 may include a container 220 executing an instance 222 of the application for that role. The container 220 may be a virtualization container, such as a virtual machine, that defines a context within which the application instance 222 executes, facilitating starting, stopping, restarting, and other management of the execution of the application instance 222. Containers 220 may include any container technology known in the art such as DOCKER, LXC, LCS, KVM, or the like. In a particular bundled application 202, there may be containers 220 of multiple different types in order to take advantage of a particular container's capabilities to execute a particular role 212. For example, one role 212 of a bundled application 202 may execute a DOCKER container 220 and another role 212 of the same bundled application 202 may execute an LCS container 220. The manifest 204 and/or provisioning 206 may define a particular container 220 of the bundled application 220 to mount each provisioned storage volume for use by that container 220.
Note that a bundled application 202 as configured in the foregoing description may be instantiated and used or may be saved as a template that can be used and modified later.
The operation of the orchestrator 112 and storage manager 102 with respect to the bundled application 202 may be performed according to the approaches described in the applications of Table 1 that are hereby incorporated herein by reference in their entirety (“the incorporated applications”):
The Kubernetes master 114 may provide the pod specification to a Kubelet 302 executing on the compute node 110 and may also invoke instantiation of the Kubelet on the compute node 110 if not already present. The Kubelet 302 then instantiates a pod 304 that may instantiate and manage one or more containers 300 according to the pod specification. The Kubelet 302 may monitor the state of operation of the containers 300 and report their status to the Kubernetes master 114.
As known in the art, a Kubernetes installation include the Kubernetes master 114 that is programmed to receive instructions in the form of a helm chart, StatefulSet, or operators received from a user or script. Accordingly, the Kubernetes master 114 is a second orchestrator in addition to the orchestrator 112 that operates in a different manner and in a different scope than the orchestrator 112.
For example, instructions to the Kubernetes master 114 may instruct it to allocate a Kubernetes node, which is a computer or virtual (e.g., cloud) computing resource that is allocated for providing a service. The Kubernetes master 114 may select an available node 110 and invoke installation of the Kubelet 302 on that node 110, which is an agent that implements instructions from the master 114 as well as reports the status of components executing on the node. A node 110 may execute one or more pods 304, which is a group of one or more containers 300 with shared resources, such as storage resources, network resources, or the like. The pod 304 may further define a virtual machine in which all containers of the pod 304 execute. The pod 304 may define a common name space that is accessible by all of the containers of the pod.
The Kubernetes master 114 is an “intent-based” orchestrator and implements containers and pods based on available resources according to instructions. The Kubernetes master 114 does not implement constraints on the nodes on which pods and containers are instantiated. The Kubernetes master 114 maintains availability of containers by instantiating new instances of a container when another instance fails. However, the Kubernetes master 114 does so without maintaining the state of the failed container, including not maintaining the identity or network address of the failed container.
The orchestrator 112 may take advantage of the services of the Kubernetes master 114 while overcoming its deficiencies according to the methods disclosed herein. In particular, a Kubelet 302 may configure the pod 304 to execute one or more executables that function as agents or interfaces for performing tasks involved in the instantiation and configuration of a container 300. These executables may be programmed to interact with the orchestrator 112 in order to provide orchestration in accordance with the bundled application 202 that would not otherwise be possible within the functionality provided by the Kubernetes master 114 on its own.
These executables may include a container storage interface (CSI) 306 that manages persistent volume claims (PVCs) 308 that define an interface to a storage volume 310, such as a logical storage volume implemented by the storage manager 102 and as described in the incorporated applications.
A container network interface (CNI) 312 is an executable that manages the acquiring of a network address for a container and otherwise manages establishing and maintaining a network connection between the container 300 and other computers on a network 104.
A container runtime interface (CRI) 314 is an executable that configures, stores, or retrieves a runtime state of the container 300 and is called by the Kubelet 302 upon instantiation of the container 300 in order to obtain data defining its state of execution.
As shown in
In some embodiments, a compute node 110 may further execute an orchestrator agent 316. The orchestrator agent 316 communicates with the orchestrator 112 and informs the orchestrator information such as an operating state of the containers 300 and roles 212 executing on the compute node 110 and may further monitor and report on the state of operation of the compute node 110 itself. The orchestrator 112 may therefore detect failure of the compute node 110 based on report of such a failure by the orchestrator agent 316 or in response to failing to receive a report from the orchestrator agent 316 within a timeout period from a time of receipt of a previous report.
Referring to
The method 400 may include the orchestrator 112 receiving 402 the application manifest 204 of a bundled application 202 and generating 404 plans for implementing the computation, storage, and networking for implementing the application manifest 204.
In particular, the computation plan may include identifying compute nodes 110 (which may be hybrid nodes) for hosting containers executing role instances 212 defined by the application manifest 204. The storage plan identifies the storage nodes 106 for hosting storage volumes defined by the application manifest 204. The networking plan defines network addresses to be assigned to containers executing role instances 212 as defined by the application manifest 204.
Generating the plans may be performed with respect to a listing of an available resources. The listing may identify available compute nodes 110 and storage nodes 106, including their storage and performance capacity and location (network domain, server rack, data center, city, etc.). The listing may be compiled by or programmed into a resource manager native to the orchestrator 112 or by means of an API (application programming interface) implemented by the Kubernetes master 114. The listing may be compiled manually or using any network discovery technique known in the art.
Generating 404 the plans may include taking into account anti-affinity constraints in the application manifest 204 that specify that a particular role instance 212 should not be on the same node, in the same network domain, the same data center, or within some other level of affinity with another role instance 212 or storage device 108 hosting a storage volume. For example, where one role instance 212 is a backup for another role instance 212, these may be constrained to be on different nodes or have some other level of anti-affinity to help ensure availability of at least one of them.
Generating the computation plan may include taking into account affinity constraints that specify that a particular role instance 212 should be on the same node, in the same network domain, the same data center, or within some other level of affinity with another role instance 212 or storage device 108 hosting a storage volume. Generating 404 the computation plan and storage plan may be performed according to the approaches of ROBN-01801 and ROBN-02700 referenced in Table 1, above.
The result of step 404 is a mapping between each role instance 212 (such as role instance identifier of the each role instance) specified in the application manifest 204 and a node 110 selected to host that role instance. The result of step 404 may further include a mapping between each storage volume (such as an identifier of the each storage volume) specified in the application manifest 204 and a storage node 106 selected to implement that storage volume. Likewise, the result of step 404 may be listing of network identifiers for each container implementing a role instance 212. The network identifiers may be specified explicitly in the application manifest 204 for each role instance 212 or may be selected based on the subnet masks of domains of the nodes on which the role instances 212 are to be instantiated.
The method 400 may then include creating 406 storage volumes according to the storage plan. This may include implementing any of the approaches described in the incorporated applications for creating a storage volume. In particular, step 406 may include allocating one or more storage devices 108 of a storage node 106 to a storage volume mapped in the storage plan to that storage node 106. This may include configuring the storage node 106 to implement the storage volume and reserve some storage capacity for the storage volume or to allocate portions of a storage device 108 to the storage volume and write data to it in response to write requests addressed to that storage volume as described in the incorporated applications.
The method 400 may include mounting 408 these storage volumes on the compute nodes 110 that will host containers using the storage volumes according to the application manifest 204 and the compute plan of step 404. This may include mounting the storage volumes to the file system of the compute nodes 110 or otherwise making the storage volumes available to be written to and read from on the compute nodes 110.
The method 400 may further include allocating 410 network addresses according to the network plan. In particular, the network plan specifies that particular containers are to be assigned particular network addresses. Accordingly, the orchestrator 112 may acquire these network addresses in the network domains to which they belong, such as according to the domain name service (DNS) protocol. In some embodiments, for a given container, the node 110 that is mapped to that container in the computation plan will be instructed to acquire the IP address for the role instance to be executed by that container.
The method 400 may further include the orchestrator 112 generating 412 pod specifications for the containers of the role instances 212 specified in the application manifest 204. In particular, the pod specification may specify such information as an identifier (e.g., IP address) of the specific compute node 110 on which a container is to be instantiated and a reference to an executable file (e.g., application binary file or files) to be instantiated as the role instance 212. The pod specification may also indicate a type of container to be instantiated, e.g. DOCKER, LXC, LCS, KVM, or the like. The pod specification may further reference a storage volume mounted to the specific compute node 110 as mounted at step 408. The pod specification may further include an identifier of the container as specified in the application manifest 204 or determined according to he compute plan at step 404.
Note that the Kubernetes master 114 is capable of and programmed to select a node and instantiate a container and application instance on it from a set of available nodes. However, the pod specification suppresses this functionality by limiting the set of available nodes to a single node selected at step 404 for the role instance 212 to be installed on that node.
The pod specifications from step 412 may then be submitted 414 to the Kubernetes master 114, which then attempts to implement 416 the pod specifications. Step 414 may include the orchestrator 112 placing jobs in queue of Kubernetes master, each job including one of the pod specifications from step 412.
In particular, for each pod specification, the Kubernetes invokes a Kubelet 302 on the compute node 110 referenced by the pod specification to create a container as specified in the pod specification: install an instance of the container type in the pod specification and load an instance of the role instance 212 referenced by the pod specification. In some embodiments, the pod specification may further reference executables to be used by the Kubelet for the CSI 306, PVC 308, CNI 312, and CRI 314. Accordingly, implementing 416 the pod specification may include invoking some or all of the CSI 306, CNI 312, and CRI 314.
In some embodiments, the CSI 306 performs no substantive functionality, i.e. it is present and executes in order to avoid interfering with the function of the Kubelet 302 but does not actually mount any storage volumes. This may be done in view of the fact that the storage volumes to be used by the container are already mounted to the node 110 referenced by the pod specification (step 408) and that mounted storage volume is described in the pod specification. The container instance on the node 110 may then be configured by the Kubelet 302 to reference the mounted storage volume without additional actions performed by the CSI 306.
The Kubelet 302 may further invoke 418 execution of the CNI 312. The CNI 312 communicates with the orchestrator 112 to obtain the network address assigned to the role instance 212 of the pod specification according to the network plan of step 404. The CNI 312 then performs actions to assign 420 that network address to the compute node 110 on which the container was instantiated according to the pod specification at step 416 and to the container.
Following performing steps 416-420 for the pod specifications generated at step 412, the bundled application 202 is instantiated comprising the containers hosting the role instances implemented at step 416. Note that instantiating the containers and initiating execution may include executing hooks as described in the incorporated applications.
The method 500 may include detecting 504 failure of a pod on a node 110 during the monitoring of step 502. Failure of a pod may include failure of a container managed by the pod or the pod itself. In response, the Kubernetes master 114 may attempt 506 to implement the pod specification from the method 400 that invoked creation of the pod. For example, this may include implementing steps 416-420 as described above with respect to the pod specification. If the attempt of step 506 is found 508 to be successful, new instances of one or more containers according to the pod specification will be instantiated on the same node 110 specified in the pod specification and this container will have the identifier and network address of the failed container and be executing the role instance 212 specified in the pod specification.
If this is not found 508 to be successful or if a node 110 hosting a pod is found 510 to have failed, then some or all of steps 512-524 may be executed. Detecting failure of the node 110 itself may be reported by the Kubernetes master 114 to the orchestrator 112 in response to failing to receive reports from the Kubelet 302 on the node 110 within a timeout period or the orchestrator 112 failing to receive a report from the orchestrator agent 316 executing on the node 110 within a timeout period. Note that the Kubernetes master 114 is capable of invoking creation of a replacement pod in response to failure of a node hosting a pod. However, inasmuch as the pod specification submitted to the Kubernetes master 114 specifies a single node for a pod, this functionality is suppressed since no other node will satisfy the constraint of the pod specification.
In response to detecting failure of the node 110, the orchestrator 112 may generate 512 a compute plan for the role instances 212 that were executing within containers on the failed node 110. This may be performed as described above with respect to step 404 of the method 400 and may include identifying compute nodes 110 with capacity to host the role instances 212 and satisfying any affinity or anti-affinity constraints. Note that where the failed node 110 was also the storage node 106 for a storage volume used by the container executing on it, step 512 may include identifying a new storage node 106 with sufficient storage capacity in the storage devices 108 thereof to implement the storage volume.
In the case that the affinity constraints require the container and storage volume to be on the same node, step 512 may include identifying a node 110 that also include storage devices 108 with sufficient storage capacity to implement the storage volume. Step 512 differs from the approach of the method 400 inasmuch as many of the compute nodes 110 and storage nodes 106 may already be allocated to other containers or storage volumes. Accordingly, step 512 may include identifying those nodes 110, 106 with available capacity and selecting those that meet affinity and anti-affinity constraints for the one or more role instances 212 of the failed node 110.
The method 500 may further include mounting 514 a storage volume to the new node 110 selected for the failed pod according to the compute plan of step 512. This may be performed as described above with respect to step 410 of the method 400. Where the storage volume was not implemented by the failed node 110, the same storage volume that was mounted at step 408 for the failed pod may be mounted to the new node 110 at step 514. Where the failed node 110 also implemented the storage volume, a new storage volume may be created on the new node 110 or a different storage node 106 selected according to affinity and anti-affinity constraints.
The method 500 may include deleting 516 the failed pod from the Kubernetes installation, such as by the orchestrator 112 instructing the Kubernetes to delete the pod specification that was used to implement the pod on the failed node 110.
The method 500 may then include generating 518 a new pod specification and submitting 520 the new pod specification to the Kubernetes master 114. For example, the pod specification may be identical to that which was submitted to the failed node 110 to invoke creation of the pod on the failed node 110 at step 414 except that the node 110 referenced by the pod specification is changed to be the new node 110 selected according to the compute plan of step 512. Likewise, the new pod specification may specify a mounting point for a storage volume on the new node, which may be the same or different from the original pod specification.
The Kubernetes master 114 may then invoke implementing 522 of the pod specification on the new node. This may include invoking 524 the CNI 312, which coordinates with the orchestrator 112 to assign 526 an IP address to the container created according to the pod specification as described above with respect to steps 418 and 420 of the method 400.
In the case of a pod specification to replace a pod executing on a failed node 110, step 522 may include the Kubelet on the new node 110 invoking the CRI 314. The CRI 314 will coordinate with the orchestrator 112 to restore a state of execution of the container on the failed node 110. Note that the CRI 314 may be invoked by the Kubelet 302 upon instantiation of the original container to load an initial state of the original container in the same manner, such as according to a specification in the application manifest 204.
In particular, a root directory of the container on the failed node 110 may be configured to be a storage volume hosted by a different node, such as a storage node 106. Changes to the state of the container prior on the failed node 110 prior to failure may then be written to the storage volume by the container. Accordingly, when the new node 110 is selected and a new container is instantiated thereon at step 522, the CRI 314 may be communicate with the orchestrator 112, which instructs the CRI 314 to use the state stored on the storage volume to configure the initial state of the new container. The manner in which a container may be configured to maintain its state in a remote storage volume may be performed as described in ROBN-01900 (see listing of incorporated applications in Table 1). Examples of the state of operation of a container that may be stored on the storage volume and used to set the state of the new container may include such things as configuration settings saved on root disk, which may be a remote storage volume used and configured in the manner described in ROBN-01900. For an apache http server, configuration settings could be settings in/var/www/httpd.conf. For mysql, configuration settings could be those typically stored in /var/lib/mysql.conf. For Cassandra, configuration settings could be those typically stored in /etc/cassandra/cassandra.yaml.
Computing device 600 includes one or more processor(s) 602, one or more memory device(s) 604, one or more interface(s) 606, one or more mass storage device(s) 608, one or more Input/output (I/O) device(s) 610, and a display device 630 all of which are coupled to a bus 612. Processor(s) 602 include one or more processors or controllers that execute instructions stored in memory device(s) 604 and/or mass storage device(s) 608. Processor(s) 602 may also include various types of computer-readable media, such as cache memory.
Memory device(s) 604 include various computer-readable media, such as volatile memory (e.g., random access memory (RAM) 614) and/or nonvolatile memory (e.g., read-only memory (ROM) 616). Memory device(s) 604 may also include rewritable ROM, such as Flash memory.
Mass storage device(s) 608 include various computer readable media, such as magnetic tapes, magnetic disks, optical disks, solid-state memory (e.g., Flash memory), and so forth. As shown in
I/O device(s) 610 include various devices that allow data and/or other information to be input to or retrieved from computing device 600. Example I/O device(s) 610 include cursor control devices, keyboards, keypads, microphones, monitors or other display devices, speakers, printers, network interface cards, modems, lenses, CCDs or other image capture devices, and the like.
Display device 630 includes any type of device capable of displaying information to one or more users of computing device 600. Examples of display device 630 include a monitor, display terminal, video projection device, and the like.
Interface(s) 606 include various interfaces that allow computing device 600 to interact with other systems, devices, or computing environments. Example interface(s) 606 include any number of different network interfaces 620, such as interfaces to local area networks (LANs), wide area networks (WANs), wireless networks, and the Internet. Other interface(s) include user interface 618 and peripheral device interface 622. The interface(s) 606 may also include one or more peripheral interfaces such as interfaces for printers, pointing devices (mice, track pad, etc.), keyboards, and the like.
Bus 612 allows processor(s) 602, memory device(s) 604, interface(s) 606, mass storage device(s) 608, I/O device(s) 610, and display device 630 to communicate with one another, as well as other devices or components coupled to bus 612. Bus 612 represents one or more of several types of bus structures, such as a system bus, PCI bus, IEEE 1394 bus, USB bus, and so forth.
For purposes of illustration, programs and other executable program components are shown herein as discrete blocks, although it is understood that such programs and components may reside at various times in different storage components of computing device 600, and are executed by processor(s) 602. Alternatively, the systems and procedures described herein can be implemented in hardware, or a combination of hardware, software, and/or firmware. For example, one or more application specific integrated circuits (ASICs) can be programmed to carry out one or more of the systems and procedures described herein.
In the above disclosure, reference has been made to the accompanying drawings, which form a part hereof, and in which is shown by way of illustration specific implementations in which the disclosure may be practiced. It is understood that other implementations may be utilized and structural changes may be made without departing from the scope of the present disclosure. References in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.
Implementations of the systems, devices, and methods disclosed herein may comprise or utilize a special purpose or general-purpose computer including computer hardware, such as, for example, one or more processors and system memory, as discussed herein. Implementations within the scope of the present disclosure may also include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures. Such computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer system. Computer-readable media that store computer-executable instructions are computer storage media (devices). Computer-readable media that carry computer-executable instructions are transmission media. Thus, by way of example, and not limitation, implementations of the disclosure can comprise at least two distinctly different kinds of computer-readable media: computer storage media (devices) and transmission media.
Computer storage media (devices) includes RAM, ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM), Flash memory, phase-change memory (“PCM”), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.
An implementation of the devices, systems, and methods disclosed herein may communicate over a computer network. A “network” is defined as one or more data links that enable the transport of electronic data between computer systems and/or modules and/or other electronic devices. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer, the computer properly views the connection as a transmission medium. Transmissions media can include a network and/or data links, which can be used to carry desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. Combinations of the above should also be included within the scope of computer-readable media.
Computer-executable instructions comprise, for example, instructions and data which, when executed at a processor, cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or acts described above. Rather, the described features and acts are disclosed as example forms of implementing the claims.
Those skilled in the art will appreciate that the disclosure may be practiced in network computing environments with many types of computer system configurations, including, an in-dash vehicle computer, personal computers, desktop computers, laptop computers, message processors, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, tablets, pagers, routers, switches, various storage devices, and the like. The disclosure may also be practiced in distributed system environments where local and remote computer systems, which are linked (either by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) through a network, both perform tasks. In a distributed system environment, program modules may be located in both local and remote memory storage devices.
Further, where appropriate, functions described herein can be performed in one or more of: hardware, software, firmware, digital components, or analog components. For example, one or more application specific integrated circuits (ASICs) can be programmed to carry out one or more of the systems and procedures described herein. Certain terms are used throughout the description and claims to refer to particular system components. As one skilled in the art will appreciate, components may be referred to by different names. This document does not intend to distinguish between components that differ in name, but not function.
It should be noted that the sensor embodiments discussed above may comprise computer hardware, software, firmware, or any combination thereof to perform at least a portion of their functions. For example, a sensor may include computer code configured to be executed in one or more processors, and may include hardware logic/electrical circuitry controlled by the computer code. These example devices are provided herein purposes of illustration, and are not intended to be limiting. Embodiments of the present disclosure may be implemented in further types of devices, as would be known to persons skilled in the relevant art(s).
At least some embodiments of the disclosure have been directed to computer program products comprising such logic (e.g., in the form of software) stored on any computer useable medium. Such software, when executed in one or more data processing devices, causes a device to operate as described herein.
While various embodiments of the present disclosure have been described above, it should be understood that they have been presented by way of example only, and not limitation. It will be apparent to persons skilled in the relevant art that various changes in form and detail can be made therein without departing from the spirit and scope of the disclosure. Thus, the breadth and scope of the present disclosure should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents. The foregoing description has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. Further, it should be noted that any or all of the aforementioned alternate implementations may be used in any combination desired to form additional hybrid implementations of the disclosure.
| Number | Name | Date | Kind |
|---|---|---|---|
| 3715573 | Vogelsberg | Feb 1973 | A |
| 4310883 | Clifton | Jan 1982 | A |
| 5602993 | Stromberg | Feb 1997 | A |
| 5680513 | Hyland | Oct 1997 | A |
| 5796290 | Takahashi | Aug 1998 | A |
| 6014669 | Slaughter | Jan 2000 | A |
| 6052797 | Ofek | Apr 2000 | A |
| 6119214 | Dirks | Sep 2000 | A |
| 6157963 | Courtright, II | Dec 2000 | A |
| 6161191 | Slaughter | Dec 2000 | A |
| 6298478 | Nally | Oct 2001 | B1 |
| 6301707 | Carroll | Oct 2001 | B1 |
| 6311193 | Sekido | Oct 2001 | B1 |
| 6851034 | Challenger | Feb 2005 | B2 |
| 6886160 | Lee | Apr 2005 | B1 |
| 6895485 | Dekoning | May 2005 | B1 |
| 6957221 | Hart | Oct 2005 | B1 |
| 7096465 | Dardinski | Aug 2006 | B1 |
| 7111055 | Falkner | Sep 2006 | B2 |
| 7171659 | Becker | Jan 2007 | B2 |
| 7246351 | Bloch | Jul 2007 | B2 |
| 7305671 | Davidov | Dec 2007 | B2 |
| 7461374 | Balint | Dec 2008 | B1 |
| 7535854 | Luo | May 2009 | B2 |
| 7590620 | Pike | Sep 2009 | B1 |
| 7698698 | Skan | Apr 2010 | B2 |
| 7721283 | Kovachka | May 2010 | B2 |
| 7734859 | Daniel | Jun 2010 | B2 |
| 7738457 | Nordmark | Jun 2010 | B2 |
| 7779091 | Wilkinson | Aug 2010 | B2 |
| 7797693 | Gustafson | Sep 2010 | B1 |
| 7984485 | Rao | Jul 2011 | B1 |
| 8037471 | Keller | Oct 2011 | B2 |
| 8046450 | Schloss | Oct 2011 | B1 |
| 8060522 | Birdwell | Nov 2011 | B2 |
| 8121874 | Guheen | Feb 2012 | B1 |
| 8171141 | Offer | May 2012 | B1 |
| 8219821 | Zimmels | Jul 2012 | B2 |
| 8250033 | De Souter | Aug 2012 | B1 |
| 8261295 | Risbood | Sep 2012 | B1 |
| 8326883 | Pizzorni | Dec 2012 | B2 |
| 8392498 | Berg | Mar 2013 | B2 |
| 8429346 | Chen | Apr 2013 | B1 |
| 8464241 | Hayton | Jun 2013 | B2 |
| 8505003 | Bowen | Aug 2013 | B2 |
| 8527544 | Colgrove | Sep 2013 | B1 |
| 8601467 | Hofhansl | Dec 2013 | B2 |
| 8620973 | Veeraswamy | Dec 2013 | B1 |
| 8666933 | Pizzorni | Mar 2014 | B2 |
| 8745003 | Patterson | Jun 2014 | B1 |
| 8775751 | Pendharkar | Jul 2014 | B1 |
| 8782632 | Chigurapati | Jul 2014 | B1 |
| 8788634 | Krig | Jul 2014 | B2 |
| 8832324 | Hodges | Sep 2014 | B1 |
| 8886806 | Tung | Nov 2014 | B2 |
| 8909885 | Corbett | Dec 2014 | B2 |
| 8954383 | Vempati | Feb 2015 | B1 |
| 8954568 | Krishnan | Feb 2015 | B2 |
| 8966198 | Harris | Feb 2015 | B1 |
| 9134992 | Wong | Sep 2015 | B2 |
| 9146769 | Shankar | Sep 2015 | B1 |
| 9148465 | Gambardella | Sep 2015 | B2 |
| 9152337 | Kono | Oct 2015 | B2 |
| 9167028 | Bansal | Oct 2015 | B1 |
| 9280591 | Kharatishvili | Mar 2016 | B1 |
| 9330155 | Bono | May 2016 | B1 |
| 9336060 | Nori | May 2016 | B2 |
| 9342444 | Minckler | May 2016 | B2 |
| 9367301 | Serrano | Jun 2016 | B1 |
| 9390128 | Seetala | Jul 2016 | B1 |
| 9436693 | Lockhart | Sep 2016 | B1 |
| 9514160 | Song | Dec 2016 | B2 |
| 9521198 | Agarwala | Dec 2016 | B1 |
| 9569274 | Tarta | Feb 2017 | B2 |
| 9569480 | Provencher | Feb 2017 | B2 |
| 9600193 | Ahrens | Mar 2017 | B2 |
| 9613119 | Aron | Apr 2017 | B1 |
| 9619389 | Roug | Apr 2017 | B1 |
| 9635132 | Lin | Apr 2017 | B1 |
| 9667470 | Prathipati | May 2017 | B2 |
| 9747096 | Searlee | Aug 2017 | B2 |
| 9870366 | Duan | Jan 2018 | B1 |
| 9880933 | Gupta | Jan 2018 | B1 |
| 9892265 | Tripathy | Feb 2018 | B1 |
| 9929916 | Subramanian | Mar 2018 | B1 |
| 9998955 | MacCarthaigh | Jun 2018 | B1 |
| 10019459 | Agarwala | Jul 2018 | B1 |
| 10042628 | Thompson | Aug 2018 | B2 |
| 10061520 | Zhao | Aug 2018 | B1 |
| 10191778 | Yang | Jan 2019 | B1 |
| 10241774 | Spivak | Mar 2019 | B2 |
| 10282229 | Wagner | May 2019 | B2 |
| 10353634 | Greenwood | Jul 2019 | B1 |
| 10430434 | Sun | Oct 2019 | B2 |
| 10657119 | Acheson | May 2020 | B1 |
| 10956246 | Bagde | Mar 2021 | B1 |
| 20020141390 | Fangman | Oct 2002 | A1 |
| 20040010716 | Childress | Jan 2004 | A1 |
| 20040153703 | Vigue | Aug 2004 | A1 |
| 20040221125 | Ananthanarayanan | Nov 2004 | A1 |
| 20050065986 | Bixby | Mar 2005 | A1 |
| 20050216895 | Tran | Sep 2005 | A1 |
| 20050256948 | Hu | Nov 2005 | A1 |
| 20060025908 | Rachlin | Feb 2006 | A1 |
| 20060053357 | Rajski | Mar 2006 | A1 |
| 20060085674 | Ananthamurthy | Apr 2006 | A1 |
| 20060259686 | Sonobe | Nov 2006 | A1 |
| 20070006015 | Rao | Jan 2007 | A1 |
| 20070016786 | Waltermann | Jan 2007 | A1 |
| 20070067583 | Zohar | Mar 2007 | A1 |
| 20070165625 | Eisner | Jul 2007 | A1 |
| 20070260842 | Faibish | Nov 2007 | A1 |
| 20070277056 | Varadarajan | Nov 2007 | A1 |
| 20070288791 | Allen | Dec 2007 | A1 |
| 20080010421 | Chen | Jan 2008 | A1 |
| 20080068899 | Ogihara | Mar 2008 | A1 |
| 20080189468 | Schmidt | Aug 2008 | A1 |
| 20080235544 | Lai | Sep 2008 | A1 |
| 20080256167 | Branson | Oct 2008 | A1 |
| 20080263400 | Waters | Oct 2008 | A1 |
| 20080270592 | Choudhary | Oct 2008 | A1 |
| 20090144497 | Withers | Jun 2009 | A1 |
| 20090172335 | Kulkarni | Jul 2009 | A1 |
| 20090240809 | La Frese | Sep 2009 | A1 |
| 20090254701 | Kurokawa | Oct 2009 | A1 |
| 20090307249 | Koifman | Dec 2009 | A1 |
| 20100100251 | Chao | Apr 2010 | A1 |
| 20100161941 | Vyshetsky | Jun 2010 | A1 |
| 20100162233 | Ku | Jun 2010 | A1 |
| 20100211815 | Mankovskii | Aug 2010 | A1 |
| 20100274984 | Inomata | Oct 2010 | A1 |
| 20100299309 | Maki | Nov 2010 | A1 |
| 20100306495 | Kumano | Dec 2010 | A1 |
| 20100332730 | Royer | Dec 2010 | A1 |
| 20110083126 | Bhakta | Apr 2011 | A1 |
| 20110119664 | Kimura | May 2011 | A1 |
| 20110188506 | Arribas | Aug 2011 | A1 |
| 20110208928 | Chandra | Aug 2011 | A1 |
| 20110246420 | Wang | Oct 2011 | A1 |
| 20110276951 | Jain | Nov 2011 | A1 |
| 20120005557 | Mardiks | Jan 2012 | A1 |
| 20120066449 | Colgrove | Mar 2012 | A1 |
| 20120102369 | Hiltunen | Apr 2012 | A1 |
| 20120216052 | Dunn | Aug 2012 | A1 |
| 20120226667 | Volvovski | Sep 2012 | A1 |
| 20120240012 | Weathers | Sep 2012 | A1 |
| 20120259819 | Patwardhan | Oct 2012 | A1 |
| 20120265976 | Spiers | Oct 2012 | A1 |
| 20120303348 | Lu | Nov 2012 | A1 |
| 20120311671 | Wood | Dec 2012 | A1 |
| 20120331113 | Jain | Dec 2012 | A1 |
| 20130054552 | Hawkins | Feb 2013 | A1 |
| 20130054932 | Acharya | Feb 2013 | A1 |
| 20130080723 | Sawa | Mar 2013 | A1 |
| 20130254521 | Bealkowski | Sep 2013 | A1 |
| 20130282662 | Kumarasamy | Oct 2013 | A1 |
| 20130332688 | Corbett | Dec 2013 | A1 |
| 20130339659 | Bybell | Dec 2013 | A1 |
| 20130346709 | Wang | Dec 2013 | A1 |
| 20140006465 | Davis | Jan 2014 | A1 |
| 20140047263 | Coatney | Feb 2014 | A1 |
| 20140047341 | Breternitz | Feb 2014 | A1 |
| 20140047342 | Breternitz | Feb 2014 | A1 |
| 20140058871 | Marr | Feb 2014 | A1 |
| 20140059527 | Gagliardi | Feb 2014 | A1 |
| 20140059528 | Gagliardi | Feb 2014 | A1 |
| 20140108483 | Tarta | Apr 2014 | A1 |
| 20140130040 | Lemanski | May 2014 | A1 |
| 20140195847 | Webman | Jul 2014 | A1 |
| 20140245319 | Fellows | Aug 2014 | A1 |
| 20140281449 | Christopher | Sep 2014 | A1 |
| 20140282596 | Bourbonnais | Sep 2014 | A1 |
| 20150046644 | Karp | Feb 2015 | A1 |
| 20150067031 | Acharya | Mar 2015 | A1 |
| 20150074358 | Flinsbaugh | Mar 2015 | A1 |
| 20150112951 | Narayanamurthy et al. | Apr 2015 | A1 |
| 20150134857 | Hahn | May 2015 | A1 |
| 20150149605 | de la Iglesia | May 2015 | A1 |
| 20150186217 | Eslami | Jul 2015 | A1 |
| 20150278333 | Hirose | Oct 2015 | A1 |
| 20150317212 | Lee | Nov 2015 | A1 |
| 20150326481 | Rector | Nov 2015 | A1 |
| 20150379287 | Mathur | Dec 2015 | A1 |
| 20160011816 | Aizman | Jan 2016 | A1 |
| 20160026667 | Mukherjee | Jan 2016 | A1 |
| 20160042005 | Liu | Feb 2016 | A1 |
| 20160124775 | Ashtiani | May 2016 | A1 |
| 20160197995 | Lu | Jul 2016 | A1 |
| 20160239412 | Wada | Aug 2016 | A1 |
| 20160259597 | Worley | Sep 2016 | A1 |
| 20160283261 | Nakatsu | Sep 2016 | A1 |
| 20160357456 | Iwasaki | Dec 2016 | A1 |
| 20160357548 | Stanton | Dec 2016 | A1 |
| 20160373327 | Degioanni | Dec 2016 | A1 |
| 20170060710 | Ramani | Mar 2017 | A1 |
| 20170060975 | Akyureklier | Mar 2017 | A1 |
| 20170139645 | Byun | May 2017 | A1 |
| 20170149843 | Amulothu | May 2017 | A1 |
| 20170168903 | Dornemann | Jun 2017 | A1 |
| 20170192889 | Sato | Jul 2017 | A1 |
| 20170214550 | Kumar | Jul 2017 | A1 |
| 20170235649 | Shah | Aug 2017 | A1 |
| 20170242617 | Walsh | Aug 2017 | A1 |
| 20170242719 | Tsirkin | Aug 2017 | A1 |
| 20170244557 | Riel | Aug 2017 | A1 |
| 20170244787 | Rangasamy | Aug 2017 | A1 |
| 20170322954 | Horowitz | Nov 2017 | A1 |
| 20170337492 | Chen | Nov 2017 | A1 |
| 20170371551 | Sachdev | Dec 2017 | A1 |
| 20180006896 | Macnamara | Jan 2018 | A1 |
| 20180024889 | Verma | Jan 2018 | A1 |
| 20180046553 | Okamoto | Feb 2018 | A1 |
| 20180082053 | Brown | Mar 2018 | A1 |
| 20180107419 | Sachdev | Apr 2018 | A1 |
| 20180113625 | Sancheti | Apr 2018 | A1 |
| 20180113770 | Hasanov | Apr 2018 | A1 |
| 20180136931 | Hendrich | May 2018 | A1 |
| 20180137306 | Brady | May 2018 | A1 |
| 20180150306 | Govindaraju | May 2018 | A1 |
| 20180159745 | Byers | Jun 2018 | A1 |
| 20180165170 | Hegdal | Jun 2018 | A1 |
| 20180218000 | Setty | Aug 2018 | A1 |
| 20180225216 | Filippo | Aug 2018 | A1 |
| 20180246670 | Baptist | Aug 2018 | A1 |
| 20180246745 | Aronovich | Aug 2018 | A1 |
| 20180247064 | Aronovich | Aug 2018 | A1 |
| 20180276215 | Chiba | Sep 2018 | A1 |
| 20180285164 | Hu | Oct 2018 | A1 |
| 20180285223 | McBride | Oct 2018 | A1 |
| 20180285353 | Ramohalli | Oct 2018 | A1 |
| 20180287883 | Joshi | Oct 2018 | A1 |
| 20180302335 | Gao | Oct 2018 | A1 |
| 20180329981 | Gupte | Nov 2018 | A1 |
| 20180364917 | Ki | Dec 2018 | A1 |
| 20180375728 | Gangil | Dec 2018 | A1 |
| 20190065061 | Kim | Feb 2019 | A1 |
| 20190065323 | Dhamdhere | Feb 2019 | A1 |
| 20190073132 | Zhou | Mar 2019 | A1 |
| 20190073372 | Venkatesan | Mar 2019 | A1 |
| 20190079928 | Kumar | Mar 2019 | A1 |
| 20190089651 | Pignatari | Mar 2019 | A1 |
| 20190102226 | Caldato | Apr 2019 | A1 |
| 20190109756 | Abu Lebdeh | Apr 2019 | A1 |
| 20190116690 | Chen | Apr 2019 | A1 |
| 20190148932 | Benesch | May 2019 | A1 |
| 20190156023 | Gerebe | May 2019 | A1 |
| 20190163460 | Kludy | May 2019 | A1 |
| 20190188094 | Ramamoorthi | Jun 2019 | A1 |
| 20190190803 | Joshi | Jun 2019 | A1 |
| 20190199601 | Lynar | Jun 2019 | A1 |
| 20190213085 | Alluboyina | Jul 2019 | A1 |
| 20190215313 | Doshi | Jul 2019 | A1 |
| 20190220266 | Doshi | Jul 2019 | A1 |
| 20190220315 | Vallala | Jul 2019 | A1 |
| 20190235895 | Ovesea | Aug 2019 | A1 |
| 20190250849 | Compton | Aug 2019 | A1 |
| 20190272205 | Jiang | Sep 2019 | A1 |
| 20190278624 | Bade | Sep 2019 | A1 |
| 20190324666 | Kusters | Oct 2019 | A1 |
| 20190334727 | Kaufman | Oct 2019 | A1 |
| 20190369273 | Liu | Dec 2019 | A1 |
| 20200019414 | Byard | Jan 2020 | A1 |
| 20200034193 | Jayaram | Jan 2020 | A1 |
| 20200034254 | Natanzon | Jan 2020 | A1 |
| 20200083909 | Kusters | Mar 2020 | A1 |
| 20200150977 | Wang | May 2020 | A1 |
| 20200257519 | Shen | Aug 2020 | A1 |
| 20200356537 | Sun | Nov 2020 | A1 |
| 20210042151 | Muller | Feb 2021 | A1 |
| 20210067607 | Gardner | Mar 2021 | A1 |
| Number | Date | Country |
|---|---|---|
| WO2017008675 | Jan 2017 | WO |
| Entry |
|---|
| Segment map, Feb. 4, 2019. |
| Implementing time critical functionalities with a distributed adaptive container architecture, Stankovski, Nov. 2016. |
| Segment map, Google, Feb. 4, 2019. |
| Fast and Secure Append-Only storage with Infinite Capacity, Zheng, Aug. 27, 2003. |
| User Mode and Kernel Mode, Microsoft, Apr. 19, 2017. |
| Precise memory leak detection for java software using container profiling, Xu, Jul. 2013. |
| Mogi et al., “Dynamic Parity Stripe Reorganizations for RAID5 Disk Arrays,” 1994, IEEE, pp. 17-26. |
| Syed et al., “The Container Manager Pattern”, ACM, pp. 1-9 (Year 2017). |
| Rehmann et al., “Performance of Containerized Database Management Systems”, ACM, pp. 1-6 (Year 2018). |
| Awada et al., “Improving Resource Efficiency of Container-instance Clusters on Clouds”, IEEE, pp. 929-934 (Year 2017). |
| Stankovski et al, “Implementing Time-Critical Functionalities with a Distributed Adaptive Container Architecture”, ACM, pp. 1-5 (Year 2016). |
| Dhakate et al, “Distributed Cloud Monitoring Using Docker as Next Generation Container Virtualization Technology” IEEE, pp. 1-5 (Year 2015). |
| Crameri et al, “Staged Deployment in Mirage, an Integrated Software Upgrade Testing and Distribution System”, ACM, pp. 221-236 (Year: 2007). |
| Cosmo et al, “Packages Upgrades in FOSS Distributions: Details and Challenges”, AC 2008). |
| Burg et al, “Atomic Upgrading of Distributed Systems”, ACM, pp. 1-5 (Year: 2008). |
| Souer et al, “Component Based Architecture forWeb Content Management: Runtime Deployable Web Manager Component Bundles”, IEEE, pp. 366-369 (Year: 2008). |
| Weingartner et al, “A distributed autonomic management framework for cloud computing orchestration.” In 2016 IEEE World Congress on Services (Year: 2016). |
| Number | Date | Country | |
|---|---|---|---|
| 20210064442 A1 | Mar 2021 | US |
