Patent Grant
10756979
This application claims priority from Chinese Patent Application Number CN201510004936.6, filed on Jan. 4, 2015 entitled “METHOD AND APPARATUS FOR PERFORMING CROSS-LAYER ORCHESTRATION OF RESOURCES IN DATA CENTER HAVING MULTI-LAYER ARCHITECTURE” the content and teachings of which is herein incorporated by reference in its entirety.
Embodiments of the present disclosure generally relate to a method and apparatus for a data center, and more particularly to a method and apparatus for performing cross-layer orchestration of resources in a data center having a multi-layer architecture.
A typical 3rd platform data center usually uses cloud technologies and has a multi-layer architecture. Simply put, the 3rd platform data center is usually divided into four layers: application layer (Apps layer), Platform as a Service layer, Infrastructure as a Service layer, and a Bare-metal layer.
The topology of real world's cloud data center would be much more complex than above description. However, the hardest thing is each layer being based on and affected by the upper layer or under layer, even cross layers. A general 3rd platform application requires: Real timely and elastically response to the request change, including scale-out when the request rapidly increasing, and scale-in at once request storm over; Capabilities requirement change according to the business evolution. For example, a small sales system requires I/O on database; but when the requests number increase, it is necessary to add more load balance layer, cache layer, and more calculation to promote the right products to potential customers, the application turns to CPU intensive type.
Former resource orchestration approaches are not regarding the data center as layered and not considering the resource allocation globally. Most of the former approaches in this area called “Orchestration Engine”, but only focus on a sub-part of a data center. In addition, most existing orchestration tools only focus on low-level feature exposing.
Therefore, when the requirement changes, how to orchestrate the layered cloud data center in a unified way in order to provide suitable capabilities immediately is a problem to be solved.
In view of the above problems existing in the prior art, an object of embodiments of the present invention is to provide a higher level orchestration than methods in the prior art, and employ some functions provided by methods in the prior art to provide a unified manner when the demand changes for orchestrating a layered cloud data center, in order to immediately provide a suitable capability.
A first aspect of the present invention provides a method for performing cross-layer orchestration of resources in a data center having a multi-layer architecture. The method comprises: performing unified control of all resources in all layers of the data center; performing unified storage of all topologies and machine-generated data of all layers of the data center; and orchestrating the resources of the data center based on the unified control and the unified storage.
According to an embodiment of the present invention, performing unified control of all resources in all layers of the data center comprises: forming a single control plane for all layers of the data center.
According to an embodiment of the present invention, performing unified control of all resources in all layers of the data center comprises performing unified control of allocation and reclaim of all resources of all layers.
According to an embodiment of the present invention, the unified storage of all topologies and machine-generated data of all layers of the data center is performed so as to determine full status of all layers of the data center.
According to an embodiment of the present invention, the full status is exposed as a query service and accessible by an external system.
According to an embodiment of the present invention, orchestrating the resources of the data center based on the unified control and the unified storage comprises: in response to requests of one or more layers for resources, performing unified allocation of the resources of the data center based on overall consideration of the data center.
According to an embodiment of the present invention, orchestrating the resources of the data center based on the unified control and the unified storage comprises: finding insufficiency or idleness of resources, and performing unified allocation or reclaim of the resources of the data center based on overall consideration of the data center.
According to an embodiment of the present invention, the unified control for all resources in all layers of the data center and the unified storage for all topologies and machine-generated data of all layers of the data center are implemented through a remote procedure call (RPC) via a message bus.
According to an embodiment of the present invention, the multi-layer architecture at least comprises: a bare-metal layer, an Infrastructure as a Service layer, a Platform as a Service layer or an application layer.
According to an embodiment of the present invention, all resources of all layers at least comprise: hardware resources in the bare-metal layer; virtual machine resources in the Infrastructure as a Service layer; resources in the Platform as a Service layer for developing, running and managing applications; or application resources in the application layer.
According to an embodiment of the present invention, all machine-generated data in all layers of the data center at least comprises: control plane data as static data or meta-data of the whole data center, and data plane data gathered from the running system.
According to an embodiment of the present invention, each layer of the multiple layers of the data center exposes its capabilities out through an interface.
According to an embodiment of the present invention, performing unified control of all resources in all layers of the data center comprises: evaluating all resources of all layers of the data center and grouping them into different service level arrangement (SLA) groups.
According to an embodiment of the present invention, evaluating all resources of all layers of the data center and grouping them into different service level arrangement (SLA) groups further comprises: manually defining in a deployment phase, or defining policies according to the data plane data.
According to an embodiment of the present invention, the method is based on task-oriented processes.
According to an embodiment of the present invention, the task-oriented processes at least comprise: an installation process from a bare-metal layer to an operating system, an installation process from the operation system to an Infrastructure as a Service layer, a process of installing and configuring storages, or an installation process from the Infrastructure as a Service layer to the Platform as a Service layer.
According to an embodiment of the present invention, the task-oriented processes can be combined or integrated in a cross-layer manner.
According to an embodiment of the present invention, the method further comprises: storing diversity information of the data center architecture, and managing diversity of the data center architecture.
According to an embodiment of the present invention, a unified user interface is provided for the diverse data center architecture.
According to an embodiment of the present invention, the method is based on a solution design of multiple layers.
According to an embodiment of the present invention, the multiple layers at least comprise: a core logic layer, which is designed to highly abstract detail operations, co-ordinate different third-party orchestration tools, and provide a unified interface from a bare-metal, an Infrastructure as a Service layer, a Platform as a Service layer to an application level; a third-party orchestration tools layer, which is designed to use various third-party orchestration tools, catalog the third-party orchestration tools, and apply a plugin-design to each catalog; and a layer of different processes, which is designed to select and combine the task-oriented processes as a plugin.
A second aspect of the present invention provides an apparatus for performing cross-layer orchestration of resources in a data center having a multi-layer architecture. The apparatus comprises: a control unit configured to perform unified control of all resources in all layers of the data center; a storage unit configured to perform unified storage of all topologies and machine-generated data of all layers of the data center; and an orchestration unit configured to orchestrate the resources of the data center based on the unified control and the unified storage.
A third aspect of the present invention provides a computer program product. The computer program product comprises program codes that cause, when executed on a processor, the processor to execute the method according to the first aspect of the present invention.
In one word, different from the mechanism in methods of the prior art, embodiments of the present invention regard a data center as layered and consider resource allocation globally. Embodiments of the present invention have the following advantages as compared with methods in the prior art: global orchestration, design for the orchestration of all the data center layers, compatibility with various solutions, and declarative resource requests, etc.
The above and other objects, features and advantages of embodiments of the present invention will be made readily understood by reading through the following detailed description with reference to the accompanying figures. In the figures, several embodiments of the present invention are illustrated in an exemplary but not restrictive manner, wherein:
The principles and spirit of the present invention are described below with reference to several exemplary embodiments shown in the figures. It should be appreciated that the description of these embodiments are only intended to enable those skilled in the art to better understand and implement the present invention, not to limit the scope of the present invention in any manner.
A typical 3rd platform data center usually uses cloud technologies and has a multi-layer architect Simply put, the 3rd platform data center is usually divided into four layers: application layer (Apps layer), Platform as a Service layer, Infrastructure as a Service layer, and a Bare-metal layer. The four layers will be briefly introduced as follows.
The application layer stands for applications deployed by customers for different tasks, e.g., HRIS, eMail, ERP, CRM, BI or real-time analytic applications, as well as some mobile applications backend for third platform. Each application may have multiple instances according to the instantaneous workload, and intelligently scale-out/scale-in capabilities. Different applications require various capabilities to serve. For example, some applications are CPU intensive type, some request heavy I/O, and some require high-security, which need network isolation.
The Platform as a Service layer is a cloud-based computing environment designed for rapid development, running and management of applications. It provides the running environment of programming languages, dependent libraries, data services, and tools for facilitating the customer's application development and deployment. Pivotal Cloud Foundry is one of the most popular “Platform as a Service (PaaS)” systems.
The Infrastructure as a Service layer is a provision model in which an organization outsources the equipment used to support operations, including servers, networks components and storages. In most case of the cloud data center, Infrastructure as a Service (IaaS) usually based on some virtualization technology, and provides the capabilities to PaaS. A common integration way of IaaS and PaaS, is installed PaaS on the resources provided by IaaS: the components of PaaS installed onto the virtual machines created by IaaS; the storage used to persistent data services or application golden images are the block or file storage from IaaS; the components of PaaS, or even the application communication each other with the virtual network created by IaaS, to make sure isolation. In this model, the PaaS benefits from the advantages of IaaS, such as elastic, high available, migration convenience and more. Quality of Service (QoS) is one of the key features of an IaaS to abstract diversity capabilities of under layers, and crux of numerous and heterogeneous strategies to VM placement and cloud federation.
The bare-metal layer generally refers to the hardware infrastructures. Since the present disclosure focuses on the cloud environment, a lot of details of hardware infrastructures are omitted and generalized them as bare-metal layer to make the data center architect easy to understand. In fact, it includes: CPUs, memories provides by server clusters; Layer1-Layer7 network protocols and network components; storage area networking (SAN) storage, server embed-storage, and more, which we can clarify from model 1 to model 4, as well as various file systems. All these hardware provides different service level agreement (SLA) capabilities. A qualified IaaS will abstract these differentials to QoS.
The topology of real world's cloud data center would be much more complex than above description. However, the hardest thing is each layer being based on and affected by the upper layer or under layer, even cross layers. A general 3rd platform application requires: Real timely and elastically response to the request change, including scale-out when the request rapidly increasing, and scale-in at once request storm over; Capabilities requirement change according to the business evolution. For example, a small sales system requires I/O on database; but when the requests number increase, it is necessary to add more load balance layer, cache layer, and more calculation to promote the right products to potential customers, the application turns to CPU intensive type.
Considering a typical case, in a diverse infrastructures data center, administrators install OpenStack as the IaaS, while Cloud Foundry as the PaaS upon OpenStack, then the enterprise applications are running in the PaaS. Suppose one day, the requests of one of the applications increasing rapidly, the metering of application asks for extension to meet the expected requests; then Cloud Foundry automatically scales the application out as adding more instances; but in a time, the Droplet Execute Proxy (DEA), which manages the lifecycle of application instances running in it, get the bottleneck, then it asks the IaaS layer, which is OpenStack, to create more virtual machines and deploy more DEA components into them; the virtual machines are running in compute nodes, eventually the compute nodes will also meet the limitation, and request to scale out, then ask the bare-metal layer to provision more servers and configure them as OpenStack compute nodes.
The reclaim process will be reverse: the unified monitoring service can find the spare resources, and kick start the reclaim process. The idle status will be found from the bare-metal layer and the resources allocated can be merged. Then, the OpenStack is requested to migrate the workload and eliminate some compute nodes. The OpenStack notifies the Cloud Foundry to remove DEAs in these affected compute nodes, and then the Cloud Foundry asks applications perform the migration or kill the idle instances.
All these need a global and unified orchestration of cross-layer resources. Such orchestration has the following features. Firstly, it contains the full status of all layers in data center. Specifically, it includes the data of both physical and virtual machines, and exposing status as a query service and accessible by an external system. Secondly, it acts as a single control plane for all layers in data center, specifically, central control for resource reclamation and allocation; each layer can request resources, but the decision made by a central controller (namely, orchestrator) according to overall consideration based on full status of all layers in data center.
An embodiment of the present invention proposes a solution to handle the cross-layer resources in a declarative way. The foundation of the solution is the unified control and unified data.
After start, the method 200 proceeds to step 201. In step 201, unified control is performed for all resources in all layers of the data center. According to an embodiment of the present invention, performing unified control of all resources in all layers of the data center may comprise: forming a single control plane for all layers of the data center. Therefore, in this way, unified control may be performed for all resources in all layers of the data center in the single control plane. According to an embodiment of the present invention, performing unified control of all resources in all layers of the data center may comprise: performing unified control of allocation and reclaim of all resources of all layers so as to implement unified control of all resources in all layers of the data center.
Then, the method 200 proceeds to step 202. In step 202, unified storage is performed for all topologies and machine-generated data of all layers of the data center. According to an embodiment of the present embodiment, unified storage may be performed for all topologies and machine-generated data of all layers of the data center so as to determine full status of all layers of the data center. Therefore, in this way, it may be realized that resources of all layers of the data center are unifiedly orchestrated according to the full status of all layers of the data center. According to an embodiment of the present invention, the full status may be exposed as a query service and accessible by an external system. Hence, in this manner, full status of all layers of the data center may be conveniently queried and accessed to facilitate unified orchestration of resources in all layers of the data center according to the full status of all layers of the data center. According to an embodiment of the present invention, all machine-generated data in all layers of the data center at least comprises: control plane data as static data or meta-data of the whole data center, and data plane data gathered from the running system.
Then, the method 200 proceeds to step 203. In step 203, the resources of the data center are orchestrated based on unified control and unified storage. According to an embodiment of the present invention, orchestrating the resources of the data center based on unified control and unified storage may comprise: in response to requests of one or more layers for resources, performing unified allocation or reclaim for the resources of the data center based on overall consideration of the data center. According to an embodiment of the present invention, orchestrating the resources of the data center based on unified control and unified storage may comprise: finding insufficiency or idleness of resources, and performing unified allocation and reclaim for the resources of the data center based on overall consideration of the data center. Those skilled in the art may understand that allocation and reclaim of the resources is only a specific example of resource orchestration, orchestration of the resources may further comprise other operations on resources such as resource addition and resource deletion or the like. The present invention does not limit specific operations of orchestrating resources. According to an embodiment of the present invention, all resources of all layers at least comprise: hardware resources in the bare-metal layer; virtual machine resources in Infrastructure as a Service layer; resources in the Platform as a Service layer for developing, running and managing applications; or application resources in the application layer.
In a specific implementation of the method 200, according to an embodiment of the present invention, the network implements, through remote procedure call (RPC) via a message bus, unified control of all resources in all layers of the data center and unified storage of all topologies and machine-generated data of all layers of the data center.
Noticeably, in the present embodiment, as shown in
The control plane data refers to the static data or meta-data of the whole data center. The control plane data comes from the deployment phase. Because every PaaS and IaaS clusters are deployed and managed by the central orchestrator, all the settings and changes can be logged.
Data plane data refers to the dynamic data, which gather from the running system. Applications, PaaS and IaaS will generate a lot of system logs. These logs will be piped up and cleaned, transformed, and loaded to a central store. The data plane data are the important factor to evaluate the QoS.
In addition, according to an embodiment of the present invention, all resources of all layers of the data center may be evaluated and grouped into different SLA groups. In this aspect, according to an embodiment of the present invention, there are two ways to perform the evaluation: manually definition in deployment phase; policies definition according to the data plane data. The resources of each layer will be grouped into different types and QoS, and exposed out as services by different mature tools with a modular design.
The above processes are grouped into phases, and each of them can be composed. For example, the process in “from bare-metal to Operating System” can continue the deployment by appending the processes in “from Operating System to IaaS”, or even cross layer integration may be performed.
The composability mechanism leads to the final data center environment diversity. However, the diversities are logged in the Central Data store, and managed by the Central Control. Embodiments of the present invention provide a unified user interface for the diverse data center architecture and abstract the differences. This is called “Modular Design” in the present invention.
According to an embodiment of the present invention, the solution of the data center is modularized and divided into the following three layers.
A core logic layer: comprising core logic, common facilities and user interfaces. This layer highly abstracts the detail operations, co-ordinates the different pieces of third-party orchestration tools, and provides unified interfaces from bare-metal layer, IaaS layer, PaaS layer to application level. The core logic layer provides a unified orchestration interface for various data center architectures, and specifically comprises provision, configuration, deployment and management, and involves modules such as a policy manager, a Razor client, a MCO broker, a process controller, a profile generator, and a cloud operator, which are exposed out as various plugins.
Third-party orchestration tools layer: as presented above, most existing orchestration tools are low-level orchestration and competent to a specified use case. Embodiments of the present invention are designed to manage from bare-metal layer to application layer and compatible with different solutions, which have to leverage more than one third-party orchestration tools. Embodiments of the present invention catalog the third-party orchestration tools, and apply a plugin-design to each catalog. The plugin-design exposes the unified interfaces out. According to details of the data center, this layer may employ mature and most suitable orchestration tools, for example, existing orchestration tools such as the Cobble, the Razor, the Mirantis Fuel, the Pivotal Ops Manager, the Heat+Puppet, the Serengeti REST API, and the Nise BOSH as shown in
A layer of different processes: as presented above, the process is task oriented and composable. Different processes may be selected and combined as a plugin, such as the VMware vSphere, the Ubuntu, the CENTOS, the vcloud, the OpenStack, the Cloud Foundry, the Hadoop etc. as shown in
According to an embodiment of the present invention, the control unit 701 may be configured to perform unified control of all resources in all layers of the data center; the storage unit 702 may be configured to perform unified storage of all topologies and machine-generated data of all layers of the data center; and the orchestration unit 703 may be configured to orchestrate the resources of the data center based on unified control and unified storage.
According to an embodiment of the present invention, the control unit 701 is further configured to form a single control plane for all layers of the data center. According to an embodiment of the present invention, the control unit 701 is further configured to perform unified control of allocation and reclaim of all resources of all layers. According to an embodiment of the present invention, the storage unit 702 performs the unified storage for all topologies and machine-generated data of all layers of the data center, so that the control unit 701 determines full status of all layers of the data center.
According to an embodiment of the present invention, the full status is exposed as a query service and accessible by an external system. According to an embodiment of the present invention, the orchestration unit 703 is further configured to, in response to requests of one or more layers for resources perform unified allocation of the resources of the data center based on overall consideration of the data center. According to an embodiment of the present invention, the orchestration unit 703 is further configured to find insufficiency or idleness of resources, and perform unified allocation or reclaim of the resources of the data center based on overall consideration of the data center. According to an embodiment of the present invention, the control unit 701 and storage unit 702 implement the unified control of all resources in all layers of the data center and the unified storage of all topologies and machine-generated data of all layers of the data center, through a remote procedure call (RPC) via a message bus.
According to an embodiment of the present invention, the multi-layer architecture at least comprises: a bare-metal layer, an Infrastructure as a Service layer, a Platform as s Service layer or an application layer. According to an embodiment of the present invention, all resources of all layers at least comprise: hardware resources in the bare-metal layer; virtual machine resource in the Infrastructure as a Service layer; resources in the Platform as a Service layer for developing, running and managing applications; or application resources in the application layer. According to an embodiment of the present invention, all machine-generated data in all layers of the data center at least comprises: control plane data as static data or meta-data of the whole data center, and data plane data gathered from the running system.
According to an embodiment of the present invention, each layer of the multiple layers of the data center exposes its capabilities out through an interface. According to an embodiment of the present invention, the control unit 701 is further configured to evaluate all resources of all layers of the data center and group them into different service level arrangement (SLA) groups. According to an embodiment of the present invention, evaluating all resources of all layers of the data center and grouping them into different service level arrangement (SLA) groups by the control unit 701 further comprises: manually defining in the deployment phase, or defining policies according to the data plane data.
According to an embodiment of the present invention, the apparatus 700 is based on task-oriented processes. According to an embodiment of the present invention, the task-oriented processes at least comprise: an installation process from a bare-metal layer to an operating system, an installation process from the operation system to an Infrastructure as a Service layer, a process of installing and configuring storages, or an installation process from the Infrastructure as a Service layer to an Platform as a Service layer. According to an embodiment of the present invention, the task-oriented processes can be combined or integrated in a cross-layer manner.
According to an embodiment of the present invention, the storage unit 702 is further configured to store diversity information of the data center architecture, and the control unit 701 is further configured to manage diversity of the data center architecture. According to an embodiment of the present invention, a unified user interface is provided for the diverse data center architecture.
According to an embodiment of the present invention, the apparatus 700 is based on a solution design of multiple layers. According to an embodiment of the present invention, the multiple layers at least comprise: a core logic layer, which is designed to highly abstract detail operations, co-ordinate the different third-party orchestration tools, and provide a unified interface from a bare-metal, an Infrastructure as a Service layer, a Platform as a Service layer to an application level; a third-party orchestration tools layer, which is designed to use various third-party orchestration tools, catalog the third-party orchestration tools, and apply a plugin-design to each catalog; and a layer of different processes, which is designed to select and combine the task-oriented processes as a plugin.
As shown in
Bus 818 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include, but not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.
Computer system/server 812 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer system/server 812, including both volatile and non-volatile media, removable and non-removable media.
System memory 828 can include computer system readable media in the form of volatile memory, such as a random access memory (RAM) 830 and/or a cache 832. Computer system/server 812 may further include other removable/non-removable, volatile/non-volatile computer system storage media. Although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a “floppy disk”), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media can be provided. In such instances, each drive can be connected to bus 818 by one or more data media interfaces. Memory 828 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
Program/utility 840, having a set (at least one) of program modules 842, may be stored in e.g. memory 828. Such program modules 842 may include, but not limited to, an operating system, one or more application programs, other program modules, and program data. Each of these examples or some combination thereof may include an implementation of a networking environment. Program modules 842 generally carry out the functions and/or methods of embodiments of the invention as described herein.
According to the requirements, computer system/server 812 may also communicate with one or more external devices such as a display device 824, a storage device 814, etc.; with one or more devices that enable a user to interact with computer system/server 812; and/or with any devices (e.g., network card, modem, etc.) that enable computer system/server 812 to communicate with one or more other computing devices. Such communication can occur via input/output (I/O) interfaces 822. Still yet, computer system/server 812 can also communicate with one or more networks such as a local area network (LAN), a wide area network (WAN), and/or a public network (e.g., the Internet) via a network adapter 820. As depicted, network adapter 820 communicates with the other components of computer system/server 812 via bus 818. It should be understood that although not shown, other hardware and/or software modules could be used in conjunction with computer system/server 812, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.
In the description of the embodiments of the present invention, the term “comprise” and like wording should be understood to be open-ended, i.e., to mean “including but not limited to.” The term “based on” should be understood as “at least partially based on.” The term “an embodiment” or “the embodiment” should be understood as “at least one embodiment.”
It should be appreciated that embodiments of the present invention may be implemented by hardware, software or a combination of the software and combination. The hardware part may be implemented using a dedicated logic; the software part may be stored in the memory, and executed by an appropriate instruction executing system, e.g., a microprocessor or a dedicatedly designed hardware. Those skilled in the art may understand that the above apparatus and method may be implemented using a computer-executable instruction and/or included in processor control codes. For example, such codes are provided on a programmable memory or a data carrier such as optical or electronic signal carrier.
In addition, although operations of the present methods are described in a particular order in the drawings, it does not require or imply that these operations must be performed according to this particular order, or a desired outcome can only be achieved by performing all shown operations. On the contrary, the execution order for the steps as depicted in the flowcharts may be varied. Additionally or alternatively, some steps may be omitted, a plurality of steps may be merged into one step, and/or a step may be divided into a plurality of steps for execution. It should also be appreciated that features and functions of two or more devices according to the present invention may be embodied in one device. On the contrary, features and functions of one device as described above may be further divided into and embodied by a plurality of devices.
Although the present invention has been described with reference to a plurality of embodiments, it should be understood that the present invention is not limited to the disclosed embodiments. The present invention intends to cover various modifications and equivalent arrangements included in the spirit and scope of the appended claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2015 1 0004936 | Jan 2015 | CN | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 7650366 | Revsin | Jan 2010 | B1 |
| 7739325 | Okita | Jun 2010 | B1 |
| 8489676 | Chaplin | Jul 2013 | B1 |
| 8862933 | Manglik | Oct 2014 | B2 |
| 8936198 | Eddey | Jan 2015 | B1 |
| 9021297 | Hayes | Apr 2015 | B1 |
| 9307019 | Manglik | Apr 2016 | B2 |
| 9319274 | Wei | Apr 2016 | B1 |
| 9449018 | Ma | Sep 2016 | B1 |
| 9621428 | Lev | Apr 2017 | B1 |
| 9652212 | Jayanti Venkata | May 2017 | B2 |
| 9825822 | Holland | Nov 2017 | B1 |
| 10069928 | Hardy | Sep 2018 | B1 |
| 20050093881 | Okita | May 2005 | A1 |
| 20050283348 | Tsui | Dec 2005 | A1 |
| 20070255854 | Khosravy | Nov 2007 | A1 |
| 20080059487 | Mi | Mar 2008 | A1 |
| 20110078352 | Williamson | Mar 2011 | A1 |
| 20110276962 | Chambers | Nov 2011 | A1 |
| 20120059823 | Barber | Mar 2012 | A1 |
| 20120226808 | Morgan | Sep 2012 | A1 |
| 20120265871 | Dowell | Oct 2012 | A1 |
| 20130080480 | Mao | Mar 2013 | A1 |
| 20130179991 | White | Jul 2013 | A1 |
| 20130205028 | Crockett | Aug 2013 | A1 |
| 20130227641 | White | Aug 2013 | A1 |
| 20130339949 | Spiers | Dec 2013 | A1 |
| 20140075031 | Doering | Mar 2014 | A1 |
| 20140075032 | Vasudevan | Mar 2014 | A1 |
| 20140130036 | Gurikar | May 2014 | A1 |
| 20150106827 | Fiedel | Apr 2015 | A1 |
| 20150237128 | Castro | Aug 2015 | A1 |
| 20160149766 | Borowiec | May 2016 | A1 |
| 20170366882 | Lingampalli | Dec 2017 | A1 |
| Number | Date | Country |
|---|---|---|
| 102546735 | Jul 2012 | CN |
| 102546735 | Jul 2012 | CN |
| 102739771 | Oct 2012 | CN |
| 102739771 | Oct 2012 | CN |
| 102750606 | Oct 2012 | CN |
| 103001945 | Mar 2013 | CN |
| 103905543 | Jul 2014 | CN |
| 103955373 | Jul 2014 | CN |
| Entry |
|---|
| Unknown, “IaaS, PaaS, SaaS (Explained and Compared)” Oct. 28, 2015, Apprenda.com. |
| Unknown—IaaS, PaaS, SaaS (Explained and Compared)—Apprenda (Year: 2014). |
| Number | Date | Country | |
|---|---|---|---|
| 20160197850 A1 | Jul 2016 | US |
