The present invention relates to cloud computing. More particularly, the present invention relates to a framework of coordinating and automating processes in cloud computing.
Cloud computing is a way for users to store data and operate computational processes on infrastructure connected by a network. Thus, instead of having to purchase physical infrastructure, users send processes and data out to be run and stored on infrastructure owned by other entities. The user only pays for the amount of data storage or processing capability that he or she desires. This allows the user to tap computing resources that would be impossible without owning actual, physical and vast computing resources.
By its very nature, cloud computing gives developers large tool sets for executing programs and tasks along with database storage but it also creates logistical hurdles to implementing those numerous programs, tasks and storage. For instance, developers may be required to write repetitive daemons for cloud managers and object managers to program the logistics each time an object or process is run. This equates to extra work for developers who must devote great resources on cloud logistics, separate and apart from the development of the actual end product.
Systems and method of managing computer resources in a cloud environment, may comprise receiving, via at least one server, cloud management instructions via an API. Also, coordinating, via the at least one server, with at least one cloud services manager via a distributed coordination and messaging service, wherein the coordination includes instructions regarding the cloud services manager managing at least one cloud resource, and communicating with at least one database to store and retrieve the management instructions.
The systems and methods may also include where the cloud services manager includes at least a cloud services framework, and an API. They may also include where the cloud services manager is further configured to establish and control at least one cloud resource.
Some example systems and methods include where the cloud services manager is located in the cloud. Also, where a controller is configured to interact with the cloud services manager.
Other examples may include where the controller is configured to communicate with the cloud services framework via the API and the cloud services framework is configured to validate the controller. Also where the controller is a cloud resource.
Further examples include where the at least one cloud resource is at least one of a Virtual Machine, an Operating System and a physical computational device. And where the controller is further configured to instantiate at least one cloud service based on parameters entered by a user.
Still other examples include where the controller instantiates at least one cloud service via a start daemon via the API. And where the cloud services framework is configured to scale the at least one cloud resource. Also, where the cloud services framework is configured to request redundant cloud resources in order to replace failed resources. And wherein the API is configured to be dynamically extended by adding services to the cloud.
Other embodiments include where the cloud resource is, or is a combination of, a virtual machine, a physical machine, an operating system, storage service, networking service, and an application service.
For a better understanding of the systems and methods described in this application, reference should be made to the description below, in conjunction with the following drawings, in which:
In the following detailed descriptions, numerous specific details are set forth to illustrate the subject matter presented in this document. It will, however, be apparent to one of ordinary skill in the art that the subject matter may be practiced without these exact specific details. Moreover, the descriptions are provided by way of example and should not be used to limit the scope of any later claimed inventions.
Computer networks of many kinds exist today. Closed and proprietary networks, open and public networks, and hybrid networks as well. These networks tie together computer servers and databases in ways that allow computer programs to run efficiently. Some embodiments of such networks are known as “cloud computing.” Over such a network, such as the internet, computer resources such as infrastructure, platforms and applications can all be used by client users. The physical resources of such cloud networks may be scattered around various physical locations, but by being networked together, can become a larger resource. Various systems and methods can be used to manage the resources that can be hosted and run on such a cloud network.
Cloud Services Framework (CSF) technology described herein, is a set of cloud management tools that aims to manage the resources for logistical cloud infrastructures. CSF abstracts the logistical cloud coordination in order to replicate it and template it for multiple uses.
CSF facilitates the object/process to pass through the cloud management system to the end Cloud Services Manager (CSM) that works with the cloud environment and resources found there. This system bypasses individualized processing of each object as each are loaded. Instead, CSF is able to handle the underlying logistics while maintaining ambivalence to the individual processes. CSF implements logistics such as backup, scalability, redundancy, fail-safes, placement of services across multiple clouds and security of the cloud environment, without separate developer coding for each process individually.
CSF accomplishes this by allowing developers to set Cloud Service Controllers (Controllers) that define Cloud Service Specification (CSS). These CSSs define how clients or Computational Resources (CRs) are to be managed, essentially pre-programming the logistics of the cloud management for each particular service.
CRs could be any computational resource including but not limited to Virtual Machines (VMs), Operating System (OS) containers, or actual physical computational machines.
CSSs become the Application Programming Interface (API) template for later usage and automatically become commands at the Command Line Interface (CLI) and User Interface (UI) levels. Thus, the Controller implements a simple interface with only a few APIs.
With CSSs, a developer can program coordination, networking, storage, permissions and other cloud objects with the CRs running on the cloud without having to write the actual code to do so. The result is easier implementation of the coordination and running of CRs, networking, storage, permissions and other cloud objects, and ensures that CRs run as the developer intended. Another result is that a developer can concentrate on the actual Cloud Service he or she wants to provide without having to worry about other logistical details.
Cloud Resource is inclusive but not limited to the following, virtual machine, physical machine or a OS container, any Storage services provided a private or public cloud, any Networking services provided by private or public cloud, any feature that is provided by the private or public cloud, any higher level application services like load balancers, AutoScaling, DNS, Platform As a Service (PaaS) provided by the cloud vendor or 3rd party cloud service developers.
CSF is a set of cloud management tools that can allow for the replication and automation of Cloud Services logistics. This reduces the amount of time developers have to program those logistics individually, each time a service is created.
Logistics include, but are not limited to coordination and running of the CRs, networking cloud resources, storage, permissions, backup, scalability, redundancy, fail-safes, placement of services across multiple clouds and security of the cloud environment.
CSF allows the object/process to pass through the system and handles the underlying logistics of the cloud(s) while maintaining ambivalence to the individual processes. CSF accomplishes this by allowing developers to set Cloud Service Controllers that define CSSs.
These CSSs become the API interface templates for later usage and automatically become commands at the CLI and UI levels.
CSSs define object coordination, networking, storage, permissions and other cloud objects. CSF uses APIs to implement the services/objects defined in the CSSs. The result is easier and faster implementation of the coordination and running of CRs, networking, storage, permissions and other cloud objects, and ensures that CRs run as the developer intended. Another result is that the developer can concentrate on the actual Cloud Service he or she wants to provide without having to worry about other logistical details.
Cloud Services Framework aims to reduce the amount of time developers need to manage their client CRs. Through the APIs, developers create a set of Cloud Service Controllers. Cloud Service Controllers are the “software/process/daemon” that implement CSF web APIs/Interfaces. They may be written in any programming paradigm and can be added dynamically by the end user or developer to extend the cloud and add functionality.
Cloud Service Controllers define the CSSs. These CSSs become the API interface template for later implementation. Thus, the Controller implements a simple interface with only a few APIs to run complex cloud processes and coordination.
In this example embodiment, CSF uses an API 110 to communicate with one or more Site Services Managers 116. This user API 110 could be a Restful API, an http based web language with requests similar to fetching a web page. Alternative APIs include Web APIs and standards like Simple Object Access Protocol (SOAP).
Continuing with
The individual Cloud Services Managers, 130, 132, 134, use Controllers 140, 142, 144, to carry out processes in the cloud 150.
Alternatively, The individual Cloud Services Managers 130, 132, 134, can be implemented within their own CRs in the cloud 150 as well.
Cloud Service Specifications (CSSs)
A Cloud Service is an instantiation of a CSS. Developers can customize the CSSs in any way they desire to manage and control desired CRs on the cloud. Multiple services can even be combined in one model.
Developers define CSSs as a set of Service Parameters. Users enter the Service Parameters to initiate a particular service. The Service Parameters are labeled by the developer so users can recognize what is required for each input field.
Once established, the CSS automatically populates in the API, CLI and the UI as a new feature/extension for that cloud implementation so end users can recognize the new feature.
When an end user wants to launch a service, a user calls that particular service using the API. When an end user (or another Cloud Service) uses a cloud service, it actually instantiates a cloud service for its consumption.
The user is presented with the set of defined parameter labels from the CSS and the user enters the parameter values for the desired service. The Service Parameters are entered and added to the DDS. The Service Parameters are validated according to the CSS.
Service Parameters specify how a Cloud Service should be instantiated. The CSF provides the instruments, or APIs, to coordinate cloud resources, saving information in the DDS allowing for many clouds to be utilized and auto scaling of cloud resources.
Once the Service Parameters are saved in the DDS, the DC Messaging Service is notified. The DC Messaging Service assigns the Service Parameters an identifier and puts that identifier into a registry. Each Cloud Services Manager is also assigned an identifier. The DC Messaging Service, using the respective identifiers, assigns a Service Parameter to a particular Cloud Services Manager. The Cloud Services Manager is assigned an instantiation of the Cloud Service according to the load by the Load Balancer or Site Manager according to the load.
Once the Cloud Services Manager has the Service Parameter values, it matches them to a type/class registry that define the different type/class services. The Cloud Services Manager also performs a set of validations on the fetched Service Parameters.
After validated and matched, the Manager can begin executing the service and launch CRs in the cloud, in accordance with the instantiation of CSS or Cloud Service Parameters specified by the user.
Cloud Services Manager and the Cloud
An example embodiment of how CSF is arranged and works is described with reference to
The Controller 270 communicates with the CSF 260 through APIs 262. In some embodiments, a Controller 270 can run in its own CR 280 which includes Virtual Machines, its own Operating System (OS) container or actual physical computational device.
Once added, the Cloud Services Manager 230 validates the Controller 270.
The Controller 270 instantiates the Cloud Service based on the parameters entered by the user. To begin, a “start” daemon is communicated from the CSF 260 to the cloud. The Controller 270 can choose to start a daemon on the CR 280 via the APIs 262 provided by the CSF 260.
Once received, CRs 280 are loaded in the cloud and the Controller 270 is informed through an “add_client” notification. Handles that corresponds to the specific CR 280 are assigned. Once a handle is assigned to a CR, 280 that handle is used for later communication with the Controller 270, and to invoke the CR API 262 to manage the CRs 280.
With the CR handle provided as part of “add_client” call, the Controller 270 can set up the CR 280 to provide the service or sub-set of a service. There are many such services but examples include a webserver, memory cache, and load-balancer.
Many multiple CRs 280 can be loaded and instantiated in this way, depending on the desires of the user and the arrangement the Controller 270 requests.
Additionally, other cloud features can be similarly utilized. For example, user specified storage and public IF could be managed by the Cloud Service to make the service resilient and publically available. The CSF 260 provides the APIs 262 necessary to manage CRs 280, storage, networking, security resources which could exist in multiple private or public clouds, or both private and public clouds.
Not only multiple processes but any kind of process may be instantiated this way. For example, storage may be attached to a CR 280, persistent public IP may be assigned, config files may be loaded or any other number of APIs 262.
CSF 260 provides APIs 262 for the Controller to save statistics and information about the Cloud Service in a persistent data store so that users can access that information. The information/data can be used to make changes to the services, add services, remove services, start and stop CRs 280.
The user decides when to delete the Cloud Service that the user instantiated. When users request the Cloud Service instantiation to go away, then all the cloud resources created by the framework and the Controller 270 are cleaned up. “Stop” API implemented by the Controller 270 is called to indicate that the service instantiation is stopped.
Auto Scalability
CSF can be programmed to automatically scale cloud usage up or down depending on criteria determined by a developer. New CRs can be launched or existing ones taken down depending on the needs of the user which allows for efficient usage of cloud resources. This is depicted in
For example, a developer can initially call three CRs 380 to be web servers. Based on the load criteria determined by the developer, CSF 360can prepare more or fewer servers for future use. The Controller 370can monitor the cloud service. The CSF 360 provides APIs 362 to help with that monitoring. When a particular threshold is met, specific to the cloud service, the Controller 370 can request to scale up or down via APIs 362 provided by the CSF 360, in this example, adding or removing web servers depending on the load.
In another example, the Controller 370 monitors the cloud service. The Controller 370 requests to scale up based on some criteria by sending a notification to the Service API 362. A new CR 380 is created and launched in the cloud. The Service API 362 informs the Controller 370 of the new CR 380 by communicating an “add_client” notification. Likewise, if the Controller 370 asks the Service API 362 to scale down, the VM is removed, the Service API 362 informs the Controller 370 by passing a “remove_client” notification.
In still another example, where in a typical cloud environment utilizing CSF 360and running a load balancing/proxy, two Apache servers and a MySQL database, another server is needed. The Cloud Service application monitors the cloud. When the load on the web servers trips a threshold value, the Controller sends a Scale Up notification to the Service API. A new VM is created and launched into the cloud. The Service API 362 notifies the Controller 370 by sending an Add-Client notification. The new VM becomes a new Apache server to handle the large load and the Controller sends a config file to the new server and a reconfigure proxy to the cloud. The new server is then integrated into the cloud system.
These parameters are fully customizable by the Controller and implemented by the cloud through the CR APIs.
Cloud Services Framework Defers Processing
CSF can instantiate services/use CRs and not have to understand or process the specific CR code until late in the process. Thus, as depicted in
CSF coordinates the delegation of the service to a particular Cloud Services Manager by assigning an identifier to the service parameter and storing it in the DDS. Thus, the CSF parameters are initially passed all the way to the Controller, while keeping the rest of the underlying infrastructure from understanding the specifics of the service parameter.
The result is a generic front-end handling processes and a delay of handling the internal code of the service until it reaches the end Cloud Services Manager to be handled by the Controller.
Delaying of processing of code allows CSF to genericize and automate the logistics. Logistics include, but are not limited to coordination and running of the CRs, networking cloud resources, storage, permissions, backup, scalability, redundancy, fail-safes, placement of services across multiple clouds and security of the cloud environment.
High Availability
With respect to
CSF 460 can be programmed using HA aspects in order to automatically assign a new CR to take over for a failed machine. It can promote CR 480 already in use to take over for failed CRs 482.
When a CR dies or becomes unavailable, the Controller 470 is notified via the “remove_client” API. Another CR is automatically arranged for and the Controller 470 is notified via the “add client” API.
In one example, HA also allows for developers to choose how redundant and fail-safe they want their processes to be. Multiple redundancies can be requested by the Controller 470. This is depicted in
Referring to
In this example, the cloud environment utilizing CSF 560, is running a Load Balancer/Proxy 580, two Apache servers 582, 584 and a MySQL database 586. If one Apache server 582 dies or becomes unavailable, a “remove_client” notification is sent from the Service API 562 to the Controller 570. A Reconfigure Proxy 592 communication is sent from the client API to the cloud 540. A new CR590 is created to take the place of the failed one and it is launched. The Controller 570 is notified by an Add-Client 592 notification from the Service API 562. Another Reconfigure Proxy is communicated to the CRs.
Still another example demonstrates how existing CRs can be reconfigured to the role of a dead or unavailable CR. Redundant CRs can run simultaneously and IP and Storage is reattached if the first CR goes down and the redundant CR is promoted. These instructions are all programmed into the CSS logistical coordination.
CSF with a Load Balancer
An alternative embodiment of
Multiple Clouds
References in this disclosure to a “cloud,” “clouds,” or a “cloud environment,” should be understood to encompass one cloud but also but any combination of cloud resources. For instance, CSF can utilize not only one cloud to run CRs, but numerous clouds, both public and private.
Private clouds can work in concert among one another, and also link to a public cloud. This arrangement can provide security options to developers and cloud administrators.
Cloud resources can be networked together seamlessly by CSF. This is because logistics of cloud resources including cloud resource management, can all be programmed into the CSSs. This allows users to utilize cloud resources without even being aware of what cloud they are utilizing.
Security
Permissions can be applied on each cloud service individually by the cloud administrator. Users can view, list, add, or delete permissions.
All CRs that are instantiated are completely isolated by firewalls.
Security rules can be dynamically applied to the CRs. For example, Cloud Services Controllers can selectively open ports via APIs for exposing only a specific service.
Each Controller can run inside its own isolated OS container or VM.
All API calls are encrypted using Secure Socket Layer (SSL) certificates.
Third Party Feature Addition
Cloud computing presents a vast set of resources for utilization. Therefore, it makes sense to open the management of the cloud services to as many entities as possible, in order to develop the best management tools for those resources. Therefore, CSF allows for the maximum participation by users in adding to, adapting and enhancing the cloud services available.
CSF does this by allowing users and Third Party developers to add their own cloud features or services. This effectively allows other Third Party users and developers to extend the cloud in any way they see fit.
New Cloud Services can be added via the RESTful API, RESTful API/http, CLI and UI dynamically and they will be available for immediate use.
The foregoing description, for purpose of explanation, has been described with reference to specific examples. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. This includes practicing the examples of the various subject matter described above in any combination. The examples were chosen and described in order to best explain the principles of the invention and its practical applications, to thereby enable others skilled in the art to best utilize the inventions with various modifications as are suited to the particular use contemplated.
As disclosed herein, features consistent with the present inventions may be implemented via computer-hardware, software and/or firmware. For example, the systems and methods disclosed herein may be embodied in various forms including, for example, a data processor, such as a computer that also includes a database, digital electronic circuitry, firmware, software, computer networks, servers, or in combinations of them. Further, while some of the disclosed implementations describe specific hardware components, systems and methods consistent with the innovations herein may be implemented with any combination of hardware, software and/or firmware. Moreover, the above-noted features and other aspects and principles of the innovations herein may be implemented in various environments. Such environments and related applications may be specially constructed for performing the various routines, processes and/or operations according to the invention or they may include a general-purpose computer or computing platform selectively activated or reconfigured by code to provide the necessary functionality. The processes disclosed herein are not inherently related to any particular computer, network, architecture, environment, or other apparatus, and may be implemented by a suitable combination of hardware, software, and/or firmware. For example, various general-purpose machines may be used with programs written in accordance with teachings of the invention, or it may be more convenient to construct a specialized apparatus or system to perform the required methods and techniques.
Aspects of the method and system described herein, such as the logic, may be implemented as functionality programmed into any of a variety of circuitry, including programmable logic devices (“PLDs”), such as field programmable gate arrays (“FPGAs”), programmable array logic (“PAL”) devices, electrically programmable logic and memory devices and standard cell-based devices, as well as application specific integrated circuits. Some other possibilities for implementing aspects include: memory devices, microcontrollers with memory (such as EEPROM), embedded microprocessors, firmware, software, etc. Furthermore, aspects may be embodied in microprocessors having software-based circuit emulation, discrete logic (sequential and combinatorial), custom devices, fuzzy (neural) logic, quantum devices, and hybrids of any of the above device types. The underlying device technologies may be provided in a variety of component types, e.g., metal-oxide semiconductor field-effect transistor (“MOSFET”) technologies like complementary metal-oxide semiconductor (“CMOS”), bipolar technologies like emitter-coupled logic (“ECL”), polymer technologies (e.g., silicon-conjugated polymer and metal-conjugated polymer-metal structures), mixed analog and digital, and so on.
It should also be noted that the various logic and/or functions disclosed herein may be enabled using any number of combinations of hardware, firmware, and/or as data and/or instructions embodied in various machine-readable or computer-readable media, in terms of their behavioral, register transfer, logic component, and/or other characteristics. Computer-readable media in which such formatted data and/or instructions may be embodied include, but are not limited to, non-volatile storage media in various forms (e.g., optical, magnetic or semiconductor storage media) and carrier waves that may be used to transfer such formatted data and/or instructions through wireless, optical, or wired signaling media or any combination thereof. Examples of transfers of such formatted data and/or instructions by carrier waves include, but are not limited to, transfers (uploads, downloads, e-mail, etc.) over the Internet and/or other computer networks via one or more data transfer protocols (e.g., HTTP, FTP, SMTP, and so on).
Unless the context clearly requires otherwise, throughout the description and the claims, the words “comprise,” “comprising,” and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is to say, in a sense of “including, but not limited to.” Words using the singular or plural number also include the plural or singular number respectively. Additionally, the words “herein,” “hereunder,” “above,” “below,” and words of similar import refer to this application as a whole and not to any particular portions of this application. When the word “or” is used in reference to a list of two or more items, that word covers all of the following interpretations of the word: any of the items in the list, all of the items in the list and any combination of the items in the list.
Although certain presently preferred implementations of the invention have been specifically described herein, it will be apparent to those skilled in the art to which the invention pertains that variations and modifications of the various implementations shown and described herein may be made without departing from the spirit and scope of the invention. Accordingly, it is intended that the invention be limited only to the extent required by the applicable rules of law.
The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications, to thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated.
This application is a U.S. National Stage application that claims priority under 35 U.S.C. 371 to PCT Application No. PCT/US2013/025186, filed on Feb. 7, 2013, titled “CLOUD COMPUTING SERVICES FRAMEWORK”, which claims the benefit and priority under 35 U.S.C. 119(e) from U.S. Provisional Application No. 61/597,610, filed Feb. 10, 2012, titled “CLOUD COMPUTING SERVICES FRAMEWORK”.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2013/025186 | 2/7/2013 | WO | 00 |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2013/119841 | 8/15/2013 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 4086628 | Woodrum | Apr 1978 | A |
| 5239648 | Nukui | Aug 1993 | A |
| 5463774 | Jenness | Oct 1995 | A |
| 5832505 | Kasso et al. | Nov 1998 | A |
| 6038399 | Fisher et al. | Mar 2000 | A |
| 6047129 | Frye | Apr 2000 | A |
| 6473800 | Jerger et al. | Oct 2002 | B1 |
| 6772350 | Belani et al. | Aug 2004 | B1 |
| 6944777 | Belani et al. | Sep 2005 | B1 |
| 7200865 | Roscoe et al. | Apr 2007 | B1 |
| 7225210 | Guthrie, II | May 2007 | B2 |
| 7366793 | Kenner et al. | Apr 2008 | B2 |
| 7475419 | Basu et al. | Jan 2009 | B1 |
| 7602756 | Gu et al. | Oct 2009 | B2 |
| 7631306 | Puig et al. | Dec 2009 | B1 |
| 7734643 | Waterhouse et al. | Jun 2010 | B1 |
| 7886038 | Ferris | Feb 2011 | B2 |
| 7890626 | Gadir | Feb 2011 | B1 |
| 7921452 | Ridion et al. | Apr 2011 | B2 |
| 7953823 | Rider et al. | May 2011 | B2 |
| 7958246 | Barber | Jun 2011 | B2 |
| 8037108 | Chang | Oct 2011 | B1 |
| 8086177 | Trift et al. | Dec 2011 | B2 |
| 8108377 | Jiang et al. | Jan 2012 | B2 |
| 8112505 | Ben-Shaul et al. | Feb 2012 | B1 |
| 8201175 | Zhu | Jun 2012 | B2 |
| 8209415 | Wei | Jun 2012 | B2 |
| 8291079 | Colton | Oct 2012 | B1 |
| 8341141 | Krislov | Dec 2012 | B2 |
| 8347263 | Offer | Jan 2013 | B1 |
| 8429162 | Wang et al. | Apr 2013 | B1 |
| 8438654 | von Eicken et al. | May 2013 | B1 |
| 8464250 | Ansel | Jun 2013 | B1 |
| 8468455 | Jorgensen et al. | Jun 2013 | B2 |
| 8472438 | Kini et al. | Jun 2013 | B2 |
| 8495611 | McCarthy et al. | Jul 2013 | B2 |
| 8504689 | Ferris et al. | Aug 2013 | B2 |
| 8509231 | Hoole et al. | Aug 2013 | B2 |
| 8577937 | Offer | Nov 2013 | B1 |
| 8584215 | Narasimha et al. | Nov 2013 | B2 |
| 8650152 | Dettinger et al. | Feb 2014 | B2 |
| 8850528 | Van Biljon et al. | Sep 2014 | B2 |
| 8938540 | Van Biljon et al. | Jan 2015 | B2 |
| 8977679 | Van Biljon et al. | Mar 2015 | B2 |
| 9021009 | Van Biljon et al. | Apr 2015 | B2 |
| 9032069 | Van Biljon et al. | May 2015 | B2 |
| 9076168 | Van Biljon et al. | Jul 2015 | B2 |
| 9087352 | Van Biljon et al. | Jul 2015 | B2 |
| 9171323 | Van Biljon et al. | Oct 2015 | B2 |
| 9202239 | Van Biljon et al. | Dec 2015 | B2 |
| 9218616 | Van Biljon et al. | Dec 2015 | B2 |
| 9232000 | Pittman | Jan 2016 | B1 |
| 20020052941 | Patterson | May 2002 | A1 |
| 20020097747 | Kirkby et al. | Jul 2002 | A1 |
| 20030018927 | Gadir et al. | Jan 2003 | A1 |
| 20030037284 | Srinivasan et al. | Feb 2003 | A1 |
| 20030105810 | McCrory et al. | Jun 2003 | A1 |
| 20030229623 | Chang et al. | Dec 2003 | A1 |
| 20040024892 | Creswell et al. | Feb 2004 | A1 |
| 20040184070 | Kiraly et al. | Sep 2004 | A1 |
| 20040250120 | Ng | Dec 2004 | A1 |
| 20050038834 | Souder et al. | Feb 2005 | A1 |
| 20050055306 | Miller et al. | Mar 2005 | A1 |
| 20050065855 | Geller | Mar 2005 | A1 |
| 20050187937 | Kawabe et al. | Aug 2005 | A1 |
| 20050193218 | Susser et al. | Sep 2005 | A1 |
| 20050278441 | Bond et al. | Dec 2005 | A1 |
| 20060112176 | Liu et al. | May 2006 | A1 |
| 20060209868 | Callaghan | Sep 2006 | A1 |
| 20060212545 | Nichols et al. | Sep 2006 | A1 |
| 20060259947 | Aarnos et al. | Nov 2006 | A1 |
| 20070072591 | McGary et al. | Mar 2007 | A1 |
| 20070162456 | Agassi et al. | Jul 2007 | A1 |
| 20070234332 | Bundridge et al. | Oct 2007 | A1 |
| 20070255798 | Schneider | Nov 2007 | A1 |
| 20080052203 | Beyer et al. | Feb 2008 | A1 |
| 20080195760 | Nudler | Aug 2008 | A1 |
| 20080228734 | Kang | Sep 2008 | A1 |
| 20080295094 | Korupolu et al. | Nov 2008 | A1 |
| 20080316980 | Ahlen et al. | Dec 2008 | A1 |
| 20090024522 | Reunert et al. | Jan 2009 | A1 |
| 20090172662 | Liu | Jul 2009 | A1 |
| 20090182622 | Agarwal et al. | Jul 2009 | A1 |
| 20090210429 | Agrawal et al. | Aug 2009 | A1 |
| 20090228950 | Reed | Sep 2009 | A1 |
| 20090235342 | Manion et al. | Sep 2009 | A1 |
| 20090240728 | Shukla et al. | Sep 2009 | A1 |
| 20090276771 | Nickolov et al. | Nov 2009 | A1 |
| 20090300210 | Ferris | Dec 2009 | A1 |
| 20090300350 | Gai et al. | Dec 2009 | A1 |
| 20090319529 | Bartlett et al. | Dec 2009 | A1 |
| 20090327471 | Astete et al. | Dec 2009 | A1 |
| 20100036736 | McGee et al. | Feb 2010 | A1 |
| 20100042720 | Stienhans et al. | Feb 2010 | A1 |
| 20100061391 | Sindhu et al. | Mar 2010 | A1 |
| 20100070501 | Walsh et al. | Mar 2010 | A1 |
| 20100071035 | Budko et al. | Mar 2010 | A1 |
| 20100083004 | Kirshenbaum | Apr 2010 | A1 |
| 20100114714 | Vitek | May 2010 | A1 |
| 20100161717 | Albrecht | Jun 2010 | A1 |
| 20100169477 | Stienhans et al. | Jul 2010 | A1 |
| 20100180014 | Kannan et al. | Jul 2010 | A1 |
| 20100185455 | Miller | Jul 2010 | A1 |
| 20100194963 | Terashima | Aug 2010 | A1 |
| 20100197267 | Raleigh | Aug 2010 | A1 |
| 20100198972 | Umbehocker | Aug 2010 | A1 |
| 20100217840 | Dehaan et al. | Aug 2010 | A1 |
| 20100223385 | Gulley et al. | Sep 2010 | A1 |
| 20100250748 | Sivasubramanian et al. | Sep 2010 | A1 |
| 20100250956 | Reed et al. | Sep 2010 | A1 |
| 20100251242 | Sivasubramanian et al. | Sep 2010 | A1 |
| 20100269109 | Cartales | Oct 2010 | A1 |
| 20100274769 | Hazlewood et al. | Oct 2010 | A1 |
| 20100275059 | Hazlewood et al. | Oct 2010 | A1 |
| 20100318645 | Hoole et al. | Dec 2010 | A1 |
| 20100332629 | Cotugno et al. | Dec 2010 | A1 |
| 20110016214 | Jackson | Jan 2011 | A1 |
| 20110022652 | Lai et al. | Jan 2011 | A1 |
| 20110022812 | van der Linden et al. | Jan 2011 | A1 |
| 20110055034 | Ferris et al. | Mar 2011 | A1 |
| 20110055378 | Ferris et al. | Mar 2011 | A1 |
| 20110055399 | Tung et al. | Mar 2011 | A1 |
| 20110055712 | Tung et al. | Mar 2011 | A1 |
| 20110078679 | Bozek et al. | Mar 2011 | A1 |
| 20110096174 | King et al. | Apr 2011 | A1 |
| 20110099146 | McAlister et al. | Apr 2011 | A1 |
| 20110106875 | Koenig | May 2011 | A1 |
| 20110119381 | Glover et al. | May 2011 | A1 |
| 20110126047 | Anderson et al. | May 2011 | A1 |
| 20110126168 | Ilyayev | May 2011 | A1 |
| 20110126197 | Larsen et al. | May 2011 | A1 |
| 20110185063 | Head et al. | Jul 2011 | A1 |
| 20110191610 | Agarwal et al. | Aug 2011 | A1 |
| 20110209064 | Jorgensen et al. | Aug 2011 | A1 |
| 20110213687 | Ferris et al. | Sep 2011 | A1 |
| 20110214124 | Ferris et al. | Sep 2011 | A1 |
| 20110225299 | Nathuji et al. | Sep 2011 | A1 |
| 20110225467 | Betzler et al. | Sep 2011 | A1 |
| 20110231525 | Balani et al. | Sep 2011 | A1 |
| 20110231899 | Pulier et al. | Sep 2011 | A1 |
| 20110243553 | Russell | Oct 2011 | A1 |
| 20110246253 | Yu et al. | Oct 2011 | A1 |
| 20110246984 | Sharp et al. | Oct 2011 | A1 |
| 20110258692 | Morrison et al. | Oct 2011 | A1 |
| 20110265081 | Lucovsky et al. | Oct 2011 | A1 |
| 20110265147 | Liu | Oct 2011 | A1 |
| 20110265164 | Lucovsky | Oct 2011 | A1 |
| 20110270721 | Kusterer | Nov 2011 | A1 |
| 20110270968 | Salsburg et al. | Nov 2011 | A1 |
| 20110282832 | Rishel et al. | Nov 2011 | A1 |
| 20110282940 | Zhang | Nov 2011 | A1 |
| 20110295999 | Ferris et al. | Dec 2011 | A1 |
| 20110296019 | Ferris et al. | Dec 2011 | A1 |
| 20110307391 | Mercuri et al. | Dec 2011 | A1 |
| 20110307899 | Lee et al. | Dec 2011 | A1 |
| 20120017112 | Broda et al. | Jan 2012 | A1 |
| 20120030069 | Garg et al. | Feb 2012 | A1 |
| 20120030672 | Zygmuntowicz et al. | Feb 2012 | A1 |
| 20120047239 | Donahue | Feb 2012 | A1 |
| 20120072985 | Davne et al. | Mar 2012 | A1 |
| 20120078849 | Bryant et al. | Mar 2012 | A1 |
| 20120079276 | Evans et al. | Mar 2012 | A1 |
| 20120096158 | Astete et al. | Apr 2012 | A1 |
| 20120110055 | Van Biljon et al. | May 2012 | A1 |
| 20120110056 | Van Biljon et al. | May 2012 | A1 |
| 20120110180 | Van Biljon et al. | May 2012 | A1 |
| 20120110188 | Van Biljon et al. | May 2012 | A1 |
| 20120110636 | Van Biljon et al. | May 2012 | A1 |
| 20120110650 | Van Biljon et al. | May 2012 | A1 |
| 20120110651 | Van Biljon et al. | May 2012 | A1 |
| 20120116937 | Van Biljon et al. | May 2012 | A1 |
| 20120117229 | Van Biljon et al. | May 2012 | A1 |
| 20120124211 | Kampas et al. | May 2012 | A1 |
| 20120150791 | Willson | Jun 2012 | A1 |
| 20120185913 | Martinez | Jul 2012 | A1 |
| 20120221603 | Kothule et al. | Aug 2012 | A1 |
| 20120246646 | Bakman | Sep 2012 | A1 |
| 20120259815 | Olson | Oct 2012 | A1 |
| 20120310880 | Giampaolo et al. | Dec 2012 | A1 |
| 20120310991 | Frantz et al. | Dec 2012 | A1 |
| 20130019015 | Devarakonda | Jan 2013 | A1 |
| 20130030853 | Agarwal et al. | Jan 2013 | A1 |
| 20130036213 | Hasan et al. | Feb 2013 | A1 |
| 20130060839 | Van Biljon et al. | Mar 2013 | A1 |
| 20130125217 | Edwards et al. | May 2013 | A1 |
| 20130173798 | Micucci et al. | Jul 2013 | A1 |
| 20150006482 | Hardy et al. | Jan 2015 | A1 |
| 20150120936 | Palan et al. | Apr 2015 | A1 |
| 20150264121 | Van Biljon et al. | Sep 2015 | A1 |
| Number | Date | Country |
|---|---|---|
| 101969475 | Feb 2011 | CN |
| 102223398 | Oct 2011 | CN |
| 103488697 | Jan 2014 | CN |
| 104303175 | Jan 2015 | CN |
| 104335179 | Feb 2015 | CN |
| 105339940 | Feb 2016 | CN |
| 104335179 | Jan 2018 | CN |
| 104303175 | Jun 2018 | CN |
| 2583211 | Apr 2013 | EP |
| 2812809 | Dec 2014 | EP |
| 2815346 | Dec 2014 | EP |
| 3014485 | May 2016 | EP |
| 2015507301 | Mar 2015 | JP |
| 2015512091 | Apr 2015 | JP |
| 2016524255 | Aug 2016 | JP |
| 6231020 | Nov 2017 | JP |
| 2009151729 | Dec 2009 | WO |
| WO2011159842 | Dec 2011 | WO |
| 2011159842 | Mar 2012 | WO |
| 2012167108 | Dec 2012 | WO |
| 2013119841 | Aug 2013 | WO |
| 2013122815 | Aug 2013 | WO |
| 2014209848 | Dec 2014 | WO |
| Entry |
|---|
| U.S. Appl. No. 13/299,262, Non-Final Office Action dated Nov. 17, 2014, 14 pages. |
| U.S. Appl. No. 13/299,157 Non-Final Office Action dated Dec. 3, 2014, 10 pages. |
| U.S. Appl. No. 13/299,206 Notice of Allowance dated Dec. 19, 2014, 10 pages. |
| U.S. Appl. No. 13/299,319 Notice of Allowance dated Jan. 13, 2015, 9 pages. |
| U.S. Appl. No. 13/299,301, Final Office Action dated Jan. 29, 2015, 13 pages. |
| U.S. Appl. No. 13/299,262, Notice of Allowance dated Mar. 3, 2015, 10 pages. |
| U.S. Appl. No. 13/299,157, Notice of Allowance dated Feb. 27, 2015, 9 pages. |
| U.S. Appl. No. 13/299,066, Non-Final Office Action dated Mar. 19, 2015, 17 pages. |
| U.S. Appl. No. 13/299,339, Notice of Allowance dated Apr. 21, 2015, 17 pages. |
| U.S. Appl. No. 14/459,104 3rd Party Submission mailed on May 29, 2015, 14 pages. |
| International Preliminary Report on Patentability for application PCT/US2013/025211 (dated Aug. 28, 2014). |
| U.S. Appl. No. 13/299,066, Final Office Action dated Jul. 18, 2014, 19 pages. |
| U.S. Appl. No. 13/299,335, Notice of Allowance dated Jul. 18, 2014, 7 pages. |
| U.S. Appl. No. 13/299,339, Non-Final Office Action dated Sep. 3, 2014, 8 pages. |
| U.S. Appl. No. 13/299,287, Notice of Allowance dated Sep. 16, 2014, 16 pages. |
| U.S. Appl. No. 13/299,335, Notice of Allowance dated Sep. 15, 2014, 10 pages. |
| International Application No. PCT/US2011/040590, International Preliminary Report on Patentability dated Jan. 3, 2013, 11 pages. |
| International Application No. PCT/US2013/025186, International Preliminary Report on Patentability dated Aug. 21, 2014, 6 pages. |
| U.S. Appl. No. 13/299,287, Corrected Notice of Allowability dated Oct. 15, 2014, 2 pages. |
| U.S. Appl. No. 13/299,335, Corrected Notice of Allowability dated Oct. 6, 2014, 4 pages. |
| U.S. Appl. No. 13/299,339, Notice of Allowance dated Jun. 27, 2014, 17 pages. |
| “A White Paper from the Open Cloud Standards”, Interoperable Clouds, Version: 1.0.0, Nov. 11, 2009, pp. 1-21. |
| “Managing VMware VirtualCenter Roles and Permissions”, Retrieved from the Internet: URL:http://www.vmware.com/pdf/vi3_vc_roles.pdf, Apr. 4, 2007, 15 pages. |
| “VMware Virtual Networking Concepts”, Retrieved from the internet: https://www.vmware.com/files/pdf/virtual_networking_concepts.pdf, Jul. 18, 2007, 12 pages. |
| Danwei et al., “Access Control of Cloud Service Based on UCON”, Cloud Computing, Springer Berlin Heidelberg, Dec. 1, 2009, pp. 559-564. |
| EP11796398.3, “Extended European Search Report”, dated May 25, 2016, 14 pages. |
| EP11796398.3, “Partial Supplementary European Search Report”, dated Apr. 7, 2016, 6 pages. |
| EP13746545.6, “Extended European Search Report”, dated Apr. 26, 2016, 8 pages. |
| Fitzpatrick, “Tonido Keeps Cloud Computing Local”, Retrieved from the Internet: URL:http:f/lifehacker.com/5208833/tonido-keeps-cloud-computing-local, Apr. 13, 2009, 2 pages. |
| Thain et al., “Distributed Computing in Practice: The Condor Experience”, Concurrency and Computation: Practice and Experience, vol. 17, Jan. 1, 2005, 37 pages. |
| U.S. Appl. No. 14/226,557, “Non-Final Office Action”, dated May 19, 2016, 9 pages. |
| U.S. Appl. No. 14/459,104 , “Non-Final Office Action”, dated May 20, 2016, 28 pages. |
| U.S. Appl. No. 14/724,043, “Non-Final Office Action”, dated Jun. 16, 2016, 24 pages. |
| International Search Report for application PCT/US2013/025186 (dated Apr. 19, 2013). |
| International Search Report for application PCT/US2013/025211 (dated Apr. 16, 2013). |
| International Search Report for application PCT/US11/040590 (dated Jan. 5, 2012). |
| Karjoth, Gunter, “Access Control with IBM Trivoli Access”, May 2003, IBM Research, Zurich Research Laboratory, vol. 6, No. 2. |
| Canetti et al. “Practical delegation of computation using multiple servers,” CCS '11 Proceedings of the 18th ACM conference on Computer and communications security, pp. 445-454(Oct. 2011). |
| U.S. Appl. No. 13/299,004, Non-Final Office Action dated Apr. 15, 2013, 58 pages. |
| U.S. Appl. No. 13/299,004, Final Office Action dated Oct. 2, 2013, 42 pages. |
| U.S. Appl. No. 13/299,066 Non-Final Office Action dated Apr. 12, 2013, 26 pages. |
| U.S. Appl. No. 13/299,066, Final Office Action dated Oct. 22, 2013, 25 pages. |
| U.S. Appl. No. 13/299,157, Non-Final Office Action dated May 24, 2013, 14 pages. |
| U.S. Appl. No. 13/299,157 Final Office Action dated Dec. 5, 2013, 9 pages. |
| U.S. Appl. No. 13/299,262, Non-Final Office Action dated May 2, 2013, 12 pages. |
| U.S. Appl. No. 13/299,262, Final Office Action dated Oct. 30, 2013, 11 pages. |
| U.S. Appl. No. 13/299,287, Non-Final Office Action dated Oct. 28, 2013, 18 pages. |
| U.S. Appl. No. 13/299,335, Non-Final Office Action dated Nov. 22, 2013, 24 pages. |
| U.S. Appl. No. 13/299,339, Non-Final Office Action dated Nov. 16, 2012, 12 pages. |
| U.S. Appl. No. 13/299,339, Final Office Action dated Aug. 7, 2013, 16 pages. |
| U.S. Appl. No. 13/299,066, Non-Final Office Action dated Mar. 12, 2014, 16 pages. |
| U.S. Appl. No. 13/299,004, Notice of Allowance dated Mar. 24, 2014, 22 pages. |
| U.S. Appl. No. 13/299,335, Notice of Allowance dated May 1, 2014, 7 pages. |
| U.S. Appl. No. 13/299,206, Non-Final Office Action dated May 22, 2014, 10 pages. |
| U.S. Appl. No. 13/299,287, Final Office Action dated May 23, 2014, 12 pages. |
| U.S. Appl. No. 13/299,301, Non-Final Office Action dated Jun. 12, 2014, 18 pages. |
| U.S. Appl. No. 13/299,319, Non-Final Office Action dated Jun. 6, 2014, 13 pages. |
| European Search Report for Application No. EP 13748986 dated Sep. 8, 2015, 7 pages. |
| International Preliminary Report on Patentability for Application No. PCT/US2014/043599, dated Jan. 7, 2016, 8 pages. |
| International Search Report and Written Opinion for Application No. PCT/US2014/043599, dated Oct. 14, 2014, 11 pages. |
| U.S. Appl. No. 13/299,066, Notice of Allowance dated Jul. 31, 2015, 23 pages. |
| U.S. Appl. No. 13/299,301, Notice of Allowance dated Jun. 12, 2015, 5 pages. |
| U.S. Appl. No. 13/299,339, Advisory Action dated Oct. 18, 2013, 2 pages. |
| Baron Schwartz et al., “High Performance MySQL: Optimization, Backups, Replication, and More”, Jun. 2008, pp. 428-429. |
| Liu et al., “NIST Cloud Computing Reference Architecture”, NIST Special Publication 500-292, Sep. 8, 2011, 35 pages. |
| M-T Schmidt et al., “The Enterprise Service Bus: Making service-oriented acrchitecture real”, IBM Systems Journal, vol. 44, No. 4, Oct. 1, 2005, pp. 781-797. |
| U.S. Appl. No. 14/226,557, Notice of Allowance dated Nov. 25, 2016, 9 pages. |
| U.S. Appl. No. 14/724,043, Non-Final Office Action dated Dec. 30, 2016, 12 pages. |
| U.S. Appl. No. 14/459,104, Final Office Action dated Dec. 2, 2016, 43 pages. |
| Chinese Application No. 201380014187.4, Office Action dated May 27, 2016, 14 pages (6 pages of Original document and 8 pages of English Translation). |
| U.S. Appl. No. 14/724,043, Notice of Allowance dated May 19, 2017, 19 pages. |
| Chinese Application No. 201380014187.4, Office Action dated Feb. 4, 2017, 7 pages (3 pages for the original document and 4 pages for the English translation). |
| Chinese Application No. 201380014746.1, Office Action dated Mar. 27, 2017, 12 pages. |
| Japanese Application No. 2014-556691, Office Action dated Jan. 31, 2017, 8 pages (4 pages for the original document and 4 pages for the English translation). |
| U.S. Appl. No. 14/459,104, “Non-Final Office Action”, dated Oct. 18, 2017, 37 pages. |
| CN201380014187.4, “Office Action”, dated Aug. 17, 2017, 15 pages. |
| JP2014-556683, “Office Action”, dated Oct. 17, 2017, 5 pages. |
| European Application No. 11796398.3, Office Action dated Aug. 8, 2017, 4 pages. |
| European Application No. 13746545.6, Summons to Attend Oral Proceedings mailed on Apr. 25, 2018, 11 pages. |
| Japanese Application No. 2014-556683, Office Action dated Jun. 5, 2018, 6 pages (3 pages of the original document and 3 pages of the English translation). |
| U.S. Appl. No. 15/692,929, Non-Final Office Action dated Jun. 14, 2018, 10 pages. |
| Chinese Application No. 201480036405.9, Office Action dated Jun. 15, 2018, 15 pages (6 pages of the original document and 9 pages of the English translation). |
| European Application No. 14740064.2, Office Action dated Aug. 8, 2018, 7 pages. |
| Hewitt, Cassandra—The Definitive Guide, Distributed Data at web Scale, O'Reilly, ISBN: 978-1-449-39041-9, Nov. 2010, 330 pages. |
| Japanese Application No. 2016-523825, Office Action dated Jul. 31, 2018, 4 pages (3 pages of the original document and 1 pages of the English translation). |
| U.S. Appl. No. 14/459,104, Advisory Action dated Aug. 28, 2018, 7 pages. |
| U.S. Appl. No. 14/459,104, Non-Final Office Action dated Jan. 11, 2019, 27 pages. |
| U.S. Appl. No. 15/692,929, Notice of Allowance dated Jan. 9, 2019, 10 pages. |
| European Application No. 11796398.3, Office Action dated Dec. 3, 2018, 4 pages. |
| European Application No. 13748986.0, Office Action dated Apr. 24, 2018, 8 pages. |
| European Application No. 11796398.3, Office Action dated Mar. 26, 2018, 5 pages. |
| U.S. Appl. No. 14/459,104, Final Office Action dated May 1, 2018, 28 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20160197843 A1 | Jul 2016 | US |
| Number | Date | Country | |
|---|---|---|---|
| 61597610 | Feb 2012 | US |