Patent Grant
9858068
The present application is a national stage filing under 35 U.S.C 371 of PCT application number PCT/US2010/039432, having an international filing date of Jun. 22, 2010, the disclosure of which is hereby incorporated by reference in its entirety.
Many modern businesses and organizations rely heavily on information technology (IT) to provide computer-based tools and services to enable them and their customers to operate efficiently. The tools and services are typically provided by a multitude of different software applications which typically run on a variety of computing hardware, such as computer servers, networking equipment, storage devices, and the like. For reasons of efficiency and ease of management, this computing hardware is increasingly being consolidated in specialized data centers.
Software applications may be conveniently arranged to run in a virtualized environment through use of software virtualization applications, such as virtual machines. In this way, a single computer server may effectively concurrently run multiple computer operating systems instances (or virtual images) and concurrently run different applications on each of the virtual images.
When deciding on what computing hardware to provision in a data center often little regard is given to the nature of the software applications that are to be run. Accordingly, poorly planned data centers may be provisioned with much more computing hardware than is actually required to run a set of software applications. Not only is such over provisioning costly, it may also lead to the software applications being deployed on the computing hardware in an inefficient manner. Inefficient deployment may lead to so-called server or virtual sprawl, the consequences of which may include significant increases in power, cooling, and space requirements.
Examples of the invention will now be described, by way of non-limiting example only, with reference to the accompanying drawings, in which:
Various embodiments will be described below with reference to various examples.
Increasingly, enterprise software applications are being deployed in virtualized environments on powerful computing hardware provisioned in specialized facilities such as data centers.
Each of the servers 102a to 102n runs a virtualization application, such as VMWare, Inc. vSphere application, which enables instances, or virtual images, of different operating systems to concurrently execute on each of the servers. For example, server 102a may execute a virtualization application that enables the concurrent execution of instances of a Microsoft, Linux, and Solaris operating system, and further enables the concurrently execution of different applications on each of the different operating systems instances.
As shown in
Currently, system architects wishing to deploy a set of applications have typically identified a set of unused computing hardware, for example in a data center, and have determined how the set of applications would be deployed on the computing hardware based largely on their personal knowledge and experience. However, it is common for system architects to substantially over-specify hardware requirements to avoid the risk of potential server overload.
Furthermore, it is common for system architects to deploy the applications on the hardware in a somewhat cautious manner by only lightly loading each server with applications to avoid the risk of server overload. However, such an approach may lead to so-called virtualization or server sprawl in which a large number of physical servers run at low utilization. This can lead to significant inefficiencies with respect to data center power, cooling systems, memory, storage, and physical space.
Referring now to
In accordance with one or more examples, the virtualization assessment engine 302 determines or generates the characteristics of a set of computing hardware that may be used to execute the set of target applications, and additionally determines an efficient manner of distributing or deploying the set of target applications of the determined hardware.
In one example, the virtualization assessment engine 302 obtains (402) data 306a to 306n relating to a set of target applications that are desired to be run in a data center or other computing facility. The virtualization assessment engine 302 processes (404) the obtained data and generates (406) the characteristics of a set of hardware requirements 304a to 304n that may be used to execute the target applications. The virtualization assessment engine 302 then generates (408) a deployment plan 308 for appropriately distributing those applications on the determined hardware. The deployment plan 308 comprises a plan 310 for distributing operating system virtual images across the determined set computing hardware and a plan 312 for distributing, deploying, or stacking, the target applications between the different operating system virtual instances.
The data 306 relating to the set of applications may, for example, be obtained automatically through use of software agents, software management tools, manually, or in any other appropriate manner. The data 306 may include technical characteristics or requirements, including, for example, the operating system on which the application is designed to run, the amount of memory required, the amount of storage required, etc. The data 306 may additionally comprise business requirements data. The business requirements data may, for example, define business-imposed restrictions or requirements. The business requirements may, for example, be obtained by a data center or system administrator or architect, or in any other appropriate manner.
Complex interactions may exist between different software applications. For example, one software application may, for business or security reasons, be the only application allowed to run on a particular physical computer server to ensure that business imposed security requirements are met. Furthermore, such an application may be required to be installed on a stand-alone virtualization application. By way of further example, an application may be dependent on a specific operating system binary library, may require application component level clustering with physical hardware divergence (for example where physical hardware servers are required to be in separate data center enclosures) for redundancy, and so on.
Making sense of all of the different parameters and factors is a particularly challenging and complex task.
Referring now to
An application model matcher module 502 obtains (602) a set 504a to 504n of application attributes of a set of target applications which it is desired to install and execute in a virtualized manner. In the present example, the application attributes include both technical 506 and non-technical 508 application attributes. In other examples, the application attributes 504a to 504n may include attributes other than both technical and non-technical attributes.
The non-technical attributes may include, for example, functional requirements that may impact the placement or cohabitation of an application in a virtual environment or on physical hardware. For example, the non-technical attributes may define, from a functional or business aspect, whether an application is allowed to cohabit with other applications in the same virtual machine, or even on the same physical hardware. This may be the case, for example, for an application that has security requirements in which the application owner does not wish for the application to share physical or virtual resources with any other application.
In some examples the set of obtained attributes 504a to 504n include attribute prioritizations or weightings. In other examples, weightings may be allocated once the attributes have been obtained, either manually by a user or automatically by applying predetermined weightings. The weightings may, for example, be used to indicate relative importance of different characteristics.
Example application attributes are shown below in Tables 1 and 2.
In some examples the application attributes may be defined using a markup language such as the extensible markup language (XML).
The system 500 comprises an application model store 510 in which is stored a set of predetermined application models.
Each application model may be defined through analysis of different software applications that may be desired to be run in a data center. Such analysis may suitably be performed manually by a system administrator or architect or automatically by an application analysis module or tool (not shown). Each application model aims to provide a unique, or substantially unique, set of technical and/or non-technical attributes that identify key application characteristics. Some application models may have only technical attributes, whereas other application models may have both technical and non-technical attributes.
Example application models are shown below in Tables 3 and 4.
In some examples the applications models may be defined using a mark-up language such as XML. Example application attributes are shown below in Tables 1 and 2.
At 604 the application model matcher 502 attempts to match or to assign each of the target applications for which application attributes 504 are obtained to an application model stored in the application model store 510.
In one example the matching or assignment process may be performed by searching for an application model that exactly matches each of the obtained application attributes 504 for a given application. In other examples, the matching process may, for example, be performed by searching for an application model that best matches, or substantially matches, the obtained application attributes, or matches with a predetermined degree of similarity at least some of the attributes. For example, some or all of the application attributes may have associated weightings defining a relative importance level or acceptable similarity level. The analysis process may use suitable XML parsing techniques in some examples, where appropriate.
Once each of the application attributes for each target application have been matched or assigned to an application model the results are passed to a virtual machine image planning module 512.
The virtual machine image planning module 512 obtains application model compatibility data stored in an application model compatibility data store 514. In some examples the application model compatibility data may be stored together with the application model data.
The application model compatibility data defines which application models are compatible with which other application models in a virtualized environment and, by inference, defines which application models are incompatible with which other application models. An example is shown in Table 5 below.
Compatibility of one application model with another may be determined by analysis of the technical and non-technical application attributes. The analysis may be performed, for example, manually by a system administrator or system architect, through use of application analysis tools, process monitors, or in any suitable manner. Example application attributes are shown below in Tables 1 and 2.
For instance, an application model requiring a Linux operating system may be determined to be incompatible with an application requiring a Windows operating system. Similarly, an application model in which application component clustering is required may be determined to be incompatible with an application model in which application clustering is not required.
The virtual machine image planning module 512 determines (606), using the application model compatibility data, which applications are compatible with each other and thus which applications may cohabit with which other applications in a given virtual machine image or on the same physical hardware. Applications for which their corresponding application models may be determined as being non-compatible are determined as not being able to cohabit with each other. For example, applications requiring the Linux system may be determined as not being able to cohabit in a virtual machine image of the Windows operating system. By way of further example, applications having attributes defining that server sharing is not allowed would not be allowed to cohabit with other applications, even with applications requiring the same operating system.
The virtual image planning module 512 determines the minimum number of virtual machine instances of each operating system necessary to execute the set of target applications, taking into account the application compatibility data, and details which applications should and should not be collocated in the same operating system instance. The virtual image planning module 512 also determines appropriate configuration parameters for each virtual image. However, this initial determination is made without any regard to required hardware resources. As described below, a greater number of virtual machine images may be required depending on the determined hardware characteristics.
The determined details are passed to a virtual machine image deployment and application deployment planning module 516.
One particular advantage of having a set of application models and predetermined compatibility between those models is that it reduces the task of determining whether different applications are compatible with each other to the relatively straightforward operation of pattern matching key characteristics of a target application to characteristics of a set of application models.
In order to determine an efficient deployment plan a vast number of factors relating to the software applications to be executed, the operating systems required, the virtual images, the virtualization applications, and the physical hardware may need to be taken into consideration. The factors may include both technical or physical factors as well as business factors.
Hardware technical factors may for example, include: processing power; multitasking capabilities; memory capacity; storage capacity; and network bandwidth. Software application technical factors may include, for example: memory requirements; networking requirements; storage requirements; security requirements; redundancy requirements; and processing power requirements.
The virtual machine deployment and application deployment planning module 516 obtains (610) virtual machine image attributes 520 which define attributes of available target virtual machine images. The virtual machine image attributes 520 may include details of minimum hardware resources required, number of execution threads possible, and other appropriate characteristics. The virtual machine attributes 520 may be obtained, for example, by a system administrator or system architect, through data supplied by virtual machine developers or suppliers, through software monitoring applications, software agents, or in any other suitable manner.
The virtual machine deployment and application deployment planning module 516 obtains (612) deployment plan preference data stored in a deployment plan preference data store 524. The deployment plan preference data may be set by a system administrator or system architect, or a default predetermined set of deployment plan preferences may be defined. The deployment plan preferences determine preferences that are to be taken into account by the module 516, as described further below.
The preferences may include hardware preferences, such as preferred processing characteristics, preferred hardware manufacturers, cost ranges, server type, and the like. The preferences may also define specific limitations or thresholds for the use of hardware by the virtual images. In one example the preferences may define a maximum amount of CPU utilization for a hardware device. Use of the plan preferences enables a final deployment or stacking plan to be tailored to specific system administrator or system architects requirements.
The deployment plan preferences are taken into account, along with the VM image attributes (520), so that the VM image deployment and application deployment planning module (516) produces a set of hardware characteristics 518, a virtual machine deployment plan (526) and an application deployment or stackability plan (528).
At 614, the module 516 determines an initial virtual machine and application deployment plan by analyzing the characteristics of each of the target applications in turn and assigning each application to a virtual image. The assignment of applications to virtual images take into account the application characteristics, such as the resources required by other applications assigned to the same virtual image, as well as application compatibility data. If the virtual machine deployment and application deployment planning module 516 determines that the virtual machine image resources would be exceeded by adding the application to the virtual image, a different virtual image is sought on which to place the application. If a suitable virtual image is found, the application is assigned to the found virtual image. If no suitable virtual image is found, a new virtual image is allocated, and the application is assigned to the new virtual image. In one example the resources allocated to a virtual image may be based on hardware preferences stored in the preference store 524. In another example, the resources allocated to a virtual image may be based on a default set of hardware characteristics.
At 616 the virtual image deployment and application deployment planning module 516 determines, based on the initial virtual machine deployment plan, a set of minimum hardware attributes 518 suitable for executing the target applications in a virtualized manner. The determination may be made, for instance, by determining the minimum requirements for processing power, storage, memory, network capabilities, etc. as determined from the application attributes.
In a further example the hardware attributes determined by the module 516 may define a set of commercially available hardware, for example from a hardware library or database (not shown), by selecting a set of physical hardware that collectively have hardware attributes that meet the determined minimum hardware requirements. The selection process may, in some examples, include pricing information and other technical and non-technical attributes.
Depending on the set of physical hardware determined at 616, and depending on the characteristics thereof, the module 618 may need to adjust the initial virtual machine and application deployment plan to produce a final virtual machine image and application deployment plan. For instance, if a cost limit is included in the hardware preference data the module 516 may determine that two smaller and cheaper servers are required, rather than one larger, more expensive server. In this case, one or more additional virtual machine image may be required and the application deployment plan would need adjusting to take into account the additional virtual machine image or images.
Using the obtained information the virtual machine deployment and application deployment planning module 516 determines (614) a final virtual image deployment plan 526 and a final application deployment or stacking plan 528. The virtual image deployment plan defines the type and number of virtual images that are determined to be suitable for hosting or executing the set of applications. The virtual image deployment plan 526 additionally defines the way in which the defined virtual images should be installed on the determined physical hardware. The application deployment or stacking plan 528 defines which of the target applications should be installed on which of the virtual images.
In some examples the virtual machine image deployment and application deployment planning module 516 calculates, for different combinations of target hardware and target applications, different virtual machine deployment plans 526 and application stacking plans 528 using appropriate ones of the application and virtual machine attributes.
In one example the module 516 determines multiple different combinations of virtual machines and application deployments and ranks the different combinations based on one or more different parameters. The module 516 may rank the different combinations based on different criteria, including, for example, the smallest number of virtual machine images, the smallest number of physical servers, the highest number of applications per virtual machine image, the lower hardware cost, etc. The module 516 may automatically select one of the deployment plans as being the preferred deployment plan based on some predetermined preferences, such as preferences stored in the preference data store 524.
In a further example a list of different deployment plans may be presented to a system administrator or architect for manual selection of a preferred deployment plan and preferred hardware attributes. The presentation may be made, for example, via a visual display unit (not shown) associated with the virtualization assessment engine 302.
The determined virtual machine and application deployment plans 526 and 528, as well as the determined hardware attributes 518, may be output in the form of appropriate metadata, for example in an XML format. The virtual machine and application deployment plans 526 and 528 may be used by virtualization management applications to automatically configure the target hardware with appropriate virtual machine images, and appropriately distribute the target applications in accordance with the selected virtual machine and application deployment plan. The determined hardware attributes 518 may be used by a procurement application for ordering the determined hardware.
An example illustration of a virtual image and application deployment plan 700 is shown in
In a yet further example the application model matcher module 502 is configured to create a new application model when it determined that the set of application attributes 504 for an application do not suitably match any application models currently stored in the application model store 510. In one example if a set of application attributes only partially match an existing application model, a new application model may be created by determining the closest matching application model and copying part of the determined closest matching application model and modifying it to create new model attributes for the non-matching elements. The newly generated application model may be stored in the application model store 510 for future use. In one example, an alert may be triggered when a new application model is created, for example to allow a system administrator or architect to update or verify the newly created application model and to add, if required, application model compatibility data to the application compatibility data store 514.
Referring now to
In a further example a carrier carrying computer-implementable instructions is provided that when interpreted by a computer, cause the computer to perform a method in accordance with any of the above-described examples.
It will be appreciated that examples can be realized in the form of hardware, software or a combination of hardware and software. Any such software may be stored in the form of tangible volatile or non-volatile storage such as, for example, a storage device like a ROM, whether erasable or rewritable or not, or in the form of memory such as, for example, RAM, memory chips device or integrated circuits or on an optically or magnetically readable medium such as, for example, a CD, DVD, magnetic disk or magnetic tape.
It will be appreciated that the storage devices and storage media are examples of machine-readable storage that are suitable for storing a program or programs that, when executed, implement embodiments of the present invention. Accordingly, examples may provide a program comprising code for implementing a system or method as described herein. Examples may additionally provide a machine readable storage storing such a program. Still further, examples may be conveyed electronically via any medium such as a communication signal carried over a wired or wireless connection and examples suitably encompass the same.
All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and/or all of the steps of any method or process so disclosed, may be combined in any combination, except combinations where at least some of such features and/or steps are mutually exclusive.
Each feature disclosed in this specification (including any accompanying claims, abstract and drawings), may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent or similar features.
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/US2010/039432 | 6/22/2010 | WO | 00 | 12/20/2012 |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2011/162744 | 12/29/2011 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 7200530 | Brown et al. | Apr 2007 | B2 |
| 7506037 | Ciano et al. | Mar 2009 | B1 |
| 7774463 | Bloching et al. | Aug 2010 | B2 |
| 8024396 | Sedukhin et al. | Sep 2011 | B2 |
| 8151256 | Ramannavar et al. | Apr 2012 | B2 |
| 8175863 | Ostermeyer et al. | May 2012 | B1 |
| 8181186 | Holcomb et al. | May 2012 | B1 |
| 8225308 | Sedukhin et al. | Jul 2012 | B2 |
| 8261266 | Pike et al. | Sep 2012 | B2 |
| 8387048 | Grechishkin et al. | Feb 2013 | B1 |
| 8407417 | Matsuda et al. | Mar 2013 | B2 |
| 8595737 | Ichikawa et al. | Nov 2013 | B2 |
| 8683548 | Curry et al. | Mar 2014 | B1 |
| 8825964 | Sopka et al. | Sep 2014 | B1 |
| 20020091702 | Mullins | Jul 2002 | A1 |
| 20030110236 | Yang et al. | Jun 2003 | A1 |
| 20030120780 | Zhu et al. | Jun 2003 | A1 |
| 20030233431 | Reddy et al. | Dec 2003 | A1 |
| 20040073673 | Santos et al. | Apr 2004 | A1 |
| 20050021530 | Garg et al. | Jan 2005 | A1 |
| 20050091227 | McCollum et al. | Apr 2005 | A1 |
| 20050154788 | Yang et al. | Jul 2005 | A1 |
| 20050204354 | Sundararajan et al. | Sep 2005 | A1 |
| 20050283759 | Peteanu et al. | Dec 2005 | A1 |
| 20060080413 | Oprea et al. | Apr 2006 | A1 |
| 20060106585 | Brown et al. | May 2006 | A1 |
| 20070006218 | Vinberg et al. | Jan 2007 | A1 |
| 20070028239 | Dyck et al. | Feb 2007 | A1 |
| 20080104605 | Steinder et al. | May 2008 | A1 |
| 20080235378 | Fried et al. | Sep 2008 | A1 |
| 20080294777 | Karve et al. | Nov 2008 | A1 |
| 20080320269 | Houlihan et al. | Dec 2008 | A1 |
| 20090012981 | Kogoh | Jan 2009 | A1 |
| 20090070760 | Khatri et al. | Mar 2009 | A1 |
| 20090070771 | Yuyitung et al. | Mar 2009 | A1 |
| 20090106409 | Murata | Apr 2009 | A1 |
| 20090112966 | Pogrebinsky et al. | Apr 2009 | A1 |
| 20090150529 | Tripathi | Jun 2009 | A1 |
| 20090172666 | Yahalom et al. | Jul 2009 | A1 |
| 20090204961 | Dehaan et al. | Aug 2009 | A1 |
| 20090222560 | Gopisetty et al. | Sep 2009 | A1 |
| 20090228589 | Korupolu | Sep 2009 | A1 |
| 20100005173 | Baskaran et al. | Jan 2010 | A1 |
| 20100030893 | Berg et al. | Feb 2010 | A1 |
| 20100031247 | Arnold et al. | Feb 2010 | A1 |
| 20100250744 | Hadad et al. | Sep 2010 | A1 |
| 20100262974 | Uyeda | Oct 2010 | A1 |
| 20100274981 | Ichikawa | Oct 2010 | A1 |
| 20100281482 | Pike et al. | Nov 2010 | A1 |
| 20100306735 | Hoff et al. | Dec 2010 | A1 |
| 20100332657 | Elyashev | Dec 2010 | A1 |
| 20100332661 | Tameshige | Dec 2010 | A1 |
| 20100333089 | Talwar et al. | Dec 2010 | A1 |
| 20110099548 | Shen et al. | Apr 2011 | A1 |
| 20110131569 | Heim | Jun 2011 | A1 |
| 20110145782 | Brukner et al. | Jun 2011 | A1 |
| 20110202640 | Pillutla | Aug 2011 | A1 |
| 20110209146 | Box et al. | Aug 2011 | A1 |
| 20110225277 | Freimuth et al. | Sep 2011 | A1 |
| 20110231839 | Bennett et al. | Sep 2011 | A1 |
| 20110246739 | Matsuda et al. | Oct 2011 | A1 |
| 20110282982 | Jain | Nov 2011 | A1 |
| 20120042311 | Biran et al. | Feb 2012 | A1 |
| 20120192181 | Gilbert et al. | Jul 2012 | A1 |
| 20120266166 | Farkas et al. | Oct 2012 | A1 |
| 20120311603 | Kudo et al. | Dec 2012 | A1 |
| 20130097293 | Gibson et al. | Apr 2013 | A1 |
| 20130097597 | Gibson et al. | Apr 2013 | A1 |
| 20130238804 | Tanino et al. | Sep 2013 | A1 |
| 20130339956 | Murase et al. | Dec 2013 | A1 |
| 20130346973 | Oda et al. | Dec 2013 | A1 |
| 20140223428 | Hackett et al. | Aug 2014 | A1 |
| Number | Date | Country |
|---|---|---|
| 1601510 | Mar 2005 | CN |
| 1836208 | Sep 2006 | CN |
| 101937357 | Jan 2011 | CN |
| 2009116852 | May 2009 | JP |
| 2011008481 | Jan 2011 | JP |
| Entry |
|---|
| International Search Report and Written Opinion for PCT/US2012/039432, Korean Intellectual Property Office, Apr. 1, 2011. |
| “Managing VMware Doesn't End with Managing VMware”, netiQ, Apr. 2008. <http://download.netiq.com/CMS/WHITEPAPER/ManagingVMware.pdf >. |
| Bailey, Michelle, “The Economics of Virtualization” Moving Toward an Application-Based Cost Model, Nov. 2009. |
| Extended European Search Report ˜ Application No. 10853787.9-1954 dated Aug. 11, 2014 ˜ 8 pages. |
| Tickoo, Omesh et al., “Modeling Virtual Machine Performance: Challenges and Approaches”, Intel Labs, Intel Corporation, vol. 37; pp. 55-60, 2010. |
| Extended European Search Report, EP Application No, 10853789.5, Dated Jun. 18, 2014, pp. 1-6, EPO. |
| Extended European Search Report, EP Application No. 11863212.4, Dated Feb. 16, 2016, pp. 1-7, EPO. |
| International Search Report and Written Opinion, International Application No. PCT/US2010/039438, Dated Mar. 24, 2011, pp. 1-7, KIPO. |
| International Search Report and Written Opinion, International Application No. PCT/US2011/031523, Dated Dec. 23, 2011, pp. 1-6. KIPO. |
| Number | Date | Country | |
|---|---|---|---|
| 20130097597 A1 | Apr 2013 | US |
