There exist data centers that comprise a plurality of servers, each server hosting one or more virtual machines (VMs). The VMs of a data center may be managed at a central location, such as with the MICROSOFT System Center Virtual Machine Manager (SCVMM) management application. A common scenario is for a multi-tier application to be hosted in a data center, in which the logical functions of an application service are divided amongst two or more discrete processes that communicate with each other, and which may be executing on separate VMs.
An example of a multi-tier application is one that separates the aspects of presentation, logic, and data into separate tiers. In such an example, the presentation tier of the application is the point of user interaction—it displays a user interface and accepts user input. The logic tier of the application coordinates the application, processes commands, makes logical decisions and evaluations, and performs calculations. The data tier of the application stores data for the application, such as in a database or file system.
There are many problems with successfully and consistently updating or patching multi-tier applications and/or the guest OSes in which they execute in within such a data center environment. Some of these problems are well known.
It would be an advantage over prior implementations to an invention for updating or patching a guest OS in a data center.
A problem with prior techniques for patching guest OSes stems from the act of patching guest OSes itself. A typical scenario for patching a guest OS involves executing computer-executable instructions within the guest OS of the VM. Patching a guest OS this way may be highly dependent on the current state of the VM and guest OS, and very error prone. For instance, VMs and guest OSes may “drift”—change their state over time so as to be different from their initial state. This may occur, for instance, where a user logged into the guest OS moves a file that is required to effectuate the patch. When the instructions effectuating the patch determine that that file is not found, the patching process may fail, or behave differently on some machines than on others.
Another problem with “on-line” patching is that files needed that need to be modified may be locked or otherwise un-modifiable, which prevents successful patching. In sum, it is difficult and risky to perform on-line patching, because the state of the machine may vary.
A data center management program allows administrators to model multi-tier applications to allow for automated deployment and servicing of those applications. Once a service template is defined, the Administrator may deploy a new instance of the service from the service template. After the service has been deployed, the data center management program maintains a link to the service template from which it was deployed.
When a service template is later updated, such as to include a new version of an application, the Administrator can decide which services to move to the new version of the service template. When a service is moved to a new version of a service template, VMM determines the changes that have been made and the list of actions that must be applied to each tier in the service to make the service instance match the service template. Prior VMM implementations never maintained this linkage, which resulted in a “fire and forget” scenario, where changes between a service template and service instances could never be detected, let alone remedied.
In the case of application and OS updates, VMM includes the ability to apply the updates using an image-based servicing technique in which new versions of the OS or application are deployed instead of using the common technique of executing code (such as a .msi or .msu file) within the OS. This greatly improves overall reliability since copying files is significantly more reliable than executing code.
During this process, the VHD that contains the guest OS image originally used to deploy the VM may be booted in on a different machine (such as a lab environment) and any patches may be applied to it there. This VHD with the newly-patched OS may then be given back to VMM so that a service template may be created that refers to this VHD. This increases the reliability of the patching process, because then an administrator may confirm that the patch(es) were applied successfully on the image.
VMM then captures any pre-existing application state from the VM that is being updated. For certain types of applications, such as some applications that run on an application virtualization platform (like MICROSOFT APPLICATION VIRTUALIZATION or APP-V), the application state is captured as a part of application execution. For applications where state is not captured as a part of execution, VMM provides an extensible mechanism that allows Administrators to identify where application state is being stored that will need to be recovered (such as particular registry keys or file system locations). To persist this state, VMM attaches a new data disk to the VM to which the application state is then persisted.
Once the application state has been persisted, the original VHD that the VM was booting from is deleted and the updated VHD is deployed to the same location. Optionally, the original VHD may be kept, such as in a scenario where an applied patch may be rolled back, and the guest OS from the original VHD is used again. The VM is then booted and the new VHD is customized and applications are redeployed based on the updated service template model. Some information regarding customizing the VHD and redeploying applications may be found within a service template; other information may be generated based on a pattern or technique set forth by the template (for example, the service template may specify the machine name should have the form “WEB-##” where # represents an integer; VMM may then generate machine names such as WEB-01 and WEB-02 as it recreates machines that have this pattern in their service template). This invention for persisting state has the added benefit of returning the machine to a known good state by effectively undoing any changes that have been made to the machine that are not captured in the application model (e.g. a setting change that was made via a remote desktop connection to the machine).
Once the virtual machine is running, the application state can then be reapplied. Again, for state separated-applications, such as applications that run on an application virtualization platform, this process is done by VMM as a part of servicing the application. For other types of applications, VMM provides an extensible mechanism that allows administrators to apply any state that was previously captured, as needed. After application state has been re-applied, the data disk may be detached from the VM so that the VM is in a state described by a service template.
It can be appreciated by one of skill in the art that one or more various aspects of the invention may include but are not limited to circuitry and/or programming for effecting the herein-referenced aspects of the present invention; the circuitry and/or programming can be virtually any combination of hardware, software, and/or firmware configured to effect the herein-referenced aspects depending upon the design choices of the system designer.
The foregoing is a summary and thus contains, by necessity, simplifications, generalizations and omissions of detail. Those skilled in the art will appreciate that the summary is illustrative only and is not intended to be in any way limiting.
The systems, methods, and computer-readable media for image-based servicing of a virtual machine are further described with reference to the accompanying drawings in which:
Embodiments may execute on one or more computer systems.
The term processor used throughout the description can include hardware components such as hardware interrupt controllers, network adaptors, graphics processors, hardware based video/audio codecs, and the firmware used to operate such hardware. The term processor can also include microprocessors, application specific integrated circuits, and/or one or more logical processors, e.g., one or more cores of a multi-core general processing unit configured by instructions read from firmware and/or software. Logical processor(s) can be configured by instructions embodying logic operable to perform function(s) that are loaded from memory, e.g., RAM, ROM, firmware, and/or mass storage.
Referring now to
A number of program modules comprising computer-readable instructions may be stored on computer-readable media such as the hard disk, magnetic disk 29, optical disk 31, ROM 24 or RAM 25, including an operating system 35, one or more application programs 36, other program modules 37 and program data 38. Upon execution by the processing unit, the computer-readable instructions cause the actions described in more detail below to be carried out or cause the various program modules to be instantiated. A user may enter commands and information into the computer 20 through input devices such as a keyboard 40 and pointing device 42. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner or the like. These and other input devices are often connected to the processing unit 21 through a serial port interface 46 that is coupled to the system bus, but may be connected by other interfaces, such as a parallel port, game port or universal serial bus (USB). A monitor 47, display or other type of display device can also be connected to the system bus 23 via an interface, such as a video adapter 48. In addition to the display 47, computers typically include other peripheral output devices (not shown), such as speakers and printers. The exemplary system of
The computer 20 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 49. The remote computer 49 may be another computer, a server, a router, a network PC, a peer device or other common network node, and typically can include many or all of the elements described above relative to the computer 20, although only a memory storage device 50 has been illustrated in
When used in a LAN networking environment, the computer 20 can be connected to the LAN 51 through a network interface or adapter 53. When used in a WAN networking environment, the computer 20 can typically include a modem 54 or other means for establishing communications over the wide area network 52, such as the Internet. The modem 54, which may be internal or external, can be connected to the system bus 23 via the serial port interface 46. In a networked environment, program modules depicted relative to the computer 20, or portions thereof, may be stored in the remote memory storage device. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used. Moreover, while it is envisioned that numerous embodiments of the present disclosure are particularly well-suited for computerized systems, nothing in this document is intended to limit the disclosure to such embodiments.
System memory 22 of computer 20 may comprise instructions that, upon execution by computer 20, cause the computer 20 to implement the invention, such as the operational procedures of
Hypervisor microkernel 202 can enforce partitioning by restricting a guest operating system's view of system memory. Guest memory is a partition's view of memory that is controlled by a hypervisor. The guest physical address can be backed by system physical address (SPA), i.e., the memory of the physical computer system, managed by hypervisor. In an embodiment, the GPAs and SPAs can be arranged into memory blocks, i.e., one or more pages of memory. When a guest writes to a block using its page table, the data is actually stored in a block with a different system address according to the system wide page table used by hypervisor.
In the depicted example, parent partition component 204, which can also be also thought of as similar to “domain 0” in some hypervisor implementations, can interact with hypervisor microkernel 202 to provide a virtualization layer. Parent partition 204 in this operational environment can be configured to provide resources to guest operating systems executing in the child partitions 1-N by using virtualization service providers 228 (VSPs) that are sometimes referred to as “back-end drivers.” Broadly, VSPs 228 can be used to multiplex the interfaces to the hardware resources by way of virtualization service clients (VSCs) (sometimes referred to as “front-end drivers”) and communicate with the virtualization service clients via communication protocols. As shown by the figures, virtualization service clients can execute within the context of guest operating systems. These drivers are different than the rest of the drivers in the guest in that they may be supplied with a hypervisor, not with a guest.
Emulators 234 (e.g., virtualized integrated drive electronics device (IDE devices), virtualized video adaptors, virtualized NICs, etc.) can be configured to run within the parent partition 204 and are attached to resources available to guest operating systems 220 and 222. For example, when a guest OS touches a register of a virtual device or memory mapped to the virtual device 202, microkernel hypervisor can intercept the request and pass the values the guest attempted to write to an associated emulator.
Each child partition can include one or more virtual processors (230 and 232) that guest operating systems (220 and 222) can manage and schedule threads to execute thereon. Generally, the virtual processors are executable instructions and associated state information that provide a representation of a physical processor with a specific architecture. For example, one virtual machine may have a virtual processor having characteristics of an INTEL x86 processor, whereas another virtual processor may have the characteristics of a PowerPC processor. The virtual processors in this example can be mapped to logical processors of the computer system such that the instructions that effectuate the virtual processors will be backed by logical processors. Thus, in an embodiment including multiple logical processors, virtual processors can be simultaneously executed by logical processors while, for example, other logical processors execute hypervisor instructions. The combination of virtual processors and memory in a partition can be considered a virtual machine.
Guest operating systems can include any operating system such as, for example, a MICROSOFT WINDOWS operating system. The guest operating systems can include user/kernel modes of operation and can have kernels that can include schedulers, memory managers, etc. Generally speaking, kernel mode can include an execution mode in a logical processor that grants access to at least privileged processor instructions. Each guest operating system can have associated file systems that can have applications stored thereon such as terminal servers, e-commerce servers, email servers, etc., and the guest operating systems themselves. The guest operating systems can schedule threads to execute on the virtual processors and instances of such applications can be effectuated.
The operational procedures of
Operation 304 depicts selecting a machine to patch based on an upgrade domain. The domain of machines to be patched and/or upgraded may be each machine of the tier identified in operation 02.
Operation 306 depicts removing the machine to patch from a load balancer. It may be appreciated that, in scenarios where there is no load balancer, the invention may be implemented without operation 306 (or operation 332, which depicts adding the machine back to the load balancer). A load balancer receives requests to use resources of the data center and determines a machine in the data center that will service that request. For instance, clients may contact the data center to access the web tier of a multi-tier application. That contact is received by the load balancer, which determines an appropriate machine to serve the web tier to the client of those machines configured to serve the web tier. This determination may be made, for instance, based on the machine with the highest available load, or in a round-robin fashion.
To determine a machine to process a request, a load balancer may maintain a list of available machines in the data center. By removing the machine to patch from the load balancer's options, the machine may be taken offline and patched without the load balancer attempting to direct requests to the machine while it is unavailable to service those requests.
Operation 314 depicts recreating the VM. A VM have an OS disk attached to it, and then may mount the disk—such as a VHD—and boot a guest OS that is stored on the disk. As depicted in
The VM may also be recreated with the same OS as before, and this may or may not involve swapping the OS disk. The act of recreating a VM may comprise both shutting down or otherwise terminating the VM, then creating or restarting it anew.
Operation 316 depicts customizing the new OS. The OS may be installed from a gold image, which comprises a genericized version of the OS—one without any machine-specific information, such as a machine name, or a security identifier (SID). That machine specific information may be unique to the INTERNET as a whole, or among an intranet or workgroup. Customizing the new OS, then, may comprise adding this machine-specific information to a generic OS. While operation 316 refers to customizing a “new” OS, it may be appreciated that there are scenarios where the VM is merely recreated with the same OS (and same VHD) as it had before. Such a scenario may occur where there is no patch to apply to the OS, but it is being recreated to avoid any possible problems due to skew.
Operation 318 depicts application profile-level pre-install. Beyond customizing the new OS, operations may be implemented that prepare all applications of the OS to be installed. These application profile-level pre-installation procedures may include configuring firewall rules, OS settings, or other machine-level configuration procedures.
Operation 320 depicts application level pre-install. Just as pre-installation procedures may be implemented across an entire profile or machine (as depicted in operation 318), pre-installation procedures may also be implemented for a single application (the application installed in operation 322). This may comprise similar operations as in operation 318, but in the per-application context, such as opening a specific port in a firewall that a specific application uses.
Operation 322 depicts installing the application. This may comprise copying files for the application to one or more places in a file system of the new guest OS. This may also comprise executing an installer for the application, such as a MICROSOFT Windows Installer installer program for versions of the MICROSOFT WINDOWS operating system.
Operation 324 depicts application-level post-install. Operation 324 may be similar to operation 320—application-level pre-install. There may be some operations done before installing the application, because installing the application is dependent on those operations having occurred. Likewise, there may be some operations that are dependent on the application having been installed, such as backing up log files that were created in the process of installing the operation.
Operation 326 depicts application profile-level post-install. Operation 326 may be similar to operation 318. Just like with operations 320 and 324 (depicting pre-install and post-install at the application level), there may be some post-install operations performed at the profile level, and these may occur in operation 326.
Operation 332 depicts adding the machine to the load balancer. This operation may be the analog of operation 406, where the machine was removed from the load balancer. Here, the machine is added to the load balancer, so that the load balancer is configured to be able to assign incoming load to the machine based on a load balancing policy or technique.
Operation 334 depicts that the operational procedures have ended. When the operational procedures reach operation 334, the machine has been serviced.
The operational procedures of
Operation 404 depicts selecting a machine to patch based on an upgrade domain. Operation 404 may be performed in a manner similar to operation 306 of
Operation 406 depicts removing the machine to patch from a load balancer. Operation 406 may be performed in a manner similar to operation 306 of
Operation 408 depicts attaching a data disk to the machine to be patched. The data disk may be used to store application state while the VM is shut down. When the VM is recreated, but for the application state saved on this data disk, it may be that the application state will be lost because it is not found in the new VM image that is used to recreate the VM. The data disk may comprise a virtual hard drive (VHD). A VHD is typically a file that represents a hard disk, including files, folders and file structure stored thereon. The data disk may be attached to the machine to be patched, such that when the machine to be patched is booted up with the new image.
In addition to using a data disk, there are other mechanisms that may be used to store application state. For instance in a could computing platform, such as the MICROSOFT Windows Azure cloud computing platform, a Blob service may be used to store application state. A Blob service provides the ability to create a blob in which application state may be stored, store application state in the blob, and retrieve application state from the blob. These acts performed on a blob may be performed by the VM from which application state is to be stored, a hypervisor that provides virtualized hardware resources to the VM, or the deployment manager that manages the deployment.
Also in addition to using a data disk, a cloud drive may be used—storage within a cloud computing environment. Generally, these techniques for storing application state to a location outside of the VM while the VM is recreated may be referred to as storing application state to a storage location.
Operation 410 depicts storing the state of an application to the data disk. As used herein, applications may be thought to generally fall within two categories: (1) the application model, where applications are directly installed to an OS, and (2) the virtualization model, where applications are deployed on a virtual application platform, like MICROSOFT's Server App-V. Storing data from applications that adhere to the application model is handled in operation 410, while storing data from applications that adhere to the virtualization model is handled below, in operation 412. Operation 410 itself may be effectuated such as by executing scripts within the guest OS that copies files from a file system of the guest OS in which state is stored to the data disk.
It may be appreciated that, in some scenarios, all the application state to be saved is state for applications that adhere to only the application model, or in the alternative, applications that adhere to only the virtualization model. In such scenarios, it may be appreciated that the present invention may be effectuated without implementing all of the operations depicted in
Typically, an application that adheres to the application model is installed to an operating system. As the application executes, as the application is installed, the application (or an installer for the application) may save state to places within the operating system. For instance, the application may store preference or configuration files somewhere within a file structure of the operating system, or in a configuration database, such as the WINDOWS Registry in versions of the MICROSOFT WINDOWS operating system. This application state may be monitored in a variety of ways. A process may execute on the operating system that is able to monitor the application's operations that invoke the operating system and determine which of those operations are likely to change the application's state. Operations that are likely to change state may include modifications to the Registry, or modification (including creation and deletion) of files in portions of a file system likely to indicate that that modification is one of state (such as the creation of a file in C:\Program Files in versions of the MICROSOFT WINDOWS operating system). The process may maintain a list of these modified files. When operation 410 is invoked, the process may provide that list of modified files, and those modified files may be copied to the data disk.
Another way that application state may be monitored is similar. As above, a process may execute on the operating system that is able to monitor the applications' operations that invoke the operating system. Rather than merely tracking those operations that may change application state, the process may re-direct those operations to virtualized portions of the file system or Registry, and maintain them in a separate location. For instance, when the application attempts to write to the operating system registry, the process may intercept this, and save the write to its own Registry. If the application later tries to read that which it has written to the Registry, the process may intercept this, fetch that Registry entry from its own Registry, and provide that fetched entry to the application. In such a scenario, that the data is not stored in the conventional place in the operating system is transparent. Then, when operation 410 is invoked, the process has all of the data that affects the application's state already collected, and may provide this collected information so that it is saved to the data disk.
When application state is saved from a location within a file system of the guest OS, that location within the file system may also be saved with the state, and later that location may be used when restoring the state to restore the state to the proper file system location.
Operation 412 depicts storing the state of a virtualized application. In some virtualized application scenarios, the state of virtualized applications is stored during execution in a centralized location, such as through SERVER APP-V virtualization, this may comprise storing the state stored in that centralized location to the data disk.
Operation 414 depicts swapping the OS disk. Operation 414 may be performed in a similar manner as operation 314 of
Operation 416 depicts customizing the new OS. Operation 416 may be performed in a similar manner as operation 316 of
Operation 418 depicts application profile level pre-install. Operation 426 may be performed in a similar manner as operation 326 of
Operation 420 depicts application level pre-install. Operation 420 may be performed in a manner similar to operation 320 of
Operation 422 depicts installing the application. Operation 422 may be performed in a manner similar to operation 320 of
Operation 424 depicts application level post-install. Operation 424 may be performed in a manner similar to operation 320 of
Operation 426 depicts application profile level post-install. Operation 426 may be performed in a similar manner as operation 326 of
Operation 428 depicts restoring the state of the virtualized application. Where the state of the virtualized application was saved in operation 412 to the data disk, along with the corresponding file system location of the guest OS where the state was stored from, operation 428 may comprise copying the virtualized application state that is stored on the data disk to that file system location.
Operation 430 depicts applying the state of the saved application. Where the state of the application was saved in operation 410 to the data disk, along with the corresponding file system location of the guest OS where the state was stored from, operation 430 may comprise copying the virtualized application state that is stored on the data disk to that file system location.
Operation 432 depicts adding the machine to the load balancer. Operation 432 may be performed in a manner similar to operation 332 of
Operation 434 depicts that the operational procedures have ended. Operation 434 may be performed in a manner similar to operation 334 of
It may be appreciated that the order of these operations is not mandatory, and that embodiments exist where permutations of these operations are implemented. For instance, where a machine comprises only virtualized applications that have state to be saved (and not traditionally installed applications that have state to be saved), operations 410 and 430 (depicting storing the state and restoring the state, respectively, of a traditionally installed application) may be omitted. In another example where the same OS disk is used to recreate the VM, and all applications are stored in the OS disk, the invention may be implemented without implementing operations 414, 418, 420, 422, 424, or 426. Likewise, permutations exist. For instance, an embodiment of the present invention may perform operation 412 before operation 410, and/or operation 430 before operation 428.
Deployment manager 502 may comprise a service or machine that manages deployment 500—it monitors the status and health of hosts 504 within deployment 500, and may also cause the creation and termination of VMs 508 on a host 504, as well as the migration of a VM 508 from one host 504 to another host 504. Deployment manager 502 may comprise, for example, MICROSOFT System Center Virtual Machine Manager (SCVMM). Load balancer 514 maintains a list of VMs 508 of deployment 500, receives connection requests (like a request for a remote presentation session) from clients of deployment 500, and assigns an incoming connection to a VM 508. Load balancer 514 typically assigns an incoming connection to a VM 508 in a manner that balances the load among VMs 508 of deployment 500. Hypervisor 506 of host 504 manages VMs 508 on the host 504, including presenting VMs with virtual hardware resources. Each VM 508 is depicted as having a corresponding OS disk 518 that it boots a guest OS from (for instance, VM-1508-1 is depicted as having corresponding OS disk 1518-1). As depicted, VM-1508-1 boots guest OS 510 from OS disk 1518-1. Two applications execute within guest OS 510—application 1512-1 and application 2512-2. An application 512 may be a traditionally installed application, or a virtualized application (such as a MICROSOFT App-V virtualized application). As depicted, data disk 516 is also mounted by VM-1508-1.
Data disk 516 and OS disks 518 need not be stored on host 504. They may be stored elsewhere and then mounted by host 504 across a communications network. For instance, OS disks 518 may be stored in a central repository for deployment 500, and then attached to a particular host 504 from that central repository.
As depicted, processes 1, 4, and 6, and communication flows 2, 3, 5, and 7 depict an order in which processes and communications may occur to effectuate the image based servicing of a VM. It may be appreciated that this series of processes and communication flows is exemplary, and other embodiments of the present invention may implement permutations and/or different combinations compared to those presented in
In process (1), deployment manager 502 processes a servicing order to patch a service. Deployment manager 502 selects a tier of the service to patch based on the servicing order, and selects a machine based on an upgrade domain. Process (1) may be effectuated in a similar manner as operations 402 and 404 of
In communication flow (2), deployment manager 502 sends load balancer 514 an instruction to remove the machine selected in process (1) from its list of available machines that it may assign load to. Communication flow (2) may occur in a similar manner as operation 406 of
In communication flow (3), deployment manager 502 adds a data disk 516 to the VM 508 selected in process (1) (herein depicted as VM-1508-1). This communication flow (3) may occur in a manner similar to operation 408 of
In process (4), VM-1508-1 stores the state of traditionally installed applications and virtualized applications (herein depicted as application 1512-1 and application 2512-2) to data disk 516. This process (4) may occur in a similar manner as operations 410 and 412 as
Communication flow (5) depicts swapping in OS disk 1518-1 for VM-1508-1. Not depicted is an OS disk that has been swapped out. Communication flow (5) may occur in a similar manner as operation 414 of
Process (6) depicts customizing a guest OS that was swapped in in communication flow (5); performing an application profile level pre-install for each guest OS of VM-1508-1 (herein depicted as guest OS 510, though in embodiments, more guest OSes may be present); and for each application of each guest OS (herein depicted as application 1512-1 and application 2512-2), performing pre-installation functions for an application; installing the application; and performing post-installation functions for the application; performing an application profile-level post install; restoring the state of any virtualized applications; and restoring the state of any traditionally installed applications. These elements of process (6) may be performed in a similar manner as operations 418, 420, 422, 424, 426, 428, and 430 of
Communication flow (7) depicts adding the patched VM 508-1 back to load balancer 514. This communication flow (7) may be performed in a similar manner as operation 432 of
While the present disclosure has been described in connection with the preferred aspects, as illustrated in the various figures, it is understood that other similar aspects may be used or modifications and additions may be made to the described aspects for performing the same function of the present disclosure without deviating there from. Therefore, the present disclosure should not be limited to any single aspect, but rather construed in breadth and scope in accordance with the appended claims. For example, the various procedures described herein may be implemented with hardware or software, or a combination of both. Thus, the methods and apparatus of the disclosed embodiments, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in tangible media, such as floppy diskettes, CD-ROMs, hard drives, or any other machine-readable storage medium. When the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus configured for practicing the disclosed embodiments. In addition to the specific implementations explicitly set forth herein, other aspects and implementations will be apparent to those skilled in the art from consideration of the specification disclosed herein. It is intended that the specification and illustrated implementations be considered as examples only.