Recovering a virtual machine after failure of post-copy live migration

Description

FIELD OF THE INVENTION

The present invention relates to the field of live migration of virtual machines.

INCORPORATION BY REFERENCE AND INTERPRETATION OF LANGUAGE

Citation or identification of any reference herein, in any section of this application, shall not be construed as an admission that such reference is necessarily available as prior art to the present application. The disclosures of each reference disclosed herein, whether U.S. or foreign patent literature, or non-patent literature, are hereby incorporated by reference in their entirety in this application, and shall be treated as if the entirety thereof forms a part of this application.

Such cited or identified references are provided for their disclosure of technologies to enable practice of the present invention, to provide basis for claim language, and to make clear applicant's possession of the invention with respect to the various aggregates, combinations, and subcombinations of the respective disclosures or portions thereof (within a particular reference or across multiple references). The citation of references is intended to be part of the disclosure of the invention, and not merely supplementary background information. The incorporation by reference does not extend to teachings which are inconsistent with the invention as expressly described herein (which may be treated as counter examples), and is evidence of a proper interpretation by persons of ordinary skill in the art of the terms, phrase and concepts discussed herein, without being limiting as the sole interpretation available.

The present specification is not to be interpreted by recourse to lay dictionaries in preference to field-specific dictionaries or usage. Where a conflict of interpretation exists, the hierarchy of resolution shall be the express specification, references cited for propositions, incorporated references, the inventors' prior publications, the scrivener's prior publications relating to the field, academic literature in the field, commercial literature in the field, field-specific dictionaries, lay literature in the field, general purpose dictionaries, and common understanding. Where the issue of interpretation of claim amendments arises, the hierarchy is modified to include arguments made during the prosecution and accepted without retained recourse.

BACKGROUND OF THE INVENTION

Live migration of a virtual machine (VM) refers to the transfer of an active VM's execution state from one physical machine to another. Live migration is a key feature and selling point for virtualization technologies. Users and service providers of a virtualized infrastructure have many reasons to perform live VM migration such as routine maintenance, load balancing, scaling to meet performance demands, and consolidation to save energy. For instance, vMotion {38} is a popular features of VMWare's ESX server product. Live migration is also extensively used in Google's production infrastructure to perform over a million migrations {35} per month.

Existing live migration mechanisms aim to move VMs as quickly as possible and with minimal impact on the applications and the cluster infrastructure, and indeed these goals have been extensively studied by both academia and industry. Two dominant live migration mechanisms underlie all migration techniques: pre-copy {6}, {30} and post-copy {16}, {17}. The two techniques differ in whether a VM's CPU execution state is transferred before or after the transfer of its memory pages. In pre-copy, the VM continues executing at the source while its memory contents are transferred to the destination over multiple iterations, at the end of which the CPU execution state is transferred and the VM is resumed at the destination. In contrast, post-copy first transfers the VM's CPU execution state to the destination, where the VM immediately resumes execution while, in the background, the memory pages are actively pushed from source and also retrieved upon page-faults at the destination. Pre-copy works well for applications that mostly read from memory whereas post-copy works well for write-intensive applications that would otherwise prevent pre-copy iterations from converging. Google's data centers {35} use both techniques depending upon the nature of a VM's workload.

An important consideration in live VM migration is the robustness of the migration mechanism itself. Specifically, the source, the destination, or the network itself can fail during live migration. Since a VM encapsulates a cloud customer's critical workload, it is essential that the VM's state is preserved accurately and not lost due to failures during migration. Let's consider a VM's recoverability after a failure during live migration. In both pre-copy and post-copy, the failure of the source node during migration results in a permanent loss of the VM because some or all of the latest state of the VM resides at the source during migration.

However, the two approaches differ in their response to failure of the destination node or the network. For pre-copy, either of these failures is not catastrophic because the source node still holds an up-to-date copy of the VM's execution state from which the VM can be resumed after migration failure. However, for post-copy, a destination or network failure has a more severe implication because the latest state of the VM is split across the source and destination. The destination node has a more up-to-date copy of the VM's execution state and a subset of its memory pages, whereas the source node has pages that have not yet been sent to the destination.

Thus, a failure of the destination or the network during post-copy migration also results in a complete loss of the VM. This failure scenario during post-copy live migration and the resulting loss of VM has not been addressed in existing literature. The problem is important because a VM is particularly vulnerable during migration. VM migration may last anywhere from a few seconds to several minutes, depending on factors such as the VM's memory size and load on the cluster. Thus, the window of vulnerability can be large. Additionally, since the VM is live, it is executing code that might communicate over the network with remote entities, altering the external world's view of the VM's state.

Pre-copy: In pre-copy live migration {6}, {30}, the VM's memory pages are transferred to the destination host over multiple iterations. In the first iteration, the entire memory state of the VM is transferred and the subsequent iterations transfer only the modified memory pages. When the estimated downtime (the duration when the VM is suspended during migration) is less than a threshold, the VM on the source host is paused and the remaining dirty pages, the device state, and the CPU state are transferred to the destination. The VM is then resumed on the destination.

Post-copy: In post-copy live migration {16}, {17}, the VM is first suspended on the source host and the CPU state is transferred to the destination host where the VM is resumed immediately. The source then actively sends the memory pages to the destination. This stage is known as the active push phase, with the expectation that most pages would be received by the destination before they are accessed by the VM. If the VM accesses a page that has not yet received by the destination, then a page fault is triggered and the source sends the faulted page to the destination (called demand paging). During post-copy, after the VM resumes on the destination, the guest OS and all applications inside the VM continue execution on the destination machine.

VM Replication: High availability solutions, such as Remus {7}, maintain a consistent replica of a VM during its normal execution. A checkpoint cycle in Remus consists of four stages: VM execution, VM replication, checkpoint transfer, and buffer release. In the first stage, the VM executes, the outgoing network packets of the VM are buffered, and the incoming network requests are served. The outgoing network packets are buffered to ensure that the state of the backup VM will be consistent with the external world during the restoration, if the primary VM crashes in the middle of a checkpoint. The buffer cannot hold the packets for too long as it increases the response latency of the network packets. When the epoch time, defined by the user, is reached, the VM is checkpointed (the second stage), which creates a replica of the primary VM. In this stage, the VM is paused in order to capture the entire system state accurately. The checkpoint is then transferred to the backup node (the third stage). Once the checkpoint is committed to the backup node, the buffered outputs are released.

Checkpointing based fault-tolerance: Almost all virtualization platforms {2}, {22}, {37} support VM checkpointing and restoration. In checkpointing, the memory state of a VM is captured and preserved locally or remotely. During a failure situation, the VM is rolled back/restored to the previously checkpointed state. However, after the restoration, the VM states between the last checkpoint and the time when the failure occurs are lost. Checkpointing can be done at either the application level {25}, {41} or the whole system level {7}, {10}, {13}, {19}, {20}, {23}, {24}, {27}, {29}, {32}. Several operating systems {1}, {28}, {31}, {33} were developed to support process-level migration. The main challenge in process level migration for fault tolerance is that the migrated process leaves residual dependencies in the source machine and the solutions for high availability process level checkpoint/restoration have to deal with such dependencies. Compared to the application specific checkpointing schemes, whole system checkpoints provide more reliability and higher availability with significantly higher cost.

The traditional way of achieving fault tolerance via check-pointing is an active-passive approach {7}, {27}, {39} where the backup node gets the control once the primary VM failed. However, the active-passive approaches suffer from extra network latency due to outgoing network packet buffering, large memory state transfer, and high checkpointing frequency. To provide seamless failover restoration, active-active replications {10}, {40} are introduced in which the primary and backup VMs execute in parallel. These systems compare the responses to client requests to decide when the checkpoint should be triggered. If the output of both primary and backup VMs diverges, committing of network responses is withheld until the primary VM's memory state is synchronized with the backup VM. Network buffering latency is common for both active-passive and active-active approaches. Instead of buffering external events (i.e., network packets), Kemari {36} initiates the checkpoint of the primary VM when the hypervisor starts duplicating the external events to the backup VM. None of the above approaches applied periodic or event-based replication fault tolerance solutions to handle destination failure during live migration.

Logging and replay based fault-tolerance: Logging mechanisms can replay events at runtime to ensure the identical backup of the primary VM. Bressoud et al. {5} proposed a hypervisor-based fault tolerance approach in which the hypervisor logs each instruction level operation on the primary VM and replayed the logged state on the secondary VM. Although logging can be done in the virtual machine monitor (VMM) {5}, {11}, {21}, {29}, deterministic replay relies on the architecture of the VMM and cannot be easily viable to multicore CPUs. Approaches such as Flight Data Recorder {42} sniff cache traffic to infer how shared memory is accessed. Dunlap {12} used CREW (concurrent read, exclusive write) protocol on shared memory to capture the access order. However, deterministic replay mechanisms work with high overhead.

Live migration as a VM fault-tolerance technique: Techniques have also been developed to quickly evict VMs from the source to the destination machine upon imminent failure of the source machine {8}, {9}, {14}, {15}. However, none of them protect the VM against the failure of the live migration itself.

SUMMARY OF THE INVENTION

A solution for VM migration is provided, called PostCopyFT, to recover a VM after the failure of destination node or network during post-copy live migration. During post-copy, once a VM resumes execution at the destination, the destination concurrently transmits reverse incremental checkpoints of the VM back to the source node. This reverse checkpointing proceeds concurrently, and in coordination with, forward post-copy migration from source to destination. If the destination or the network fails during migration then the source node recovers the VM from the last consistent checkpoint that it received from the destination.

PostcopyFT provides an approach to recover a VM when the source, the destination or the network fail during the post-copy live migration. To handle the source failure during the migration, prior to the migration, the source host periodically transfers the incremental memory checkpoints of the VM to the checkpoint cache store. Once the migration starts, the latest memory state of the VM is captured and transferred to the checkpoint cache store. To handle the destination and network failure during the migration, the destination host transmits the reverse incremental checkpoints of the VM back to the checkpoint cache store as soon as a VM resumes execution at the destination. Reverse checkpointing proceeds concurrently, and in coordination with, the forward post-copy migration from source to destination. If the source, the destination, or the network fails during the migration, then the destination or the source node recovers the VM from the last consistent checkpoint that it received from the checkpoint cache store.

PostCopyFT supports either periodic or event-based reverse incremental checkpointing, with different benefits. The reverse checkpoints are much smaller than full-VM checkpoints because they consist only of the VM's modified memory pages since the last checkpoint, and its CPU and I/O state. For checkpoint consistency, PostCopyFT buffers packet transmissions to external world between successive reverse checkpoints.

PostCopyFT is developed to recover the VM when the source, the destination, or the network fails during the post-copy migration of the VM. FIG. 1B gives the architecture of PostCopyFT. PostCopyFT consists of two main components: a forward checkpointing component (called PostCopyFT-forward) that handles the failure of the source host and a backward checkpointing component (called PostCopyFT-reverse) that handles the failure of the destination host and the network. The timeline of PostCopyFT is shown in FIG. 1B.

A PostCopyFT prototype was implemented and evaluated in the KVM/QEMU virtualization platform. The results show that, when using PostCopyFT, the total migration time of post-copy remains unchanged. There is a small increase in the cumulative downtime experienced by the VM when using periodic reverse incremental checkpoints, while event-based checkpoints further reduce this overhead.

Design

During the post-copy migration, the VM's state is split between the source and the destination nodes. As a result, a failure of the destination node or in the network during the migration may lead to the complete loss of the VM. PostCopyFT is developed to recover the VM in this failure scenario. FIG. 1A shows the architecture of PostCopyFT. FIG. 1B shows a timeline for operation of PostCopyFT.

A. Reverse Incremental Checkpointing

The first step of post-copy migration is to transfer the execution state of the VM, along with a minimal non-pageable memory state, to the destination. The VM is resumes at the destination while concurrently receiving the VM's pages from the source. PostCopyFT superimposes a reverse incremental checkpointing mechanism over this forward transfer of VM state. Specifically, once the migrating VM is resumed at the destination, PostCopyFT captures the the VM's initial execution state and memory at the destination and transfers them to a checkpoint store. This checkpoint store is an in-memory key-value store located at either the source node or a third staging node. Then onwards, PostCopyFT captures any incremental changes in the VM's state, including the execution state and any modified memory pages, either periodically or upon an I/O activity of the VM, and forwards these to the checkpoint store. This checkpointing mechanism stops once post-copy migration successful completes the VM's migration to the destination.

B. Failure Recovery

When the network or the destination node fails due to a hardware or software failure, the source node triggers a restoration process. The source node uses heartbeat messages to monitor the liveness and reachability of the destination node. When successive heartbeat messages are not acknowledged by the destination, the migration is considered to have failed. The source then recovers the VM by restoring the last consistent copy of each memory page from the checkpoint store on the VM's memory address space. Pages not modified by the destination do not need to be overwritten. Finally, the VM is resumed at the source from the latest checkpointed CPU execution state to complete the VM's recovery.

C. Network Buffering

To ensure the consistency of VM checkpoints, PostCopyFT buffers packet transmissions to external world between successive incremental checkpoints. The incoming network packets of the migrating VM are delivered to the VM immediately, but the outgoing network packets are buffered until the current reverse checkpoint is committed. The packets in the network buffer are then transmitted and the VM is resumed. This ensures that no network packets are transmitted before the corresponding checkpoint is committed to the checkpoint store. Thus, if the destination or network fails during the migration, PostCopyFT guarantees that the latest committed checkpoint reflects a consistent state of the VM to the external world.

D. Checkpointing Overhead Reduction

One of the requirements in PostCopyFT design is that the reverse incremental checkpointing mechanism should not significantly increase the total migration time if the migration succeeds. To satisfy this requirement, reverse checkpointing is implemented as a separate thread that runs concurrently with the VM. The only time this thread affects the VM's execution is when suspending the vCPUs briefly to capture their execution state. The active-push phase from the source to destination runs concurrently with the reverse checkpointing mechanism even when the vCPUs are paused. This helps PostCopyFT to achieve similar total migration time as postcopy live migration.

A second design requirement is that PostCopyFT's reverse checkpointing mechanism should not significantly increase the VM downtime. PostCopyFT introduces the following two optimizations for this purpose.

Performing incremental VM checkpointing in two stages: Periodic VM checkpointing may increase the downtime of migrating memory-write intensive VMs whose memory pages are dirtied rapidly. To reduce the downtime, PostCopyFT performs VM checkpointing in two stages. In Stage I, PostCopyFT checkpoints only the modified memory pages of the VM, but not its execution state, which includes the VM's CPU and device states. The modified memory pages are checkpointed without pausing the VM and hence Stage I does not increase VM downtime. In Stage II, the VM is paused briefly to capture the VM's execution state. Once Stage II is completed, the VM resumes its execution. The committed checkpoint contains the memory pages checkpointed in both stages. If a memory page is checkpointed in both stages, then the page checkpointed in Stage I is overwritten by that checkpointed in Stage II to ensure that the checkpoint contains the most up-to-date page. The experimental results show that performing checkpointing in two stages significantly reduces the migration downtime, compared to if the VM was paused during memory capture.

Storing checkpoints locally: In Stage I, the VM's memory states are first copied to a local in-memory storage without waiting for the checkpoints to be transferred or synchronized with the staging node storing the checkpoints. After State II, the VM is resumed and the checkpointed memory and execution states are transferred to the staging node.

In contrast to existing techniques which aim to recover a VM after failure during normal execution, PostCopyFT is the first approach to address the problem of VM recovery after failure during live migration.

Forward Checkpointing

When a VM starts executing on a physical machine, PostCopyFT-forward checkpoints the VM's memory state and transfers the checkpoint to a checkpoint cache store. The checkpoint cache store is an in-memory key-value store located at a staging node (neither the source nor the destination host). PostCopyFT-forward then captures the VM's modified memory pages periodically. Note that each individual snapshot captured in the forward checkpointing stage does not need to be complete or fully consistent; instead, all memory snapshots together represent the progress towards the complete VM memory state. The above process is similar to pre-copy except that the snapshots are transferred during the normal execution of the VM (instead of during the migration), and are sent periodically. When the migration starts, the VM is suspended, the modified memory pages are checkpointed, and the execution state of the VM is transferred to the destination. As these live incremental memory checkpoints are captured without pausing the VM, the VM's down-time is unchanged due to forward checkpoints.

If the source machine fails during the migration, PostCopyFT detects the absence of the source and triggers the VM restoration at destination by retrieving the checkpointed state from the checkpoint cache that stores incremental memory checkpoints.

To reduce the checkpointing overhead, PostCopyFT-forward may be implemented as a separate thread that runs in parallel to VM's normal execution. The VM's memory state is captured without suspending the VM's vCPUs. Once the migration process starts, PostCopyFT-forward checkpoints only the incremental changes of the VM's memory state. As the VM's snapshot size linearly grows with the VM's size. The time taken to restore VM upon failover relies on the checkpoint size. In addition, the checkpoints can be loaded ahead of time to reduce the VM recovery time upon failure.

Reverse Incremental Checkpointing

The first step of post-copy migration is to transfer the execution state of the VM, along with a minimal non-pageable memory state, to the destination. The VM is resumed at the destination while concurrently receiving the VM's pages from the source. PostCopyFT-reverse superimposes a reverse incremental checkpointing mechanism over this forward transfer of VM state. Specifically, once the migrating VM is resumed at the destination, PostCopyFT-reverse captures the VM's initial execution state and memory at the destination and transfers them to a checkpoint store. From then onwards, PostCopyFT-reverse captures any incremental changes in the VM's state, including the execution state and any modified memory pages, either periodically or upon an I/O activity of the VM, and forwards these to the checkpoint store. This checkpointing mechanism stops once post-copy migration successful completes the VM's migration to the destination.

Network Buffering

To ensure the consistency of VM checkpoints, PostCopyFT-reverse buffers packet transmissions to external world between successive incremental checkpoints. The incoming network packets of the migrating VM are delivered to the VM immediately, but the outgoing network packets are buffered until the current reverse checkpoint is committed. The packets in the network buffer are then transmitted and the VM is resumed. This ensures that no network packets are transmitted before the corresponding checkpoint is committed to the checkpoint store. Thus, if the destination or network fails during the migration, PostCopyFT-reverse guarantees that the latest committed checkpoint reflects a consistent state of the VM to the external world.

PostCopyFT-Reverse Checkpointing Overhead Reduction

One of the requirements in PostCopyFT-reverse design is that the reverse incremental checkpointing mechanism should not significantly increase the total migration time if the migration succeeds. To satisfy this requirement, reverse checkpointing is preferably implemented as a separate thread that runs concurrently with the VM. The only time this thread affects the VM's execution is when suspending the vCPUs briefly to capture their execution state. The active-push phase from the source to destination runs concurrently with the reverse checkpointing mechanism even when the vCPUs are paused. This helps PostCopyFT-reverse to achieve similar total migration time as post-copy live migration.

A second design requirement is that PostCopyFT-reverse's reverse checkpointing mechanism should not significantly increase the VM downtime. PostCopyFT-reverse introduces the following two optimizations for this purpose.

Performing incremental VM checkpointing in two stages: Periodic VM checkpointing may increase the downtime of migrating memory-write intensive VMs whose memory pages are dirtied rapidly. Algorithm 1 gives the pseudo code for periodic checkpointing while algorithm 2 gives pseudo code for event based checkpointing. To reduce the downtime, PostCopyFT-reverse performs VM checkpointing in two stages. In Stage I, PostCopyFT-reverse checkpoints only the modified memory pages of the VM, but not its execution state, which includes the VM's CPU and device states. The modified memory pages are checkpointed without pausing the VM and hence Stage I does not increase VM downtime.

In Stage II, the VM is paused briefly to capture the VM's execution state. Once Stage II is completed, the VM resumes its execution. The committed checkpoint contains the memory pages checkpointed in both stages. If a memory page is checkpointed in both stages, then the page checkpointed in Stage I is overwritten by that checkpointed in Stage II to ensure that the checkpoint contains the most up-to-date page. Experimental results show that performing checkpointing in two stages significantly reduces the migration downtime, compared to if the VM was paused during memory capture.

Failure Recovery

When the network or the destination node fails due to a hardware or software failure, the source node triggers a restoration process. Upon failure of the source node during migration, the destination triggers a restoration process. The source and destination nodes use heartbeat messages to monitor the liveness and reachability of each other. When successive heartbeat messages are not acknowledged by the source or the destination, the migration is considered to have failed. Upon failure of the destination, the source then recovers the VM by restoring the last consistent copy of each memory page from the checkpoint store on the VM's memory address space. Pages not modified by the destination do not need to be overwritten. Finally, the VM is resumed at the source from the latest checkpointed CPU execution state to complete the VM's recovery. Restoration process for the source node failure is live. The restoration manager at the destination loads missing pages from the checkpoint cache pages that have not yet received by destination during migration. Recovery of VM during source failure is live because destination already contains latest execution state where VM is continuously running during restoration process.

Algorithm 1 PostCopyFT Periodic Checkpointing Algorithm

1: I> Initial preparation

2: Perform initial ft bitmap declaration, memory allocation;

3: Set all bits in ft bitmap to zero

4: I> Snapshot Phase

5: Start global dirty log

6: do

7: Set Network barrier to Buffer network packets

8: sleep (snapshot interval)

9: sync bitmap (ft bitmap)

10: Capture dirty pages and store in in-memory local data structure I> Memory Snapshot I

11: Pause the VM

12: Capture CPU and I/O state and store in in-memory local data structure

13: sync bitmap (ft bitmap)

14: Capture dirty pages and store in in-memory local data structure I> Memory Snapshot II

15: Resume the VM

16: Transfer in-memory locally stored entire checkpoint to cache store

17: Commit the completed checkpoint iteration to cache store

18: Release Network barrier to deliver buffered network packets

19: while (Incoming migration is active)

20: Stop global dirty log

Algorithm 2 PostCopyFT Event-based Checkpointing Algorithm

1: I> Initial preparation

2: Perform initial ft bitmap declaration, memory allocation

3: Set all bits in ft bitmap to zero

4: I> Snapshot Phase

5: Start global dirty log

6: do

7: if (external event triggered)

8: Set network barrier to Buffer network packets

9: Pause the VM

10: Capture CPU and I/O state and store in in-memory local data structure;

11: sync bitmap (ft bitmap)

12: Capture dirty pages and store in in-memory local data structure

13: Resume the VM

14: Transfer in-memory locally stored entire checkpoint to cache store

15: Commit the completed checkpoint iteration to cache store

16: Release network buffer and Deliver buffered network packets

17: end if

18: while (Incoming migration is active)

19: Stop global dirty log

Implementation

PostCopyFT was implemented on the KVM/QEMU virtualization platform version 2.8.1.1. The guest OS and applications running inside the VM is unmodified. Each VM in KVM/QEMU is associated with a userspace management process, called QEMU, which performs I/O device emulation and various management functions, including migration and checkpointing. The userspace QEMU process communicates with a kernel-space hypervisor, called KVM, which uses hardware virtualization features to execute the VM in guest mode (or non-root mode).

In the implementation, a heartbeat mechanism is used as a simple detector to detect source, destination and network failure. The heartbeat module was implemented as a separate thread on the source and destination host to continuously monitor the availability of the destination/source node accordingly by periodically sending ping requests to the destination/source and wait for response. If the heartbeat module cannot reach the destination/source host for a time interval, then destination/source node is considered to be in a crashed or failure state and the source/destination host immediately triggers a failover. This will result the source/destination to start restoring VM from the latest available state.

PostCopyFT-forward: Forward Checkpointing

PostCopyFT-forward was implemented based on the pre-copy implementation in KVM/QEMU. During the normal execution of VMs, PostCopyFT-forward periodically captures the memory state of a running VM incrementally transfers checkpoint state to in-memory checkpoint cache store. The first checkpoint round in PostCopyFT-forward captures the entire VM's memory footprint and the successive rounds capture the modified memory pages in the previous rounds. When VM is suspended at source during post-copy migration, PostCopyFT-forward captures the final checkpoint to reflect the latest VM state and commit to the in-memory checkpoint cache. This final checkpoint state capture runs in parallel to the migration process.

PostCopyFT-forward also utilizes the dirty page tracking mechanism in KVM/QEMU to identify the modified memory pages of the VM in userspace as part of the checkpointing processes. The dirty page tracking mechanism represents the VM's memory content as a bitmap, in which each bit specifies whether a guest page is modified or not. PostCopyFT-forward utilizes separate bitmap that is different to migration bitmap to identify VM's incremental memory state updates. These bitmaps are synchronized with the physical memory to reflect the latest modified pages on the userspace bitmaps. In PostCopyFT-forward, memory checkpoints are incremental and memory footprint of the VM is scattered over all captured checkpoints.

Live Restoration in PostCopyFT-Forward

During the migration, PostCopyFT-forward restoration manager is implemented as a separate thread that starts loading pre-captured memory checkpoints as a safeguard step to minimize failover time upon a failure. Further, a separate thread starts accounting successfully loaded memory during migration. Upon source failure, the restoration process on the destination machine tries to restore the VM from the forward snapshots. To reform the latest complete memory footprint of the VM, all memory checkpoints are merged to avoid duplicates and to showcase the latest VM state. Then restoration manger compares already loaded memory pages from the migration to find the missing memory pages from the destination VM. Then pages that have not yet been sent to the destination (i.e., delta difference) is loaded by the restoration manager. During this live restoration VM continues to execute.

PostCopyFT-Reverse: Reverse Checkpointing

In parallel to the post-copy migration, PostCopyFT-reverse was implemented as a simultaneous thread to keep track of the modified memory and execution states of the VM running on the destination node, and transfer them to the checkpoint cache store on an external node. This parallel execution helps to rapidly offload the memory and execution state of the migrating VM to the checkpoint cache store upon the destination failure.

Checkpoint Cache Store

Several key factors are considered when selecting a checkpoint cache for PostCopyFT. In order to store both forward and reverse incremental snapshots, the checkpoint cache store should be an in-memory cache, provide duplicate filtering, and allow for checkpoint versioning. Upon a failure situation, the checkpoint cache store may contain several captured checkpoints in different checkpoint iterations. The checkpoint cache needs to maintain each VM checkpoint iteration separately along with version that represents the most recently committed checkpoint. That way, there is no ambiguity as to if a checkpoint is complete or not, and partially incomplete checkpoints can be discarded if a failure occurs in the middle of a checkpoint. To implement PostCopyFT checkpoint cache, Redis{85} in-memory distributed key-value store was used. The redis clients reside on the source and destination hosts, while the Redis server resides on the backup staging node which excludes source or destination nodes. The checkpointed memory state is stored in a map data structure in Redis. Each completed state in checkpoint cycle is separated with a version number to denote the checkpoint iteration and each memory page is stored as key-value pair. Once the checkpoint is transferred to the checkpoint cache store, a version number is updated to uniquely identify each completed checkpointed state.

To improve the resiliency of checkpoint cache, existing fault-tolerant techniques may be used, such as running multiple masters as backups or use continuous health monitoring techniques such as Redis Sentinel to detect failure of checkpoint cache beforehand.

Capturing Memory and Execution State

To begin with PostCopyFT-reverse at the destination node, a network barrier is inserted to buffer outgoing network packets from the VM. The checkpointing thread then periodically sends the memory state of the VM to the staging node. In the KVM/QEMU post-copy migration, dirty page tracking is performed only at the source node to capture the modified memory state of the migrated VM. In PostCopyFT-reverse, dirty page tracking also needs to be performed at the destination in order to send the reverse incremental snapshots of the VM to the checkpoint cache store. As to result, traditional post-copy implementation was modified to enable dirty page tracking on the destination. Once the VM resumes on the destination, the reverse checkpoint thread on QEMU makes an ioctl call to inform KVM to start dirty page tracking. During each checkpointing cycle, to capture the modified VM states, PostCopyFT-reverse uses another ioctl call to synchronize the QEMU bitmap with the KVM physical bitmap. PostCopyFT-reverse then captures the modified memory state by reading the destination QEMU bitmap and transfers the snapshot to the checkpoint cache store.

The execution state of a VM consists of the vCPU state and the I/O device state, which keeps changing during the execution of the VM. PostCopyFT-reverse captures the execution state of the VM and writes it to a channel buffer. The channel buffer is an in-memory buffer that provides facility to perform I/O operations to and from memory buffers. The channel buffer then transfers the entire device/CPU state (as a blob of all devices state) to the checkpoint cache store.

An event-based reverse checkpointing algorithm was implemented to reduce the network packet buffering latency. The event-based approach checkpoints system states only when an external event (e.g., outgoing network packets) is triggered. As a result, the event-based approach captures the system state less frequently and hence reduces the migration downtime. In addition, because the network-intensive workload usually does not dirty significant amount of memory pages, the event-based approach synchronizes the memory state only once during the checkpoint cycle (i.e., when the VM is paused), which reduces the overhead of delaying network packets.

Reducing Migration Downtime in PostCopyFT-Reverse

Traversing the bitmap for modified pages and transferring each modified page to the checkpoint cache increases the migration downtime due to the synchronization overhead of each write request. As the VM memory size grows, the time taken to transfer each memory page also increases. To address this issue, instead of sending the checkpoint directly to the checkpoint cache store on the source host, the entire memory state (i.e., memory, vCPU and I/O state) is first transferred to an in-memory, dynamically sized local data structure checkpoint stage at the destination and then to the checkpoint store. The checkpoint stage is similar to the implementation of Linux Kernel cache-slab. It consists of a vector of pointers that point to contiguous memory chunks. Each memory chunk contains a series of page data and page keys. Once all chunks are filled, the list is doubled, and new chunks are allocated. Storing checkpointed state locally reduces the VM downtime caused by the synchronous writes and provides the assurance of completeness in checkpoint. As this extra memory layer contains the complete VM state, the VM can resume and transfer the checkpointed state to the checkpoint cache store and release buffered network packets.

Restoration in PostCopyFT-Reverse

Once PostCopyFT-reverse detects the destination or network failure in post-copy, the restoration process on the source machine tries to restore the VM from the reverse snapshots. The restoration process was implemented as a non-live process. To build up the entire memory state, PostCopyFT restoration process loads the memory checkpoints into a hash table. The hash table provides the capability to update the memory page with the most recently modified the page and merge duplicate pages. Restoration thread then loads the memory pages and places the page content on the allocated and mapped host memory address space using mmap( ). Next, the restoration process loads the most recent execution state of the VM received from the destination and resumes the VM.

It is therefore an object to provide a method for migration of a virtual machine from a source to a destination, comprising: communicating an execution state of the virtual machine at the source to the destination; initiating execution of the tasks by the virtual machine on the destination, based on the communicated state of the central processing unit of the virtual machine; transferring a respective memory page from the source to the destination, during a migration of the virtual machine, in response to a page fault occurring at the destination due to unavailability of the respective memory page; communicating an incremental checkpoint from the virtual machine executing on the destination; and recovering the virtual machine at the source based on at least the communicated incremental checkpoint from the virtual machine executing on the destination.

The incremental checkpoint may comprise memory pages modified by the virtual machine since a prior incremental checkpoint, a central processing unit state of the virtual machine, and an input/output state of the virtual machine.

The incremental checkpoint may be communicated from the virtual machine executing on the destination is a consistent checkpoint.

The method may further comprise determining if the virtual machine on the destination fails successful migration, recovering the virtual machine at the source in event of migration failure.

The virtual machine at the source may be recovered in a state subsequent to the initiation of execution of the tasks by the virtual machine on the destination.

The method may further comprise proactively transferring memory pages from the source to the destination independent of occurrence of a page fault.

The method may further comprise ceasing execution of tasks by the virtual machine on the source before initiating execution of the tasks by the virtual machine on the destination.

The method may comprise defining an execution state of the virtual machine at the source before communicating the execution state of the virtual machine.

The method may further comprise communicating the incremental checkpoint from the virtual machine executing on the destination, while the migration of the virtual machine to the destination is in progress.

The method may further comprise providing a checkpoint manager, configured to: store a checkpoint of the source comprising the execution state of the virtual machine at the source; and store the incremental checkpoint from the virtual machine executing on the destination.

The method may further comprise communicating the incremental checkpoint from the virtual machine executing on the destination to a checkpoint manager.

The method may further comprise concurrently transferring the respective memory page from the source to the destination in response to the page fault occurring at the destination and communicating the incremental checkpoint from the virtual machine executing on the destination.

The method may further comprise, prior to initiating execution of the tasks by the virtual machine on the destination, obtaining a series of incremental checkpoints of the source.

The series of incremental checkpoints of the source may be stored in a checkpoint cache store.

The communicated incremental checkpoint from the virtual machine executing on the destination may comprise a virtual CPU state, an input/output state, and a memory state of the virtual machine executing on the destination prior to persistently changing a state outside of the virtual machine since a prior communicated incremental checkpoint.

The incremental checkpoint may be committed before external environment changes are made by the virtual machine.

The incremental checkpoint from the virtual machine executing on the destination may be communicated periodically.

The method may further comprise remapping input/output in a communication device between the source and the destination.

The method may further comprise, upon initiation of migration of the virtual machine, informing a system resource which manages input/output operations of migration between the source and the destination.

The method may further comprise, upon initiation of migration of the virtual machine, informing a system resource which manages memory pages of migration between the source and the destination.

The incremental checkpoint from the virtual machine executing on the destination may consist essentially of memory pages modified since a prior incremental checkpoint, a CPU state of the virtual machine and an input/output state of the virtual machine.

The method may further comprise buffering communications from the virtual machine at the destination before committing changes to an external environment until an incremental checkpoint is communicated reflecting the prospective changes to the external environment.

The communicated incremental checkpoint may be stored in a checkpoint store as an in-memory key-value.

The incremental checkpoint may be communicated is response to uncommitted input/output requests of the virtual machine at the destination.

The method may further comprise ceasing communicating the incremental checkpoint from the virtual machine executing on the destination when migration of the virtual machine to the destination is complete.

Contingent upon a failure of the virtual machine at the destination, or communications with the virtual machine at the destination, the source may initiate recovery of the virtual machine.

The method may further comprise communicating heartbeat messages from the destination to the source, and in event of failure of the heartbeat messages, triggering recovery of the virtual machine at the source.

The method may further comprise communicating acknowledgement of memory page receipt from the destination to the source, and in event of failure of the acknowledgement of memory page receipt, triggering recovery of the virtual machine at the source.

The virtual machine may execute under a hypervisor, and heartbeat messages may be communicated from the destination to the source, and in event of failure of the heartbeat messages, trigger recovery of the virtual machine by the hypervisor at the source.

The virtual machine executes under a hypervisor, and acknowledgement of memory page receipt from the destination to the source communicated, and in event of failure of the acknowledgement of memory page receipt, trigger recovery of the virtual machine by the hypervisor at the source.

The virtual machine may execute under a hypervisor, and the incremental checkpoint may be generated by the hypervisor.

A liveness of the destination may be determined based on a series of heartbeat messages, a failure of the migration determined dependent on loss of a plurality of heartbeat messages.

The method may further comprise recovering the virtual machine at the source selectively dependent on the loss of the plurality of heartbeat messages.

The method may further comprise recovering the virtual machine at the source from a last consistent copy of each memory page from a series of incremental checkpoints from the destination.

The method may further comprise resuming execution of the virtual machine at the source.

The method may further comprise buffering outgoing packet transmissions to external resources between successive incremental checkpoints.

The method may further comprise immediately delivering incoming network from external resources between successive incremental checkpoints.

The method may further comprise pausing execution of the virtual machine at the destination during generation of an incremental checkpoint.

The method may further comprise transmitting the buffered outgoing packets after resumption of execution of the virtual machine.

The incremental checkpoint may be generated within a hypervisor that supports the virtual machine as a separate thread or process from the virtual machine.

The source may proactively communicate memory pages to the destination absent a memory fault with respect to the memory pages, concurrently with receipt of the incremental checkpoint.

The source may communicate memory pages to the destination, concurrently with receipt of the incremental checkpoint, while execution of the virtual machine is paused.

The method may further comprise determining memory pages modified by the virtual machine at the destination, generating a first partial checkpoint of the modified memory pages without pausing execution of the virtual machine, pausing execution of the virtual machine, generating a second partial checkpoint of an execution state of a CPU of the virtual machine, an input/output state of the virtual machine, and modified memory pages, combining the first and second partial checkpoints to generate the incremental checkpoint with subsequent modified memory pages preempting preceding modified memory pages, and resuming execution of the virtual machine.

The method may further comprise storing the first and second partial checkpoints locally at the destination.

The incremental checkpoint may be stored in a checkpoint store separate from the source. The source may load a plurality of incremental checkpoints from the destination to build a virtual machine image concurrent with execution of the virtual machine at the destination.

It is also an object to provide a virtual machine, comprising: a communication port configured to receive an initial execution state of the virtual machine and to transmit incremental checkpoints; and an automated processor configured to: execute the virtual machine initiated based on the initial execution state, request memory pages through the communication port in response to page faults during execution of the virtual machine, and transmit incremental checkpoints through the communication port.

The incremental checkpoint may be a consistent checkpoint.

The communicated incremental checkpoint may comprise a virtual CPU state, an input/output state, and a memory state of the virtual machine prior to persistently changing a state outside of the virtual machine since a prior communicated incremental checkpoint.

The system may further comprise a buffer, configured to store requests for persistent change of an external environment, wherein the automated processor is configured to generate the incremental checkpoint and then communicate the buffered requests for persistent change of the external environment reflected in the incremental checkpoint.

The incremental checkpoint may consist essentially of memory pages modified since a prior incremental checkpoint, a CPU state of the virtual machine and an input/output state of the virtual machine.

The automated processor may be configured to communicate the incremental checkpoint is response to uncommitted input/output requests of the virtual machine.

The automated processor may be further configured to communicate acknowledgement of memory page receipt.

The virtual machine may execute under a hypervisor, wherein the incremental checkpoint is generated by the hypervisor.

The virtual machine may execute under a hypervisor, wherein the incremental checkpoint is generated by the hypervisor and wherein a portion of the incremental checkpoint is defined without pausing the virtual machine.

The system may further comprise a hypervisor which executes the virtual machine as a first thread or process, wherein the incremental checkpoint is generated within the hypervisor in a second thread or process distinct from the first thread or process.

The automated processor may be configured to generate the incremental checkpoint in two phases, a first phase comprising determining memory pages modified by the virtual machine during which the virtual machine is operational; and a second phase comprising execution state of a CPU of the virtual machine, an input/output state of the virtual machine, and modified memory pages, during which the virtual machine is paused.

It is another object to provide a system for migration of a virtual machine from a source to a destination, comprising: a memory configured to store an execution state of the virtual machine; a communication port configured to communicate the stored execution state to a destination; an automated processor at the destination configured to: execute the virtual machine at the destination, based on the communicated execution state, request memory pages in response to page faults due to incomplete migration, during execution of the virtual machine, and communicate incremental checkpoints to a checkpoint manager, while the migration of the virtual machine to the destination is in progress; and an automated processor configured to recover the virtual machine in event of failure of migration based on at least a preceding checkpoint communicated from the destination to the checkpoint manager.

It is a still further object to provide a virtual machine system, comprising: a memory configured to store an execution state of the virtual machine; a communication port configured to communicate the stored execution state; and an automated processor configured to: execute the virtual machine, request memory pages in response to page faults due to incomplete migration during execution of the virtual machine, and communicate incremental checkpoints through the communication port.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A shows the architecture of PostCopyFT.

FIG. 1B shows a timeline of PostCopyFT Migrations.

FIG. 2 shows the total migration time of PostCopyFT and Post-copy for migrating 1 GB-8 GB idle VMs.

FIG. 3 shows the downtime and replication time for migrating idle VMs with different sizes.

FIG. 4 shows the total migration time of migrating memory write-intensive VMs using post-copy and PostCopyFT.

FIG. 5 shows the downtime and replication time for migrating memory-write intensive VMs using PostCopyFT.

FIG. 6 shows the impact of checkpointing interval on the total migration time (idle VM).

FIG. 7 shows the impact of checkpointing interval on the migration downtime and the replication time (idle VM).

FIG. 8 shows the impact of checkpointing interval on the total migration time when migrating a 5 GB memory-write intensive VM.

FIG. 9 shows the impact of checkpointing interval on downtime and replication time when migrating a 5 GB memory-write intensive VM.

FIG. 10 shows the performance impact of post-copy and PostCopyFT on a CPU-intensive QuickSort benchmark.

FIG. 11 shows outgoing network bandwidth of post-copy, PostCopyFT with periodic checkpointing, and PostCopyFT with event-based checkpointing, when the migrating VM generates outgoing TCP traffic.

FIG. 12 shows outgoing network bandwidth of post-copy, PostCopyFT with periodic checkpointing, and PostCopyFT with event-based checkpointing, when the migrating VM generates outgoing UDP traffic.

FIG. 13 shows incoming network bandwidth of post-copy, PostCopyFT with periodic checkpointing, and PostCopyFT with event-based checkpointing, when the migrating VM receives incoming TCP traffic.

FIG. 14 shows incoming network bandwidth of post-copy, PostCopyFT with periodic checkpointing, and PostCopyFT with event-based checkpointing, when the migrating VM receives incoming UDP traffic.

FIG. 15 shows the time taken for recovering a memory write-intensive VM from checkpoints with different sizes.

FIG. 16 shows a flowchart according to the present invention.

FIG. 17 shows a schematic drawing according to the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

PostCopyFT was implemented on the KVM/QEMU {3}, {22} virtualization platform version 2.8.1.1. The guest OS and applications running inside the migrating VM are unmodified. Each VM in KVM/QEMU is associated with a userspace management process, called QEMU, which performs I/O device emulation and various management functions such as VM migration and checkpointing. The userspace QEMU process communicates with a kernel-space hypervisor called KVM, which uses hardware virtualization features to execute the VM in guest mode (or non-root mode).

A. Capturing VM's Memory and Execution State

In parallel to forward post-copy migration, the reverse incremental checkpointing mechanism is implemented as a concurrent QEMU thread to capture consistent incremental checkpoints. To track modified memory pages for successive incremental checkpoints, PostCopyFT performs dirty page tracking on the VM executing at the destination node. The dirty page tracking mechanism represents the VM's memory content as a bitmap, in which each bit specifies whether a page is modified or not. KVM uses this bitmap to identify pages dirtied by the VM at a given instant. When the VM resumes on the destination host during post-copy migration, the reverse checkpointing thread informs KVM to mark all memory pages as read-only using an ioctl call. When the VM attempts to write to any of its resident memory pages, a write fault is triggered, upon which KVM marks the page as read-write and turns on the corresponding bit in the dirty bitmap. Successive writes to the same page do not trigger any bitmap updates until the bitmap is reset for the next checkpointing round. In each checkpointing iteration, the reverse checkpointing thread in QEMU retrieves the current dirty bitmap from KVM to userspace for better manageability. Once QEMU transfers dirty pages to the checkpoint store, the corresponding dirty bits are cleared.

The execution state of a VM consists of the CPU and the I/O device states, which keeps changing during the execution of the VM. PostCopyFT modifies QEMU's default method of capturing the VM's execution state to include the corresponding checkpoint version number (used to uniquely identify a checkpoint cycle) in an in-memory QEMUFile data structure. The QEMUFile structure is then transferred to the checkpoint store.

Between successive checkpointing rounds, a network barrier is inserted to buffer outgoing network packets of the migrating VM. An event-based VM checkpointing algorithm was implemented to reduce the network packet buffering latency. In the event-based approach, VM checkpointing is triggered only when an external event occurs, such as when the VM sends an outgoing network packet. As a result, the event-based approach captures the system state less frequently and hence reduces the migration downtime.

B. Reducing the Migration Downtime

Traversing the bitmap for modified pages and transferring each modified page to the checkpoint store increases the VM migration downtime due to the synchronization overhead of each write request. To address this issue, instead of sending the checkpoint directly to the checkpoint store on the source host, the incremental memory changes are first captured in an in-memory dynamically sized local data structure checkpoint stage. The checkpoint stage is similar to the implementation of Linux Kernel cache-slab {4}, {26}. It consists of a vector of pointers that point to contiguous memory chunks. Each memory chunk contains a series of page data and page keys. Once all chunks are filled, the list is doubled, and new chunks are allocated. Storing memory states locally reduces the VM downtime caused by the synchronous writes and provides the assurance of completeness in checkpoints.

C. Checkpoint Store

Once the VM's memory states are captured in a checkpointing cycle, memory states stored in checkpoint stage are transferred to the checkpoint store at the source node. Several key factors were considered when selecting a checkpoint store. In order to store incremental checkpoints, the checkpoint store should be an in-memory cache that provides duplicate filtering and allows for checkpoint versioning. During a failure situation, the checkpoint store may contain several checkpoints captured in different checkpoint iterations. The checkpoint store needs to maintain each VM checkpoint iteration separately along with a version number that represents the most recently committed checkpoint. That way, there is no ambiguity as to if a checkpoint is complete or not, and incomplete checkpoints discarded if a failure occurs in the middle of a checkpoint. Redis {34}, an in-memory distributed key-value store, was used to implement PostCopyFT's checkpoint store. The Redis client resides on the destination host while the Redis server resides on the source or the staging node. The checkpointed memory state is stored in a map data structure in Redis. Each memory page is stored as a key-value pair using function HMSET, where the key is a unique page address consisting of page block ID and page offset, and the value is the page content. Once the checkpoint is transferred to the checkpoint store, the checkpoint is marked as complete.

D. Detection of Destination/Network Failure

PostCopyFT uses the heartbeat mechanism as a simple detector to detect the destination/network failure. The heartbeat module was implemented as a separate thread on the source host that continuously monitors the availability of the destination node by periodically sending ping requests to the destination. If the heartbeat module cannot reach the destination host for a time interval, then the destination node is considered be in a failure state and the source host immediately triggers a failover.

E. VM Recovery after Failure

After detecting the migration failure, the restoration process on the source machine tries to restore the VM from the checkpoints previously received from the destination. Restoration was implemented as a non-live process. To build up the entire memory state, the restoration process loads the memory pages for each checkpoint version up to the latest committed complete checkpoint version number at once into a hashtable using the HGETALL function provided in Redis. The hashtable provides the capability to update memory pages with the most recently modified pages and merge duplicate pages. The restoration thread then places the page content on the allocated and mapped host memory address space using mmap( ). Next, the restoration process loads the most recent execution state of the VM using the GET function and loads the execution state into a in-memory QEMUFile structure. This file is then passed to the modified qemu loadvm section start full( ) function, which proceeds to unpack and load the execution state. Finally, the VM is resumed.

Evaluation

The performance results of PostCopyFT are evaluated using the following metrics:

- Total migration time: Time taken to transfer a VM's state entirely from the source to the destination host.
- Downtime: Duration that a VM is suspended during the migration.
- Replication time: Time taken to transfer the checkpoint to the checkpoint cache store.
- Application performance degradation: The performance impact on applications running inside the VM during migration and checkpointing.
- Network Bandwidth degradation: Reduction in network bandwidth during migration and checkpointing.
- Failover time: Time taken to restore the VM from the last committed checkpointing during a failure situation.

The test environment consists of dual six-core 2.1 GHz Intel Xeon machines with 128 GB memory connected through a Gigabit Ethernet switch with 1 Gbps full-duplex ports.

To avoid network interference and contention between the migration traffic and the application-specific traffic, separate NIC interfaces are used for the migration and the application traffic. VMs in each experiment are configured with 1 vCPU and 8 GB of memory with 4 KB page size unless specified otherwise. Virtual disks are accessed over the network from an NFS server, which enables each VM to access its storage from both the source and the destination machines over local area network.

A. Baseline Comparison of PostCopyFT and Post-Copy

To measure the total migration time and the downtime of PostCopyFT, the checkpointing thread ran continuously on the destination host without triggering a failure in the migration in order to evaluate the overhead of VM checkpointing.

FIG. 2 compares the time taken to migrate an idle VM using PostCopyFT and the KVM/QEMU vanilla post-copy implementation {16}, {17}. The size of the VM ranges from 1 GB to 8 GB. As shown in FIG. 2, the total migration time of PostCopyFT and post-copy is almost the same for idle VMs with different sizes.

FIG. 3 shows the downtime and the replication time of migrating an idle VM using PostCopyFT. The downtime of PostCopyFT for migrating an idle VM ranges between 1.1 seconds and 1.9 seconds, which is higher than that of post-copy (9 ms-11.6 ms). This is because the VM pauses continuously for every checkpointing interval (100 μs) to checkpoint memory states. The figure also shows that the replication time is much higher than the downtime. This is because VM checkpointing is performed in two stages and only stage II pauses the VM.

FIG. 4 shows the total migration time for migrating a memory-write intensive VM using PostCopyFT and post-copy. The memory-write intensive application running inside the VM is a C program that continuously writes random numbers to a large region of main memory. The size of the working set (i.e., the size of the memory written) ranges from 1 GB to 5 GB. FIG. 4 shows that the total migration time of PostCopyFT is almost the same as post-copy. This is because, even when the VM is paused for checkpointing, the destination is actively gathering pages from the source host.

FIG. 5 shows the downtime and the replication time of PostCopyFT for migrating a memory-write intensive VM. The downtime of PostCopyFT ranges between 1.8 s and 2.2 s, which is higher than that of post-copy (7 ms-9 ms). The downtime respect to migration thread is almost constant for PostCopyFT and post-copy as the migration process is not interrupted by PostCopyFT during its checkpointing phase. But with the added overhead due to frequent checkpointing, the migration downtime of PostCopyFT increases. FIG. 5 also shows that the replication time increases when the size of the working set increases. This is because when the size of the working set increases, the number of dirty pages also increases.

B. Impact of Checkpointing Interval

The impact of the checkpointing interval on VM migration and the overhead incurred due to checkpointing is evaluated.

FIG. 6 shows the total migration time of migrating an idle VM when the checkpointing interval varies between 0.1 ms and 100 ms. Checkpointing interval 0 refers to the vanilla post-copy migration. PostCopyFT imposes 0.2%-0.9% overhead compared to the post-copy migration and the overhead decreases linearly when the checkpointing interval increases.

FIG. 7 shows the downtime and the replication time of PostCopyFT by varying the checkpointing interval for idle VMs. FIG. 7 shows that when the checkpointing interval decreases, the downtime increases. This is because, when the checkpointing interval is low, the checkpointing is performed frequently with the overhead of bitmap synchronization, state transfer, network buffering, and the pause of the VM.

FIG. 8 shows the total migration time of migrating a 5 GB memory-write intensive VM when the checkpointing interval varies between 0.1 ms and 100 ms. Checkpointing interval 0 refers to the post-copy migration. PostCopyFT imposes 0.1%-0.7% overhead compared to the post-copy migration and the overhead decreases linearly when the checkpointing interval increases.

FIG. 9 shows the downtime and the replication time of migrating the 5 GB memory-write intensive VM. The figure shows that the downtime and the replication time decrease when the checkpointing interval increases. This is because, when the checkpointing interval decreases, more memory pages get dirty, which results in higher bitmap synchronizations and larger checkpoint state.

C. Performance Impact on CPU-Intensive Workload

PostCopyFT and post-copy affect the performance of CPU-intensive workloads were measured using the QuickSort benchmark, which repeatedly allocated 400 Bytes of memory, wrote random integers to the allocated memory segment, and sorted the integers using the QuickSort algorithm. The number of sorts performed (i.e., the number of times the random integers are written to memory and are sorted) per second during the migration was measured.

FIG. 10 shows that PostCopyFT has similar performance as post-copy and there is no observable adverse impact of the reverse incremental checkpointing mechanism during migration.

D. Network Buffering Overhead

VM migration itself is a network intensive process. To mitigate the contention of network resources between the migration process and the applications running inside the VM, PostCopyFT uses separate NIC interfaces for the migration traffic and the application traffic. The impact of PostCopyFT's periodic and event-based checkpointing mechanisms on network-intensive VMs is considered. As the event-based checkpointing mechanism captures the VM's memory states only when the VM transmits an outgoing network packet, compared to the periodic checkpointing, the event-based checkpointing reduces the network latency and the time taken to pause/restart the VM and synchronize the dirty page bitmap. The event based checkpointing mechanism shows 2-3 times lower downtime than the periodic checkpointing mechanism.

iPerf {18}, a network intensive application to measure the outgoing and incoming network bandwidth of PostCopyFT, was used, where the maximum network bandwidth is set to 100 Mbit/s. The iPerf server runs on an external machine (i.e., neither source nor destination host) in the same cluster and the iPerf client runs inside the migrating VM. During the migration, the client continuously sends data to the server through a TCP and UDP connection as shown in FIGS. 11 and 12. The network bandwidth is captured using iPerf every 0.1 second. Bandwidth fluctuations are seen for both TCP based and UDP based outgoing network traffic due to the impact of network buffering and releasing. Compared to the periodic checkpointing mechanism, the event-based checkpointing mechanism shows more frequent buffer releases and hence has higher bandwidth. Low bandwidth throughput is shown in the TCP based outgoing traffic due to TCP acknowledgments and timeouts.

When migrating an incoming network-intensive VM, the iPerf server runs inside the migrating VM and the iPerf client runs on an external machine in the same cluster. FIGS. 13 and 14 give the network bandwidth of incoming network intensive applications for TCP and UDP connections, respectively. For the TCP protocol, although PostCopyFT serves incoming network packets, the acknowledgment packets are being buffered as they are considered as outgoing network packet. Thus, bandwidth fluctuations are seen in the incoming TCP network traffic workloads due to the acknowledgment packets. Compared to the periodic checkpointing mechanism, the event-base checkpointing mechanism shows much higher bandwidth.

E. Failover Time

Upon detecting a failure during the migration of a VM, the time taken to recover the VM from the latest available committed state is called the failover time. FIG. 15 shows the failover time when migrating a 5 GB memory-write intensive VM using PostCopyFT. The number of checkpointed pages and captured the time taken to restore the VM on the source host was measured. FIG. 15 shows that, the failover time increases linearly when the size of the checkpoint increases.

F. System and Method Description

FIG. 16 shows a method for migration of a virtual machine from a source to a destination, comprising: communicating an execution state of the virtual machine at the source to the destination 101; initiating execution of the tasks by the virtual machine on the destination, based on the communicated state of the central processing unit of the virtual machine 102; transferring a respective memory page from the source to the destination in response to a page fault occurring at the destination due to unavailability of the respective memory page 103; communicating an incremental checkpoint from the virtual machine executing on the destination 104; and recovering the virtual machine at the source based on at least the communicated incremental checkpoint from the virtual machine executing on the destination 105.

FIG. 17 shows a schematic drawing of a system according to the present invention. A communication port 201 and memory 206 are provided. An automated processor 202, which may be an AMD ×64 architecture, Intel ×32 architecture, ARM Cortex architecture, etc., is provided to execute virtual machine 203, request memory pages responsive to page faults 204, and transmit incremental checkpoints 205. Another automated processor 207 is provided, communicating with the automated processor 202 through the communication port 201, capable of recovering the virtual machine in event of failure 208.

With reference to the computer-readable media, including both memory device and secondary computer-readable medium, a computer-readable media and the processor are structurally and functionally interrelated with one another. The computer-readable media may be implemented using a hard disk drive (rotating magnetic media), a memory device, flash memory, solid state storage, random access memory, and/or a variety of other computer-readable media known in the art, and when including functional descriptive material, data structures are created that define structural and functional interrelationships between such data structures and the computer-readable media. Such interrelationships permit the data structures' functionality to be realized. For example, in one embodiment the processor reads (e.g., accesses or copies) such functional descriptive material from the network interface, the computer-readable media, and the information processing system (more particularly, at least one processor) performs its operations, as described elsewhere herein, in response to such material stored in the memory device of the information processing system. In addition to reading such functional descriptive material from the computer-readable medium, the processor is capable of reading such functional descriptive material from (or through) the network. The information processing system may include at least one type of computer-readable media that is non-transitory. For explanatory purposes below, singular forms such as “computer-readable medium,” “memory,” and “disk” are used, but it is intended that these may refer to all or any portion of the computer-readable media available in or to a particular information processing system, without limiting them to a specific location or implementation.

VII. CONCLUSION

A solution is presented for recovering a virtual machine when post-copy live migration fails due to destination or network failure. Reverse incremental checkpointing is provided, called PostCopyFT, which proceeds concurrently with traditional post-copy migration and transfers incremental changes to a VM's memory and execution state from the destination back to the source, either periodically or upon external I/O events. Upon network or destination failure, PostCopyFT recovers the VM on the source node from the latest consistent checkpoint. The presented implementation of PostCopyFT in the KVM/QEMU platform yields similar total migration time compared to the traditional post-copy live migration with low impact on application performance.

Although the invention(s) have been described with reference to specific embodiments, it will be understood by those skilled in the art that various changes may be made, and equivalents may be substituted, for elements thereof without departing from the true spirit and scope of the invention. In addition, modifications may be made without departing from the essential teachings of the invention. The invention is described by way of various embodiments and features. This disclosure is intended to encompass all consistent combinations, subcombinations, and permutations of the different options and features, as if expressly set forth herein individually.

Each reference cited herein is expressly incorporated herein in its entirety. Such references provide examples representing aspects of the invention, uses of the invention, disclosure of the context of the invention and its use and application. The various aspects disclosed herein, including subject matter incorporated herein by reference, may be employed, in combination or subcombination and in various permutations, consistent with the claims.

The invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described embodiment is to be considered in all respects only illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims rather by the foregoing description. All changes that fall within the meaning and range of equivalency of the claims are to be embraced within their scope.

REFERENCES

Each reference cited herein is expressly incorporated by reference in its entirety.

{1} A. Barak and R. Wheeler. Mosix: An integrated multiprocessor UNIX. In Mobility, pages 41-53. ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 1999.
{2} P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. Xen and the art of virtualization. SIGOPS Oper. Syst. Rev., 37(5):164-177, October 2003.
{3} F. Bellard. Qemu, a fast and portable dynamic translator. In Proc. of USENIX Annual Technical Conference, April 2005.
{4} J. Bonwick and J. Adams. Magazines and vmem: Extending the slab allocator to many cpus and arbitrary resources. In USENIX Annual Technical Conference, pages 15-33, 2001.
{5} T. C. Bressoud and F. B. Schneider. Hypervisor-based fault tolerance. In Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles, SOSP '95, pages 1-11, New York, NY, USA, 1995. ACM.
{6} C. Clark, K. Fraser, S. Hand, J. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield. Live Migration of Virtual Machines. In USENIX Symposium on Networked Systems Design and Implementation (NSDI), pages 273-286, 2005.
{7} B. Cully, G. Lefebvre, D. Meyer, M. Feeley, N. Hutchinson, and A. Warfield. Remus: High availability via asynchronous virtual machine replication. In USENIX Symposium on Networked Systems Design and Implementation (NSDI), pages 161-174, 2008.
{8} U. Deshpande, D. Chan, S. Chan, K. Gopalan, and N. Bila. Scattergather live migration of virtual machines. IEEE Transactions on Cloud Computing, PP(99):1-1, 2015.
{9} U. Deshpande, Y. You, D. Chan, N. Bila, and K. Gopalan. Fast server deprovisioning through scatter-gather live migration of virtual machines. In 2014 IEEE 7th International Conference on Cloud Computing, pages 376-383, June 2014.
{10} Y. Dong, W. Ye, Y. Jiang, I. Pratt, S. Ma, J. Li, and H. Guan. Colo: Coarse-grained lock-stepping virtual machines for non-stop service. In the 4th Annual Symposium on Cloud Computing, pages 1-16, 2013.
{11} G. W. Dunlap, S. T. King, S. Cinar, M. A. Basrai, and P. M. Chen. Revirt: Enabling intrusion analysis through virtual-machine logging and replay. SIGOPS Oper. Syst. Rev., 36:211-224, December 2002.
{12} G. W. Dunlap, III. Execution Replay for Intrusion Analysis. PhD thesis, Ann Arbor, MI, USA, 2006.
{13} C. Engelmann, G. R. Vallee, T. Naughton, and S. L. Scott. Proactive fault tolerance using preemptive migration. In 17th Euromicro International Conference on Parallel, Distributed and Network-based Processing, pages 252-257, 2009.
{14} D. Fernando, Hardik Bagdi, Yaohui Hu, Ping Yang, Kartik Gopalan, Charles Kamhoua, and Kevin Kwiat. “Quick eviction of virtual machines through proactive live snapshots.” In 2016 IEEE/ACM 9th International Conference on Utility and Cloud Computing (UCC), pp. 99-107. IEEE, 2016.
{15} D. Fernando, H. Bagdi, Y. Hu, P. Yang, K. Gopalan, C. Kamhoua, and K. Kwiat. Quick eviction of virtual machines through proactive snapshots. In 2016 IEEE International Conference on Cluster Computing (CLUSTER), pages 156-157, September 2016.
{16} M. Hines, U. Deshpande, and K. Gopalan. Post-copy live migration of virtual machines. In ACM SIGOPS Operating Systems Review, 43, no. 3: 14-26 Jul. 2009.
{17} M. R. Hines and Kartik Gopalan. “Post-copy based live virtual machine migration using adaptive pre-paging and dynamic self-ballooning.” In Proceedings of the 2009 ACM SIGPLAN/SIGOPS international conference on Virtual execution environments (VEE), pp. 51-60. 2009.
{18} Iperf. The TCP/UDP Bandwidth Measurement Tool, dast.nlanr.net/Projects/Iperf/.
{19} R. Jhawar, V. Piuri, and M. Santambrogio. Fault tolerance management in cloud computing: A system-level perspective. IEEE Systems Journal, 7(2):288-297, June 2013.
{20} A. Kangarlou, P. Eugster, and D. Xu. Vnsnap: Taking snapshots of virtual networked infrastructures in the cloud. IEEE Transactions on Services Computing, 5(4):484-496, 2012.
{21} S. T. King, G. W. Dunlap, and P. M. Chen. Debugging operating systems with time-traveling virtual machines. In Annual Conference on USENIX Annual Technical Conference, pages 1-1, 2005.
{22} A. Kivity, Y. Kamay, D. Laor, U. Lublin, and A. Liguori. Kvm: The linux virtual machine monitor. In Proc. of Linux Symposium, June 2007.
{23} Y. Kwon, M. Balazinska, and A. Greenberg. Fault-tolerant stream processing using a distributed, replicated file system. Proc. VLDB Endow., 1(1), August 2008.
{24} M. Lu and T. c. Chiueh. Fast memory state synchronization for virtualization-based fault tolerance. In 2009 IEEE/IFIP International Conference on Dependable Systems Networks, 2009.
{25} D. Marques, G. Bronevetsky, R. Fernandes, K. Pingali, and P. Stodghil. Optimizing checkpoint sizes in the c3 system. In the 19th IEEE International Parallel and Distributed Processing Symposium, 2005.
{26} R. McDougall and J. Maura. Solaris internals. In Rachael Borden, 2001. {27} Micro Checkpointing. wiki.qemu.org/features/microcheckpointing. {28} S. J. Mullender, G. van Rossum, A. S. Tananbaum, R. van Renesse, and H. van Staveren. Amoeba: a distributed operating system for the 1990s. Computer, 23(5):44-53, May 1990.
{29} A. B. Nagarajan, F. Mueller, C. Engelmann, and S. L. Scott. Proactive fault tolerance for hpc with xen virtualization. In the 21st Annual International Conference on Supercomputing, pages 23-32, 2007.
{30} M. Nelson, B. H. Lim, and G. Hutchins. Fast transparent migration for virtual machines. In Proc. of USENIX Annual Technical Conference, April 2005.
{31} J. K. Ousterhout, A. R. Cherenson, F. Douglis, M. N. Nelson, and B. B. Welch. The sprite network operating system. Computer, 21(2):23-36, February 1988.
{32} E. Park, B. Egger, and J. Lee. Fast and space-efficient virtual machine checkpointing. In the 7th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, pages 75-86, 2011.
{33} R. F. Rashid and G. G. Robertson. Accent: A communication oriented network operating system kernel. In the Eighth ACM Symposium on Operating Systems Principles, pages 64-75, 1981.
{34} Redis. In-memory data structure store, redis.io/.
{35} A. Ruprecht, D. Jones, D. Shiraev, G. Harmon, M. Spivak, M. Krebs, M. Baker-Harvey, and T. Sanderson. Vm live migration at scale. In the 14th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, pages 45-56.
{36} Y. Tamura, K. Sato, S. Kihara, and S. Moriai. Kemari: Virtual machine synchronization for fault tolerance using domt,” ntt cyber space labs. Technical report, Proc. USENIX Annual Technical Conference, 2008.
{37} VMWare Inc. VMWare IncArchitecture of VMware ESXi, www.vmware.com/files/pdf/esxi architecture.pdf.
{38} VMWare Inc. VMWare vMotion www.vmware.com/pdf/vmotion datasheet.pdf.
{39} VMWare Inc. VMWare vSphere 6 Fault Tolerance www.vmware.com/files/pdf/techpaper/vmware-vsphere6-ft-arch-perf.pdf. {40} C. Wang, X. Chen, Z. Wang, Y. Zhu, and H. Cui. A fast, general storage replication protocol for active-active virtual machine fault tolerance. In 2017 IEEE 23rd International Conference on Parallel and Distributed Systems (ICPADS), pages 151-160, December 2017.
{41} C. Wang, F. Mueller, C. Engelmann, and S. L. Scott. Proactive process-level live migration in hpc environments. In International Conference for High Performance Computing, Networking, Storage and Analysis, 2008.
{42} Min Xu, Rastislav Bodik, and Mark D. Hill. A, “flight data recorder” for enabling full-system multiprocessor deterministic replay. In the 30th International Symposium on Computer Architecture, pages 122-135, 2003.
{43} Abe, Yoshihisa, Roxana Geambasu, Kaustubh Joshi, and Mahadev Satyanarayanan. “Urgent virtual machine eviction with enlightened post-copy.” ACM SIGPLAN Notices 51, no. 7 (2016): 51-64.
{44} Asai, Hirochika. “P2V Migration with Post-Copy Hot Cloning for Service Downtime Reduction.” In 2013 International Conference on Cloud and Green Computing, pp. 1-8. IEEE, 2013.
{45} Ben-Yehuda, Muli, Rom Frieman, Abel Gordon, Benoit Hudzia, and Maor Vanmak. “Resilient post-copy live migration using eviction to shared storage in a global memory architecture.” U.S. patent application Ser. No. 14/588,424, filed Apr. 7, 2016.
{46} Chen, Xiao, Jian-Hui Jiang, and Qu Jiang. “A Method of Self-Adaptive Pre-Copy Container Checkpoint.” In 2015 IEEE 21st Pacific Rim International Symposium on Dependable Computing (PRDC), pp. 290-300. IEEE, 2015.
{47} Chou, Chih, Yuan Chen, Dejan Milojicic, Narasimha Reddy, and Paul Gratz. “Optimizing Post-Copy Live Migration with System-Level Checkpoint Using Fabric-Attached Memory.” In 2019 IEEE/ACM Workshop on Memory Centric High Performance Computing (MCHPC), pp. 16-24. IEEE, 2019.
{48} Dadrwal, Ashu, Suryaprakash Nehra, Ali Ahmad Khan, and Mohit Dua. “Checkpoint Based Live Migration of Virtual Machine.” In 2018 International Conference on Communication and Signal Processing (ICCSP), pp. 1083-1086. IEEE, 2018.
{49} Deshpande, Umesh, Danny Chan, Steven Chan, Kartik Gopalan, and Nilton Bila. “Scatter-gather live migration of virtual machines.” IEEE Transactions on Cloud Computing 6, no. 1 (2015): 196-208.
{50} Dhuria, Ritika, and Kiranbir Kaur. “A Hybrid Approach For Reducing Downtime, Migration Time And Energy Consumption of Live VM Migration.” International Journal of Advanced Research in Computer Science 8, no. 8 (2017).
{51} Fernando, Dinuni, Jonathan Terner, Kartik Gopalan, and Ping Yang. “Live Migration Ate My VM: Recovering a Virtual Machine after Failure of Post-Copy Live Migration.” In IEEE INFOCOM 2019-IEEE Conference on Computer Communications, pp. 343-351. IEEE, 2019.
{52} Gad, Ramy, Simon Pickartz, Tim SuB, Lars Nagel, Stefan Lankes, Antonello Monti, and André Brinkmann. “Zeroing memory deallocator to reduce checkpoint sizes in virtualized HPC environments.” The Journal of Supercomputing 74, no. 11 (2018): 6236-6257.
{53} Gao, Xing, Jidong Xiao, Haining Wang, and Angelos Stavrou. “Understanding the Security Implication of Aborting Live Migration.” IEEE Transactions on Cloud Computing (2020).
{54} Hines, Michael R., G. Kartik, and D. Umesh. “Post-Copy Live Migration Of VM.” In Proc. of ACM International Conference on Virtual Execution Environments. 2009.
{55} Hirofuchi, Takahiro, Hidemoto Nakada, Satoshi Itoh, and Satoshi Sekiguchi. “Reactive consolidation of virtual machines enabled by postcopy live migration.” In Proceedings of the 5th international workshop on Virtualization technologies in distributed computing, pp. 11-18. 2011.
{56} Hirofuchi, Takahiro, Hidemoto Nakada, Satoshi Itoh, and Satoshi Sekiguchi. “Making vm consolidation more energy-efficient by postcopy live migration.” In Cloud Computing 2011, The Second International Conference on Cloud Computing, GRIDs, and Virtualization, pp. 195-204. 2011.
{57} Hirofuchi, Takahiro, Hidemoto Nakada, Satoshi Itoh, and Satoshi Sekiguchi. “Reactive cloud: Consolidating virtual machines with postcopy live migration.” Information and Media Technologies 7, no. 2 (2012): 614-626.
{58} Hirofuchi, Takahiro, Hidemoto Nakada, Satoshi Itoh, and Satoshi Sekiguchi. “Reactive cloud: Consolidating virtual machines with postcopy live migration.” Information and Media Technologies 7, no. 2 (2012): 614-626.
{59} Hirofuchi, Takahiro, Isaku Yamahata, and Satoshi Itoh. “Postcopy live migration with guest-cooperative page faults.” IEICE TRANSACTIONS on Information and Systems 98, no. 12 (2015): 2159-2167.
{60} Hursey, Joshua. “A Transparent Process Migration Framework for Open MPI.” (2012).
{61} Lu, Peng, Antonio Barbalace, and Binoy Ravindran. “HSG-LM: hybrid-copy speculative guest OS live migration without hypervisor.” In Proceedings of the 6th International Systems and Storage Conference, pp. 1-11. 2013.
{62} Mabe, Dusty. “The State of Virtual Machine Checkpointing, Live Migration and Replication.” (2013)
{63} Mosko, Marc. “Process Migration over CCNx.” arXiv preprint arXiv:1707.04835 (2017).
{64} Olivier, Pierre, AKM Fazla Mehrab, Stefan Lankes, Mohamed Lamine Karaoui, Robert Lyerly, and Binoy Ravindran. “HEXO: Offloading HPC Compute-Intensive Workloads on Low-Cost, Low-Power Embedded Systems.” In Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing, pp. 85-96. 2019.
{65} Rapoport, Mike, and Joel Nider. “User space memory management for post-copy migration.” In Proceedings of the 10th ACM International Systems and Storage Conference, pp. 1-1. 2017.
{66} Rathi, Rakesh. “VM Migration Techniques in Cloud: A Review.” International Journal of Computer Applications 975: 8887. 2019.
{67} Sahni, Shashank, and Vasudeva Varma. “A hybrid approach to live migration of virtual machines.” In 2012 IEEE International Conference on Cloud Computing in Emerging Markets (CCEM), pp. 1-5. IEEE, 2012.
{68} Shah, Syed Asif Raza, Amol Hindurao Jaikar, and Seo-Young Noh. “A performance analysis of precopy, postcopy and hybrid live VM migration algorithms in scientific cloud computing environment.” In 2015 International Conference on High Performance Computing & Simulation (HPCS), pp. 229-236. IEEE, 2015.
{69} Shamsinezhad, Elham, Asadollah Shahbahrami, Alireza Hedayati, Ahmad Khadem Zadeh, and Hamid Banirostam. “Presentation methods for task migration in cloud computing by combination of yu router and post-copy.” International Journal of Computer Science Issues (IJCSI) 10, no. 4 (2013): 98.
{70} Shan, Zhongyuan, Jianzhong Qiao, and Shukuan Lin. “Fix page fault in post-copy live migration with RemotePF page table assistant.” In 2018 17th International Symposium on Distributed Computing and Applications for Business Engineering and Science (DCABES), pp. 40-43. IEEE, 2018.
{71} Shribman, Aidan, and Benoit Hudzia. “Pre-copy and post-copy vm live migration for memory intensive applications.” In European Conference on Parallel Processing, pp. 539-547. Springer, Berlin, Heidelberg, 2012.
{72} Sivagami, V. M. “Efficient Data Transmission during Virtual Machine Failures Using Hybrid Copy Live Migration.” Proceedings of National Conference on Communication and Informatics-2016, Organized by Department of Information Technology, Sri Venkateswara College of Engineering, Sriperumbudur
{73} Soni, Sushil Kumar, and Ravi Kant Kapoor. “Enhanced live migration of virtual machine using comparison of modified and unmodified pages.” IJCSMC 3, no. 2 (2014): 779-787.
{74} Stoyanov, Radostin, and Martin J. Kollingbaum. “Efficient live migration of Linux containers.” In International Conference on High Performance Computing, pp. 184-193. Springer, Cham, 2018.
{75} Su, Kui, Wenzhi Chen, Guoxi Li, and Zonghui Wang. “Rpff: A remote page-fault filter for post-copy live migration.” In 2015 IEEE International Conference on Smart City/SocialCom/SustainCom (SmartCity), pp. 938-943. IEEE, 2015.
{76} Sun, Michael H., and Douglas M. Blough. Fast, lightweight virtual machine checkpointing. Georgia Institute of Technology, 2010.
{77} Svärd, Petter, B. Hudzia, S. Walsh, Johan Tordsson, and Erik Elmroth. “The Noble Art of Live VM Migration-Principles and performance of pre copy, post copy and hybrid migration of demanding workloads.” Technical Report, UMINF-12.11 (2014).
{78} Takano, Ryousei, Hidemoto Nakada, Takahiro Hirofuchi, Yoshio Tanaka, and Tomohiro Kudoh. “Cooperative VM migration for a virtualized HPC cluster with VMM-bypass I/O devices.” In 2012 IEEE 8th International Conference on E-Science, pp. 1-8. IEEE, 2012.
{79} Fernando, Denuni, “Fast And Robust Live Virtual Machine Migration”, Ph.D. Dissertation, Binghamton University (2019).
{80} M. Nelson, B. Lim, and G. Hutchins. Fast transparent migration for virtual machines. In Proceedings of the annual conference on USENIX Annual Technical Conference, pages 25-25, 2005.
{81} M. Nelson, B. H Lim, and G. Hutchins. Fast Transparent Migration for Virtual Machines. In USENIX Annual Technical Conference, 2005.
{82} John K. Ousterhout, Andrew R. Cherenson, Frederick Douglis, Michael N. Nelson, and Brent B. Welch. The sprite network operating system. Computer, 21(2):23-36, February 1988.
{83} Eunbyung Park, Bernhard Egger, and Jaejin Lee. Fast and space-efficient virtual machine checkpointing. In the 7th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, pages 75-86, 2011.
{84} Richard F. Rashid and George G. Robertson. Accent: A communication oriented network operating system kernel. In the Eighth ACM Symposium on Operating Systems Principles, pages 64-75, 1981.
{85} Redis. Key-value Cache and Store, redis.io.
{86} Michael Richmond and Michael Hitchens. A new process migration algorithm. SIGOPS Oper. Syst. Rev., 31(1):31-42, 1997.
{87} P. Riteau, C. Morin, and T. Priol. Shrinker: Improving live migration of virtual clusters over wans with distributed data deduplication and content-based addressing. In Proc. of EURO-PAR, September 2011.
{88} Adam Ruprecht, Danny Jones, Dmitry Shiraev, Greg Harmon, Maya Spivak, Michael Krebs, Miche Baker-Harvey, and Tyler Sanderson. Vm live migration at scale. In the 14th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, pages 45-56.
{89} Jonathan M. Smith. A survey of process migration mechanisms. SIGOPS Oper. Syst. Rev., 22(3):28-40, July 1988.
{90} Gang Sun, Dan Liao, Vishal Anand, Dongcheng Zhao, and Hongfang Yu. A new technique for efficient live migration of multiple virtual machines. Future Gener. Comput. Syst., 55:74-86, February 2016.
{91} P. Svard, B. Hudzia, J. Tordsson, and E. Elmroth. Evaluation of delta compression techniques for efficient live migration of large virtual machines. In VEE, 2011.
{92} T. Hirofuchi and I. Yamahata. Postcopy live migration for qemu/kvm. grivon.apgrid.org/quick-kvm-migration.
{93} Yoshiaki Tamura, Koji Sato, Seiji Kihara, and Satoshi Moriai. Kemari: Virtual machine synchronization for fault tolerance using domt, ntt cyber space labs. Technical report, Proc. USENIX Annual Technical Conference, 2008.
{94} VMWare Inc. VMware DRS: Dynamic Scheduling of System Resources, www.vmware.com/files/pdf/drs datasheet.pdf.
{95} VMWare Inc. VMWare Inc. Architecture of VMware ESXi, www.vmware.com/files/pdf/esxi-architecture.pdf.
{96} VMWare Inc. VMWare vMotion www.vmware.com/pdf/vmotion datasheet.pdf. 1971 VMWare Inc. VMWare vSphere 6 Fault Tolerance www.vmware.com/files/pdf/techpaper/vmware-vsphere6-ft-arch-perf.pdf.
{98} VMWare Inc. VMware vSphere vMotion Architecture, Performance and Best Practices in VMware vSphere 5, www.vmware.com/files/pdf/vmotion-perf-vsphere5.pdf.
{99} VMWare Inc. Protecting Mission-Critical Workloads with VMware Fault Tolerance. White Paper, Revision: 20090507 Item: WP-084-PRD-01-02, 2009.
{100} VMWare Knowledge Base. Virtual machine performance degrades while a vMotion is being performed, kb.vmware.com/kb/2007595.
{101} C. A. Waldspurger. Memory resource management in VMware ESX server. In Operating Systems Design and Implementation, December 2002.
{102} C. Wang, F. Mueller, C. Engelmann, and S. L. Scott. Proactive process-level live migration in hpc environments. In International Conference for High Performance Computing, Networking, Storage and Analysis, 2008.
{103} H. Wang, Y. Li, Y. Zhang, and D. Jin. Virtual machine migration planning in software-defined networks. IEEE Transactions on Cloud Computing, pages 1-1, 2017.
{104} Jian Wang, Kwame-Lante Wright, and Kartik Gopalan. Xenloop: A transparent high performance inter-vm network loopback. In Proceedings of the 17th International Symposium on High Performance Distributed Computing, HPDC '08, pages 109— 118, 2008.
{105} D. Williams, H. Jamjoom, Y.-H. Liu, and H. Weatherspoon. Overdriver: Handling memory overload in an oversubscribed cloud. In ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE), 2011.
{106} T. Wood, K. K. Ramakrishnan, P. Shenoy, and J. van der Merwe. Cloudnet: Dynamic pooling of cloud resources by live WAN migration of virtual machines. In ACM SIG-PLAN/SIGOPS International Conference on Virtual Execution Environments (VEE), 2011.
{107} T. Wood, P. Shenoy, A. Venkataramani, and M. Yousif. Sandpiper: Black-box and gray-box resource management for virtual machines. Intl. Journal of Computer and Telecommunications Networking, 53(17), 2009.
{108} Timothy Wood, Prashant Shenoy, Arun Venkataramani, and Mazin Yousif. Blackbox and gray-box strategies for virtual machine migration. In Proceedings of the 4th USENIX Conference on Networked Systems Design & Implementation, NSDI′07, pages 17-17, 2007.
{109} N. Bila, E. de Lara, K. Joshi, H. A. Lagar-Cavilla, M. Hiltunen, and M. Satyanarayanan. Jettison: Efficient Idle Desktop Consolidation with Partial VM Migration. In Eurosys, April 2012.
{110} I. Zhang, A. Garthwaite, Y. Baskakov, and K. C. Barr. Fast restore of checkpointed memory using working set estimation. In ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE), 2011.
{111} M. F. Zhani, Q. Zhang, G. Simona, and R. Boutaba. Vdc planner: Dynamic migration-aware virtual data center embedding for clouds. In 2013 IFIP/IEEE International Symposium on Integrated Network Management (IM 2013), pages 18-25, 2013.
{112} M. Zhao and R. J. Figueiredo. Experimental study of virtual machine migration in support of reservation of cluster resources. In Virtualization Technology in Distributed Computing (VTDC), 2007 Second International Workshop on, pages 1-8, 2007.
{113} Jeff Bonwick and Jonathan Adams. Magazines and vmem: Extending the slab allocator to many cpus and arbitrary resources. In USENIX Annual Technical Conference, pages 15-33, 2001.
{114} T. C. Bressoud and F. B. Schneider. Hypervisor-based fault tolerance. In Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles, SOSP '95, pages 1-11, New York, NY, USA, 1995. ACM.
{115} Martin Casado, Michael J. Freedman, Justin Pettit, Jianying Luo, Nick McKeown, and Scott Shenker. Ethane: Taking control of the enterprise. In Proceedings of the 2007 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications, SIGCOMM '07, pages 1-12, New York, NY, USA, 2007. ACM.
{116} J. Chiang, H. Li, and T. Chiueh. Working Set-based Physical Memory Ballooning. In The International Conference on Autonomic Computing (ICAC), June 2013.
{117} C. Clark, K. Fraser, S. Hand, J. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield. Live migration of virtual machines. In Symposium on Networked Systems Design & Implementation, pages 273-286, 2005.
{118} C. Clark, K. Fraser, S. Hand, J. G. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield. Live Migration of Virtual Machines. In USENIX Symposium on Networked Systems Design and Implementation (NSDI), pages 273-286, 2005.
{119} C. Clark, K. Fraser, S. Hand, J. G. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield. Live Migration of Virtual Machines. In USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2005.
{120} C. Clark, K. Fraser, S. Hand, J. G. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield. Live migration of virtual machines. In Network System Design and Implementation, 2005.
{121} Brian F. Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. Benchmarking cloud serving systems with ycsb. In Proceedings of the 1st ACM Symposium on Cloud Computing, SoCC '10, pages 143-154. ACM, 2010.
{122} Brendan Cully, Geoffrey Lefebvre, Dutch Meyer, Mike Feeley, Norm Hutchinson, and Andrew Warfield. Remus: High availability via asynchronous virtual machine replication. In Proceedings of the 5th USENIX Symposium on Networked Systems Design and Implementation, NSDI′08, pages 161-174, 2008.
{123} Tathagata Das, Pradeep Padala, Venkata N. Padmanabhan, Ramachandran Ramjee, and Kang G. Shin. Litegreen: Saving energy in networked desktops using virtualization. In Proceedings of the 2010 USENIX Conference on USENIX Annual Technical Conference, USENIXATC'10, pages 3-3, 2010.
{124} U. Deshpande, D. Chan, T. Y. Guh, J. Edouard, K. Gopalan, and N. Bila. Agile live migration of virtual machines. In 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS), 2016.
{125} U. Deshpande, B. Schlinker, E. Adler, and K. Gopalan. Gang migration of virtual machines using cluster-wide deduplication. In IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, May 2013.
{126} U. Deshpande, B. Wang, S. Hague, M. Hines, and K. Gopalan. Memx: Virtualization of cluster-wide memory. In Proc. of International Conference on Parallel Processing, September 2010.
{127} U. Deshpande, X. Wang, and K. Gopalan. Live gang migration of virtual machines. In Proc. of High Performance Distributed Computing, June 2010.
{128} U. Deshpande, X. Wang, and K. Gopalan. Live gang migration of virtual machines. In ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC), 2011.
{129} U. Deshpande, Y. You, D. Chan, N. Bila, and K. Gopalan. Fast server deprovisioning through scatter-gather live migration of virtual machines. In 2014 IEEE 7th International Conference on Cloud Computing, pages 376-383, June 2014.
{130} Umesh Deshpande, Danny Chan, Steven Chan, Kartik Gopalan, and Nilton Bila. Scatter-gather live migration of virtual machines. IEEE Transactions on Cloud Computing, PP(99):1-1, 2015.
{131} Umesh Deshpande, Danny Chan, Steven Chan, Kartik Gopalan, and Nilton Bila. Scatter-gather live migration of virtual machines. IEEE Transactions on Cloud Computing, 2015.
{132} Umesh Deshpande, Danny Chan, Ten-Young Guh, James Edouard, Kartik Gopalan, and Nilton Bila. Agile live migration of virtual machines. In IEEE International Parallel and Distributed Processing Symposium, Chicago, IL, USA, May 2016.
{133} YaoZu Dong, Wei Ye, YunHong Jiang, Ian Pratt, ShiQing Ma, Jian Li, and HaiBing Guan. Colo: Coarse-grained lock-stepping virtual machines for non-stop service. In the 4th Annual Symposium on Cloud Computing, pages 1-16, 2013.
{134} George W. Dunlap, Samuel T. King, Sukru Cinar, Murtaza A. Basrai, and Peter M. Chen. Revirt: Enabling intrusion analysis through virtual-machine logging and replay. SIGOPS Oper. Syst. Rev., 36:211-224, December 2002.
{135} George Washington Dunlap, III. Execution Replay for Intrusion Analysis. PhD thesis, Ann Arbor, MI, USA, 2006.
{136} C. Engelmann, G. R. Vallee, T. Naughton, and S. L. Scott. Proactive fault tolerance using preemptive migration. In 17th Euromicro International Conference on Parallel, Distributed and Network-based Processing, pages 252-257, 2009.
{137} D. Fernando, H. Bagdi, Y. Hu, P. Yang, K. Gopalan, C. Kamhoua, and K. Kwiat. Quick eviction of virtual machines through proactive snapshots. In 2016 IEEE International Conference on Cluster Computing (CLUSTER), pages 156-157, September 2016.
{138} Dinuni Fernando, Hardik Bagdi, Yaohui Hu, Ping Yang, Kartik Gopalan, Charles Kamhoua, and Kevin Kwiat. Quick eviction of virtual machines through proactive live snapshots. In Proceedings of the 9th International Conference on Utility and Cloud Computing, UCC '16, pages 99-107, 2016.
{139} Dinuni Fernando, Jonathan Terner, Ping Yang, and Kartik Gopalan. Live migration ate my vm: Recovering a virtual machine after failure of live migration. In Proceedings of IEEE International Conference on Computer Communications, INFOCOM, 2019.
{140} Google Cloud Platform for Data Center Professionals. cloud.google. com/docs/compare/data-centers/, 2017.
{141} Soudeh Ghorbani and Matthew Caesar. Walk the line: Consistent network updates with bandwidth guarantees. In Proceedings of the First Workshop on Hot Topics in Software Defined Networks, HotSDN '12, pages 67-72, New York, NY, USA, 2012. ACM.
{142} Edward L. Haletky. VMware ESX Server in the Enterprise: Planning and Securing Virtualization Servers. 1 edition.
{143} F. Hermenier, J. Lawall, and G. Muller. Btrplace: A flexible consolidation manager for highly available applications. IEEE Transactions on Dependable and Secure Computing, 10(5):273-286, 2013.
{144} Fabien Hermenier, Xavier Lorca, Jean-Marc Menaud, Gilles Muller, and Julia Lawall. Entropy: A consolidation manager for clusters. In Proceedings of the 2009 ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, VEE '09, pages 41-50, 2009.
{145} Fabien Hermenier, Xavier Lorca, Jean-Marc Menaud, Gilles Muller, and Julia Lawall. Entropy: A consolidation manager for clusters. In Proceedings of the 2009 ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, VEE '09, pages 41-50, 2009.
{146} M. R. Hines, U. Deshpande, and K. Gopalan. Post-copy live migration of virtual machines. SIGOPS Operating System Review, 43(3):14-26, 2009.
{147} Michael R. Hines, Umesh Deshpande, and Kartik Gopalan. Post-copy live migration of virtual machines. SIGOPS Operating System Review, 43(3):14-26, 2009.
{148} Michael R. Hines and Kartik Gopalan. Post-copy based live virtual machine migration using adaptive pre-paging and dynamic self-ballooning. In VEE, pages 51-60, 2009.
{149} T. Hirofuchi, H. Nakada, S. Itoh, and S. Sekiguchi. Reactive cloud: Consolidating virtual machines with postcopy live migration. IPSJ Transactions on Advanced Computing Systems, pages 86-98, March 2012.
{150} R. Jhawar, V. Piuri, and M. Santambrogio. Fault tolerance management in cloud computing: A system-level perspective. IEEE Systems Journal, 7(2):288-297, June 2013.
{151} H. Jin, L. Deng, S. Wu, X. Shi, and X. Pan. Live virtual machine migration with adaptive, memory compression. In Proc. of Cluster Computing and Workshops, August 2009.
{152} C. Jo, E. Gustafsson, J. Son, and B. Egger. Efficient live migration of virtual machines using shared storage. In ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE), 2013.
{153} A. Kangarlou, P. Eugster, and D. Xu. Vnsnap: Taking snapshots of virtual networked infrastructures in the cloud. IEEE Transactions on Services Computing, 5(4):484-496, 2012.
{154} S. M. Kerner. Why facebook does sdn. www.enterprisenetworkingplanet.com/datacenter/why-facebook-does-sdn. html, 2014.
{155} Samuel T. King, George W. Dunlap, and Peter M. Chen. Debugging operating systems with time-traveling virtual machines. In Annual Conference on USENIX Annual Technical Conference, pages 1-1, 2005.
{156} Avi Kivity, Yaniv Kamay, Dor Laor, Uri Lublin, and Anthony Liguori. Kvm: The linux virtual machine monitor. In Linux Symposium, June 2007.
{157} Thomas Knauth and Christof Fetzer. Vecycle: Recycling VM checkpoints for faster migrations. In Proceedings of the 16th Annual Middleware Conference, pages 210— 221, 2015.
{158} C. Kolivas. Kernbench. ck.kolivas.org/apps/kernbench/kernbench-0.50/.
{159} YongChul Kwon, Magdalena Balazinska, and Albert Greenberg. Fault-tolerant stream processing using a distributed, replicated file system. Proc. VLDB Endow., 1(1), August 2008.
{160} Linux Foundation. openvswitch.org/.
{161} H. Liu and B. He. Vmbuddies: Coordinating live migration of multi-tier applications in cloud environments. IEEE Transactions on Parallel and Distributed Systems, 26(4):1192-1205, 2015.
{162} H. Lu, C. Xu, C. Cheng, R. Kompella, and D. Xu. vhaul: Towards optimal scheduling of live multi-vm migration for multi-tier applications. In 2015 IEEE 8th International Conference on Cloud Computing, pages 453-460, June 2015.
{163} M. Lu and T. c. Chiueh. Fast memory state synchronization for virtualization-based fault tolerance. In 2009 IEEE/IFIP International Conference on Dependable Systems Networks, 2009.
{164} V. Mann, A. Vishnoi, A. Iyer, and P. Bhattacharya. Vmpatrol: Dynamic and automated qos for virtual machine migrations. In 2012 8th international conference on network and service management (cnsm) and 2012 workshop on systems virtualiztion management (svm), pages 174-178, October 2012.
{165} Vijay Mann, Akanksha Gupta, Partha Dutta, Anilkumar Vishnoi, Parantapa Bhattacharya, Rishabh Poddar, and Aakash Iyer. Remedy: Network-aware steady state vm management for data centers. In Proceedings of the 11th International IFIP TC 6 Conference on Networking, Volume Part I, IFIP′12, pages 190-204. Springer Verlag, 2012.
{166} Daniel Marques, Greg Bronevetsky, Rohit Fernandes, Keshav Pingali, and Paul Stodghil. Optimizing checkpoint sizes in the c3 system. In the 19th IEEE International Parallel and Distributed Processing Symposium, 2005.
{167} Ali Mashtizadeh, Emré Celebi, Tal Garfinkel, and Min Cai. The design and evolution of live storage migration in vmware esx. In Proceedings of the 2011 USENIX Conference on USENIX Annual Technical Conference, USENIXATC′11, pages 14-14, 2011.
{168} John D. McCalpin. Memory bandwidth and machine balance in current high performance computers. IEEE Computer Society Technical Committee on Computer Architecture (TCCA) Newsletter, pages 19-25, December 1995.
{169} Richard McDougall and Jim Maura. Solaris internals. In Rachael Borden, 2001.
{170} Nick McKeown, Tom Anderson, Hari Balakrishnan, Guru Parulkar, Larry Peterson, Jennifer Rexford, Scott Shenker, and Jonathan Turner. Openflow: Enabling innovation in campus networks. SIGCOMM Comput. Commun. Rev., 38(2):69-74, March 2008.
{171} Micro Checkpointing. wiki.qemu.org/features/microcheckpointing.
{172} Microsoft Corp. Hyper-v server 2008 r2. www.microsoft.com/hyper-v-server/en/us/overview.aspx.
{173} R. Miller. Failure rates in google data centers www.datacenterknowledge.com/archives/2008/05/30/failure-rates-in-google-data-centers, 2008.
{174} Dejan S. Milóičić, Fred Douglis, Yves Paindaveine, Richard Wheeler, and Songnian Zhou. Process migration. ACM Comput. Surv., 32(3):241-299, September 2000.
{175} David Mosberger and Tai Jin. Httperf— a tool for measuring web server performance. SIGMETRICS Perform. Eval. Rev., 26(3):31-37, December 1998.
{176} MOSIX. www.mosix.org.
{177} S. J. Mullender, G. van Rossum, A. S. Tananbaum, R. van Renesse, and H. van Staveren. Amoeba: a distributed operating system for the 1990s. Computer, 23(5):44-53, May 1990.
{178} Arun Babu Nagarajan, Frank Mueller, Christian Engelmann, and Stephen L. Scott. Proactive Fault Tolerance for HPC with Xen Virtualization. In Proceedings of the 21st Annual International Conference on Supercomputing, pages 23-32, 2007.
{179} Arun Babu Nagarajan, Frank Mueller, Christian Engelmann, and Stephen L. Scott. Proactive fault tolerance for hpc with xen virtualization. In the 21st Annual International Conference on Supercomputing, pages 23-32, 2007.
{180} Iperf—the TCP/UDP bandwidth measurement tool. dast.nlanr.net/Projects/Iperf/.
{181} S. Al-Haj and E. Al-Shaer. A formal approach for virtual machine migration planning. In Proceedings of the 9th International Conference on Network and Service Management (CNSM 2013), pages 51-58, 2013.
{182} Samer Al-Kiswany, Dinesh Subhraveti, Prasenjit Sarkar, and Matei Ripeanu. Vm-flock: Virtual machine co-migration for the cloud. In Proceedings of the 20th International Symposium on High Performance Distributed Computing, HPDC '11, pages 159-170, 2011.
{183} Amazon Corporation. Amazon Elastic Compute Cloud (EC2), aws.amazon.comkc2.
{184} Amazon Inc. AWS Server Migration Service. docs.aws.amazon. com/server-migration-service/latest/userguide/server-migration-ug.pdf.
{185} I. Banerjee, P. Moltmann, K. Tati, and R. Venkatasubramanian. VMware ESX Memory Resource Management: Swap. In VMWare Technical Journal, 2014.
{186} P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. Xen and the art of virtualization. SIGOPS Operating Systems Review, 37(5):164-177, 2003.
{187} Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, and Andrew Warfield. Xen and the art of virtualization. In Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles, SOSP '03, 2003.
{188} M. F. Bari, M. F. Zhani, Q. Zhang, R. Ahmed, and R. Boutaba. Cqncr: Optimal vm migration planning in cloud data centers. In 2014 IFIP Networking Conference, pages 1-9, June 2014.
{189} F. Bellard. Qemu, a fast and portable dynamic translator. In In ATEC 05: Proceedings of the annual conference on USENIX Annual Technical Conference, 2005.
{190} SPEC CPU 2017 benchmark suite. www.spec.org/cpu2017/.
{191} U.S. patent and patent applications: U.S. Pat. Nos. 6,671,707; 7,484,208; 7,673,113; 7,680,919; 7,849,286; 7,900,005; 8,004,998; 8,010,495; 8,078,824; 8,196,138; 8,219,653; 8,260,904; 8,302,094; 8,312,453; 8,332,848; 8,364,638; 8,386,428; 8,386,731; 8,407,518; 8,418,176; 8,429,630; 8,453,144; 8,458,284; 8,473,557; 8,489,699; 8,490,088; 8,504,686; 8,505,003; 8,521,912; 8,533,305; 8,533,713; 8,554,900; 8,560,791; 8,583,709; 8,627,310; 8,627,456; 8,646,028; 8,650,565; 8,661,436; 8,671,238; 8,689,213; 8,694,644; 8,694,828; 8,700,867; 8,761,187; 8,775,590; 8,806,480; 8,832,688; 8,856,255; 8,856,339; 8,862,744; 8,869,144; 8,903,705; 8,904,384; 8,924,571; 8,935,506; 8,943,296; 8,973,098; 8,977,736; 8,984,507; 8,990,531; 8,990,800; 8,990,808; 9,015,709; 9,027,024; 9,032,398; 9,043,493; 9,043,790; 9,047,110; 9,052,949; 9,053,068; 9,055,119; 9,058,199; 9,063,866; 9,071,550; 9,081,599; 9,086,904; 9,086,981; 9,087,076; 9,092,318; 9,110,700; 9,110,704; 9,141,578; 9,152,448; 9,164,795; 9,172,587; 9,172,588; 9,183,060; 9,183,157; 9,197,489; 9,201,676; 9,201,704; 9,229,516; 9,244,710; 9,244,731; 9,250,959; 9,262,199; 9,262,211; 9,262,227; 9,268,586; 9,280,378; 9,286,100; 9,292,275; 9,304,878; 9,317,326; 9,323,921; 9,342,346; 9,342,360; 9,348,526; 9,348,655; 9,354,905; 9,354,921; 9,361,145; 9,361,218; 9,372,726; 9,372,727; 9,384,024; 9,384,062; 9,389,933; 9,396,004; 9,405,642; 9,411,620; 9,413,538; 9,417,754; 9,425,965; 9,436,386; 9,436,751; 9,442,753; 9,448,831; 9,454,400; 9,454,489; 9,459,900; 9,459,902; 9,473,598; 9,495,183; 9,501,305; 9,507,615; 9,507,672; 9,509,868; 9,519,512; 9,519,548; 9,529,616; 9,529,618; 9,529,620; 9,531,799; 9,542,108; 9,552,215; 9,552,217; 9,558,005; 9,558,028; 9,575,789; 9,578,088; 9,588,796; 9,588,803; 9,588,836; 9,588,844; 9,594,583; 9,594,598; 9,619,297; 9,626,212; 9,639,388; 9,648,134; 9,652,296; 9,671,966; 9,672,120; 9,703,655; 9,720,722; 9,720,728; 9,740,627; 9,760,443; 9,766,930; 9,767,039; 9,778,997; 9,792,138; 9,798,560; 9,798,567; 9,804,866; 9,811,376; 9,823,842; 9,830,090; 9,830,176; 9,836,327; 9,842,032; 9,852,137; 9,852,198; 9,854,036; 9,870,246; 9,880,868; 9,880,870; 9,880,872; 9,898,320; 9,898,430; 9,904,570; 9,928,091; 9,928,107; 9,928,260; 9,934,056; 9,934,157; 9,934,407; 9,936,019; 9,965,311; 9,971,616; 9,996,396; 10,013,276; 10,031,607; 10,033,595; 10,037,334; 10,044,558; 10,049,064; 10,061,622; 10,073,713; 10,078,529; 10,083,053; 10,089,011; 10,114,662; 10,120,480; 10,120,594; 10,127,049; 10,129,331; 10,135,948; 10,140,112; 10,146,353; 10,156,921; 10,156,986; 10,157,077; 10,162,448; 10,169,068; 10,178,073; 10,187,466; 10,191,756; 10,198,299; 10,198,377; 10,203,794; 10,209,806; 10,209,807; 10,209,808; 10,209,809; 10,210,567; 10,222,891; 10,222,892; 10,222,893; 10,222,894; 10,222,895; 10,223,276; 10,235,235; 10,241,814; 10,268,595; 10,275,086; 10,275,087; 10,289,435; 10,289,436; 10,289,684; 10,310,977; 10,318,329; 10,324,851; 10,338,736; 10,345,961; 10,365,758; 10,386,960; 10,394,656; 10,394,668; 10,423,437; 10,430,221; 10,430,249; 10,430,991; 10,437,627; 10,452,559; 10,466,999; 10,467,106; 10,474,369; 10,474,489; 10,481,940; 10,481,942; 10,503,551; 10,503,659; 10,509,662; 10,509,687; 10,514,938; 10,521,047; 10,530,822; 10,534,474; 10,540,039; 10,541,928; 10,545,777; 10,551,966; 10,552,080; 10,552,230; 10,552,267; 10,572,271; 10,579,409; 10,579,412; 10,592,039; 10,592,273; 10,592,428; 10,606,396; 10,628,200; 10,635,474; 10,635,477; 10,642,413; 10,642,539; 10,649,571; 10,649,578; 10,649,579; 10,649,580; 10,649,581; 10,656,752; 10,656,753; 10,656,754; 10,656,755; 10,656,756; 10,656,757; 10,656,758; 10,656,759; 10,664,097; 10,666,516; 10,671,212; 10,671,213; 10,684,874; 10,698,711; 10,713,123; 10,725,581; 10,725,823; 10,740,016; 10,782,819; 10,788,931; 10,796,035; 10,810,037; 10,824,716; 10,826,930; 10,838,542; 10,853,119; 10,936,114; 10,936,330; 10,956,197; 10,963,367; 20020078057; 20070294676; 20080163239; 20080172492; 20080222375; 20080263258; 20090007106; 20090113109; 20090125904; 20090157882; 20090271786; 20090276771; 20090290501; 20100125845; 20100169253; 20100169537; 20100180274; 20100235825; 20100318608; 20110066819; 20110113136; 20110119670; 20110134931; 20110141124; 20110145418; 20110145458; 20110145819; 20110145820; 20110145821; 20110145886; 20110145916; 20110153697; 20110231455; 20110238820; 20110271062; 20110271270; 20110296411; 20120011504; 20120011508; 20120017031; 20120042034; 20120054367; 20120066375; 20120066677; 20120102084; 20120137098; 20120198440; 20120233282; 20120233331; 20120233611; 20120254862; 20120278525; 20120278807; 20120284234; 20120304176; 20120310888; 20120324442; 20130024598; 20130041977; 20130060947; 20130061225; 20130086272; 20130124675; 20130139154; 20130139155; 20130179655; 20130205106; 20130205113; 20130246355; 20130268643; 20130268799; 20130275375; 20130290468; 20130298126; 20130304903; 20130305242; 20130305246; 20130318515; 20130326173; 20130326174; 20130326175; 20130346613; 20140019974; 20140032767; 20140040343; 20140068703; 20140081984; 20140101657; 20140101754; 20140108854; 20140109180; 20140115162; 20140115164; 20140139869; 20140149635; 20140149984; 20140173600; 20140173628; 20140189041; 20140189690; 20140196037; 20140201357; 20140215172; 20140245294; 20140297780; 20140298338; 20140325515; 20140359607; 20140372717; 20150052282; 20150063166; 20150066859; 20150149999; 20150160962; 20150160963; 20150160964; 20150163270; 20150186220; 20150193248; 20150205673; 20150234684; 20150235308; 20150242228; 20150277789; 20150277962; 20150301931; 20150309839; 20150324236; 20150324297; 20150331715; 20150339148; 20150370596; 20150372862; 20150373096; 20150378623; 20150378766; 20150378767; 20150378771; 20150378783; 20150378785; 20150378831; 20150378847; 20150378931; 20150381589; 20160026489; 20160028761; 20160034702; 20160098302; 20160103728; 20160117122; 20160132443; 20160139962; 20160147556; 20160196449; 20160203015; 20160224371; 20160232023; 20160239328; 20160239428; 20160246639; 20160266940; 20160274896; 20160274934; 20160283421; 20160306862; 20160381125; 20170005813; 20170024246; 20170075706; 20170090964; 20170090968; 20170132062; 20170132430; 20170139755; 20170147501; 20170199755; 20170201574; 20170220374; 20170270012; 20170272515; 20170288971; 20170322815; 20170322828; 20170357527; 20170371696; 20180004679; 20180018135; 20180024854; 20180041477; 20180052701; 20180060107; 20180074842; 20180074908; 20180074909; 20180088975; 20180101452; 20180113610; 20180121247; 20180129523; 20180136838; 20180136963; 20180173547; 20180183869; 20180203721; 20180225221; 20180232254; 20180276025; 20180329737; 20180349421; 20180357098; 20180373557; 20190005576; 20190020732; 20190034112; 20190050235; 20190087220; 20190087244; 20190095232; 20190114197; 20190129740; 20190129814; 20190146853; 20190158588; 20190179659; 20190205220; 20190207764; 20190220364; 20190222522; 20190228557; 20190258503; 20190265904; 20190278624; 20190278632; 20190319896; 20190334909; 20190354378; 20190370049; 20190384495; 20190384673; 20190391835; 20190391843; 20200019429; 20200019512; 20200026547; 20200034176; 20200034182; 20200034211; 20200073674; 20200110629; 20200150997; 20200174817; 20200183729; 20200293697; 20200319897; 20200319904; 20200334067; 20200387392; 20200389469; 20200403940; 20210049079; 20210055946; and 20210058425.

Claims

1. A virtual machine system, comprising: a communication port configured to receive an initial execution state of the virtual machine to transmit an incremental checkpoint;a buffer, configured to store a request for a persistent change of an external environment; andan automated processor configured to: execute a virtual machine initiated based on the received initial execution state,generate the incremental checkpoint reflecting the buffered request for the persistent change of an external environment,request a memory page through the communication port during a migration of the virtual machine, in response to a page fault during execution of the virtual machine, andtransmit the incremental checkpoint reflecting the buffered request for the persistent change of an external environment through the communication port.
2. The virtual machine system according to claim 1, wherein the incremental checkpoint comprises memory pages modified by the virtual machine since a prior incremental checkpoint, a central processing unit state of the virtual machine, and an input/output state of the virtual machine.
3. The virtual machine system according to claim 1, wherein the incremental checkpoint is a consistent checkpoint.
4. The virtual machine system according to claim 1, wherein the communicated incremental checkpoint comprises a virtual CPU state, an input/output state, and a memory state of the virtual machine prior to persistently changing a state outside of the virtual machine since a prior communicated incremental checkpoint.
5. The virtual machine system according to claim 1, wherein the incremental checkpoint consists essentially of memory pages modified since a prior incremental checkpoint, a CPU state of the virtual machine, and an input/output state of the virtual machine.
6. The virtual machine system according to claim 1, wherein the automated processor is configured to communicate the incremental checkpoint in response to uncommitted input/output requests of the virtual machine.
7. The virtual machine system according to claim 1, wherein the automated processor is further configured to communicate acknowledgement of memory page receipt.
8. The virtual machine system according to claim 1, wherein the virtual machine executes under a hypervisor, wherein the incremental checkpoint is generated by the hypervisor.
9. The virtual machine system according to claim 1, wherein the virtual machine executes under a hypervisor, wherein the incremental checkpoint is generated by the hypervisor and wherein a portion of the incremental checkpoint is defined without pausing the virtual machine.
10. The virtual machine system according to claim 1, further comprising a hypervisor which executes the virtual machine as a first thread or process, wherein the incremental checkpoint is generated within the hypervisor in a second thread or process distinct from the first thread or process.
11. The virtual machine system according to claim 1, wherein the automated processor is configured to generate the incremental checkpoint in two phases, a first phase comprising determining memory pages modified by the virtual machine during which the virtual machine is operational; anda second phase comprising execution state of a CPU of the virtual machine, an input/output state of the virtual machine, and modified memory pages, during which the virtual machine is paused.
12. The virtual machine system according to claim 1, wherein the automated processor is further configured to: determine a failure of successful migration if execution of the tasks by the virtual machine at the destination fails; andupon failure of successful migration, recover the virtual machine at the source.
13. The virtual machine system according to claim 12, wherein the automated processor is further configured to recover the virtual machine at the source in a state subsequent to the initiation of execution of the tasks by the virtual machine at the destination.
14. The virtual machine system according to claim 1, wherein the automated processor is further configured to: proactively transfer memory pages from the source to the destination independent of occurrence of a page fault;cease execution of tasks by the virtual machine at the source before initiating an execution of the tasks by the virtual machine at the destination; andcommunicate the incremental checkpoint from the virtual machine executing at the destination, while the migration of the virtual machine to the destination is in progress.
15. The virtual machine system according to claim 1, further comprising providing a checkpoint manager, configured to: store a checkpoint of the source comprising the execution state of the virtual machine at the source; andstore the incremental checkpoint from the virtual machine executing at the destination.
16. The virtual machine system according to claim 1, wherein the transmitted incremental checkpoint comprises a virtual CPU state, an input/output state, and a memory state of the virtual machine executing at the destination prior to a persistent change of a state outside of the virtual machine since a prior transmitted incremental checkpoint.
17. The virtual machine system according to claim 1, wherein the automated processor is further configured to recover the virtual machine at the source from a last consistent copy of each memory page from a series of incremental checkpoints from the destination;further comprising a memory configured to buffer outgoing packet transmissions to external resources between successive incremental checkpoints, and to immediately deliver incoming network from external resources between successive incremental checkpoints.
18. A virtual machine system, comprising: a memory configured to store an execution state of a virtual machine;a communication;a buffer, configured to store a request for a persistent change of an external environment communicated through the communication port; andan automated processor configured to: execute the virtual machine,request a memory page in response to a page fault due to incomplete migration, during execution of the virtual machine, andgenerate and communicate an incremental change in the execution state reflecting the request for the persistent change of an external environment through the communication port.
19. A method for migrating a virtual machine, comprising: receiving an initial execution state of the virtual machine through a communication port;transmitting an incremental checkpoint through the communication port;communicating a request for a persistent change of an external environment through the communication port;buffering the request for the persistent change of an external environment;executing on an automated processor the virtual machine initiated based on the initial execution state,generate an incremental checkpoint of the virtual machine reflecting the request for the persistent change of an external environment,requesting a memory page through the communication port during a migration of the virtual machine, in response to a page fault during execution of the virtual machine, andtransmitting the incremental checkpoint through the communication port.
20. The method system according to claim 19, further comprising communicating the incremental checkpoint in response to an uncommitted input/output request of the virtual machine.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a Continuation of U.S. patent application Ser. No. 17/242,508, filed Apr. 28, 2021, now U.S. Pat. No. 11,409,619, issued Aug. 9, 2022, which is a Non-provisional of, and claims benefit of priority from, U.S. Provisional Patent Application No. 63/017,320, filed Apr. 29, 2020, the entirety of which is expressly incorporated herein by reference.

STATEMENT OF GOVERNMENT INTEREST

This invention was made with government support under CNS 1320689 and CNS 1527338 awarded by the National Science Foundation. The government has certain rights in the invention.

US Referenced Citations (467)

Number	Name	Date	Kind
6671707	Hudson et al.	Dec 2003	B1
7484208	Nelson	Jan 2009	B1
7673113	Sugumar et al.	Mar 2010	B2
7680919	Nelson	Mar 2010	B2
7849286	Sugumar et al.	Dec 2010	B2
7900005	Kotsovinos et al.	Mar 2011	B2
8004998	Levy et al.	Aug 2011	B2
8010495	Kuznetzov et al.	Aug 2011	B1
8078824	Sugumar et al.	Dec 2011	B2
8196138	Allwell et al.	Jun 2012	B2
8219653	Keagy et al.	Jul 2012	B1
8260904	Nelson	Sep 2012	B2
8302094	Rogel et al.	Oct 2012	B2
8312453	Shah	Nov 2012	B2
8332848	Azulay et al.	Dec 2012	B2
8364638	Nickolov et al.	Jan 2013	B2
8386428	Kuznetzov et al.	Feb 2013	B2
8386731	Mashtizadeh et al.	Feb 2013	B2
8407518	Nelson et al.	Mar 2013	B2
8418176	Keagy et al.	Apr 2013	B1
8429630	Nickolov et al.	Apr 2013	B2
8453144	Keagy et al.	May 2013	B1
8458284	Huang et al.	Jun 2013	B2
8473557	Ramakrishnan et al.	Jun 2013	B2
8489699	Goggin et al.	Jul 2013	B2
8490088	Tang	Jul 2013	B2
8504686	Phaal	Aug 2013	B2
8505003	Bowen	Aug 2013	B2
8521912	Aloni et al.	Aug 2013	B2
8533305	Keagy et al.	Sep 2013	B1
8533713	Dong	Sep 2013	B2
8554900	Nelson	Oct 2013	B2
8560791	Mashitizadeh et al.	Oct 2013	B2
8583709	Joukov et al.	Nov 2013	B2
8627310	Ashok et al.	Jan 2014	B2
8627456	McKenzie et al.	Jan 2014	B2
8646028	McKenzie et al.	Feb 2014	B2
8650565	Narasimhan	Feb 2014	B2
8661436	McKenzie et al.	Feb 2014	B2
8671238	Mashtizadeh et al.	Mar 2014	B2
8689213	Philipson et al.	Apr 2014	B2
8694644	Chen et al.	Apr 2014	B2
8694828	Nelson et al.	Apr 2014	B2
8700867	Chen	Apr 2014	B2
8761187	Barde	Jun 2014	B2
8775590	Devarakonda et al.	Jul 2014	B2
8806480	Araujo, Jr. et al.	Aug 2014	B2
8832688	Tang et al.	Sep 2014	B2
8856255	Ramakrishnan et al.	Oct 2014	B2
8856339	Mestery et al.	Oct 2014	B2
8862744	Garg et al.	Oct 2014	B2
8869144	Pratt et al.	Oct 2014	B2
8903705	Douceur et al.	Dec 2014	B2
8904384	Low et al.	Dec 2014	B2
8924571	Pratt et al.	Dec 2014	B2
8935506	Gopalan	Jan 2015	B1
8943296	Serebrin et al.	Jan 2015	B2
8973098	Raghunath et al.	Mar 2015	B2
8977736	Nelson	Mar 2015	B2
8984507	Knowles et al.	Mar 2015	B2
8990531	Ahmad	Mar 2015	B2
8990800	Cherian et al.	Mar 2015	B2
8990808	Nakagawa	Mar 2015	B2
9015709	Voccio	Apr 2015	B2
9027024	Mick et al.	May 2015	B2
9032398	Ahmad et al.	May 2015	B2
9043493	Bacher et al.	May 2015	B2
9043790	Castillo et al.	May 2015	B2
9047110	Takeda	Jun 2015	B2
9052949	Tarasuk-Levin et al.	Jun 2015	B2
9053068	Tsirkin et al.	Jun 2015	B2
9055119	Zhou et al.	Jun 2015	B2
9058199	Tsirkin	Jun 2015	B2
9063866	Tati et al.	Jun 2015	B1
9071550	Tang et al.	Jun 2015	B2
9081599	Tsirkin et al.	Jul 2015	B2
9086904	Goggin et al.	Jul 2015	B2
9086981	Tati et al.	Jul 2015	B1
9087076	Nickolov et al.	Jul 2015	B2
9092318	Ahmad et al.	Jul 2015	B2
9110700	McKenzie et al.	Aug 2015	B2
9110704	Tsirkin	Aug 2015	B2
9141578	Mashtizadeh et al.	Sep 2015	B2
9152448	Wang et al.	Oct 2015	B2
9164795	Vincent	Oct 2015	B1
9172587	Kalyanaraman et al.	Oct 2015	B2
9172588	Kalyanaraman et al.	Oct 2015	B2
9183060	Wakamiya	Nov 2015	B2
9183157	Qiu et al.	Nov 2015	B2
9197489	Vincent	Nov 2015	B1
9201676	Tsirkin et al.	Dec 2015	B2
9201704	Chang et al.	Dec 2015	B2
9229516	Hiltunen et al.	Jan 2016	B2
9244710	Bacher et al.	Jan 2016	B2
9244731	Siu et al.	Jan 2016	B2
9250959	Nakagawa	Feb 2016	B2
9262199	Castillo et al.	Feb 2016	B2
9262211	Mo et al.	Feb 2016	B2
9262227	Doi	Feb 2016	B2
9268586	Voccio et al.	Feb 2016	B2
9280378	Shah	Mar 2016	B2
9286100	Salmela et al.	Mar 2016	B2
9292275	Bowen	Mar 2016	B2
9304878	Antony et al.	Apr 2016	B2
9317326	Ramanathan et al.	Apr 2016	B2
9323921	Hunt et al.	Apr 2016	B2
9342346	Maislos et al.	May 2016	B2
9342360	Bacher et al.	May 2016	B2
9348526	White et al.	May 2016	B2
9348655	Tsirkin et al.	May 2016	B1
9354905	Nakagawa	May 2016	B2
9354921	Nelson	May 2016	B2
9361145	Wilson et al.	Jun 2016	B1
9361218	Ahmad et al.	Jun 2016	B2
9372726	Gopalan	Jun 2016	B2
9372727	Knowles et al.	Jun 2016	B2
9384024	Ramakrishnan Nair	Jul 2016	B2
9384062	Tan	Jul 2016	B2
9389933	Baumann et al.	Jul 2016	B2
9396004	Bester et al.	Jul 2016	B1
9405642	Tsirkin	Aug 2016	B2
9411620	Wang et al.	Aug 2016	B2
9413538	Baumann et al.	Aug 2016	B2
9417754	Smith	Aug 2016	B2
9425965	Baumann et al.	Aug 2016	B2
9436386	White	Sep 2016	B2
9436751	Serebrin	Sep 2016	B1
9442753	Zheng	Sep 2016	B2
9448831	Dong et al.	Sep 2016	B2
9454400	Tsirkin et al.	Sep 2016	B2
9454489	Tati et al.	Sep 2016	B2
9459900	Tsirkin	Oct 2016	B2
9459902	Noel et al.	Oct 2016	B2
9473598	Kashyap	Oct 2016	B2
9495183	Bond et al.	Nov 2016	B2
9501305	Groth	Nov 2016	B2
9507615	McKenzie et al.	Nov 2016	B2
9507672	Ye	Nov 2016	B2
9509868	Honma	Nov 2016	B2
9519512	Kami	Dec 2016	B2
9519548	Serebrin et al.	Dec 2016	B2
9529616	Borntraeger et al.	Dec 2016	B2
9529618	Borntraeger et al.	Dec 2016	B2
9529620	Dow et al.	Dec 2016	B1
9531799	Chandrasekaran et al.	Dec 2016	B2
9542108	White	Jan 2017	B2
9552215	Voccio et al.	Jan 2017	B2
9552217	Tarasuk-Levin et al.	Jan 2017	B2
9558005	Sathyanarayana	Jan 2017	B2
9558028	Joshi et al.	Jan 2017	B2
9575789	Rangari et al.	Feb 2017	B1
9578088	Nickolov et al.	Feb 2017	B2
9588796	Tarasuk-Levin et al.	Mar 2017	B2
9588803	Howell et al.	Mar 2017	B2
9588836	Shetty et al.	Mar 2017	B2
9588844	Bissett et al.	Mar 2017	B2
9594583	Bonzini	Mar 2017	B2
9594598	Brouwer et al.	Mar 2017	B1
9619297	Jain et al.	Apr 2017	B2
9626212	Beveridge et al.	Apr 2017	B2
9639388	Tsirkin et al.	May 2017	B2
9648134	Frank	May 2017	B2
9652296	Tsirkin et al.	May 2017	B1
9671966	Miwa et al.	Jun 2017	B2
9672120	Tarasuk-Levin et al.	Jun 2017	B2
9703655	Hardt et al.	Jul 2017	B1
9720722	Tsirkin	Aug 2017	B2
9720728	Yu et al.	Aug 2017	B2
9740627	White et al.	Aug 2017	B2
9760443	Tarasuk-Levin et al.	Sep 2017	B2
9766930	Tarasuk-Levin et al.	Sep 2017	B2
9767039	Serebrin et al.	Sep 2017	B2
9778997	Tsao et al.	Oct 2017	B2
9792138	Kaplan et al.	Oct 2017	B2
9798560	Keagy et al.	Oct 2017	B1
9798567	Hu et al.	Oct 2017	B2
9804866	Halls et al.	Oct 2017	B2
9811376	Brouwer et al.	Nov 2017	B2
9823842	Gopalan et al.	Nov 2017	B2
9830090	Hardt et al.	Nov 2017	B1
9830176	Andrus et al.	Nov 2017	B2
9836327	Brouwer et al.	Dec 2017	B1
9842032	Blake	Dec 2017	B2
9852137	Mann et al.	Dec 2017	B2
9852198	Kuznetzov et al.	Dec 2017	B1
9854036	Yang et al.	Dec 2017	B2
9870246	Koladi et al.	Jan 2018	B2
9880868	Shah	Jan 2018	B2
9880870	Danilov et al.	Jan 2018	B1
9880872	Srinivas	Jan 2018	B2
9898320	Tarasuk-Levin et al.	Feb 2018	B2
9898430	Davda et al.	Feb 2018	B2
9904570	Ramanathan et al.	Feb 2018	B2
9928091	Lotlikar et al.	Mar 2018	B2
9928107	Vincent	Mar 2018	B1
9928260	Sitsky et al.	Mar 2018	B2
9934056	Noel et al.	Apr 2018	B2
9934157	Tsirkin et al.	Apr 2018	B2
9934407	Sikka	Apr 2018	B2
9936019	Sanderson	Apr 2018	B2
9965311	Tsirkin et al.	May 2018	B2
9971616	Kaul	May 2018	B2
9996396	Knowles et al.	Jun 2018	B2
11409619	Gopalan	Aug 2022	B2
20020078057	Wang et al.	Jun 2002	A1
20070294676	Mellor et al.	Dec 2007	A1
20080163239	Sugumar et al.	Jul 2008	A1
20080172492	Raghunath et al.	Jul 2008	A1
20080222375	Kotsovinos et al.	Sep 2008	A1
20080263258	Allwell et al.	Oct 2008	A1
20090007106	Araujo, Jr. et al.	Jan 2009	A1
20090113109	Nelson et al.	Apr 2009	A1
20090125904	Nelson	May 2009	A1
20090157882	Kashyap	Jun 2009	A1
20090271786	Groth	Oct 2009	A1
20090276771	Nickolov et al.	Nov 2009	A1
20090290501	Levy et al.	Nov 2009	A1
20100125845	Sugumar et al.	May 2010	A1
20100169253	Tan	Jul 2010	A1
20100169537	Nelson	Jul 2010	A1
20100180274	Cherian et al.	Jul 2010	A1
20100235825	Azulay et al.	Sep 2010	A1
20100318608	Huang et al.	Dec 2010	A1
20110066819	Mashtizadeh et al.	Mar 2011	A1
20110099318	Hudzia	Apr 2011	A1
20110113136	Phaal	May 2011	A1
20110119670	Sugumar et al.	May 2011	A1
20110134931	Merwe et al.	Jun 2011	A1
20110141124	Halls et al.	Jun 2011	A1
20110145418	Pratt et al.	Jun 2011	A1
20110145458	Narasimhan	Jun 2011	A1
20110145819	McKenzie et al.	Jun 2011	A1
20110145820	Pratt et al.	Jun 2011	A1
20110145821	Philipson et al.	Jun 2011	A1
20110145886	McKenzie et al.	Jun 2011	A1
20110145916	McKenzie et al.	Jun 2011	A1
20110153697	Nickolov et al.	Jun 2011	A1
20110231455	Joukov et al.	Sep 2011	A1
20110238820	Matsuoka	Sep 2011	A1
20110271062	Chen	Nov 2011	A1
20110271270	Bowen	Nov 2011	A1
20110296411	Tang et al.	Dec 2011	A1
20120011504	Ahmad et al.	Jan 2012	A1
20120011508	Ahmad	Jan 2012	A1
20120017031	Mashtizadeh et al.	Jan 2012	A1
20120042034	Goggin et al.	Feb 2012	A1
20120054367	Ramakrishnan et al.	Mar 2012	A1
20120066375	Phaal	Mar 2012	A1
20120066677	Tang	Mar 2012	A1
20120102084	Hiltunen et al.	Apr 2012	A1
20120137098	Wang et al.	May 2012	A1
20120198440	Shah	Aug 2012	A1
20120233282	Voccio et al.	Sep 2012	A1
20120233331	Voccio et al.	Sep 2012	A1
20120233611	Voccio	Sep 2012	A1
20120254862	Dong	Oct 2012	A1
20120278525	Serebrin et al.	Nov 2012	A1
20120278807	Nakagawa	Nov 2012	A1
20120284234	Mashtizadeh et al.	Nov 2012	A1
20120304176	Takeda	Nov 2012	A1
20120310888	Kuznetzov et al.	Dec 2012	A1
20120324442	Barde	Dec 2012	A1
20130024598	Serebrin et al.	Jan 2013	A1
20130041977	Wakamiya	Feb 2013	A1
20130060947	Nelson	Mar 2013	A1
20130061225	Nakagawa	Mar 2013	A1
20130086272	Chen et al.	Apr 2013	A1
20130124675	Mo et al.	May 2013	A1
20130139154	Shah	May 2013	A1
20130139155	Shah	May 2013	A1
20130179655	Mashitizadeh et al.	Jul 2013	A1
20130205106	Tati et al.	Aug 2013	A1
20130205113	Ahmad et al.	Aug 2013	A1
20130246355	Nelson et al.	Sep 2013	A1
20130268643	Chang et al.	Oct 2013	A1
20130268799	Mestery et al.	Oct 2013	A1
20130275375	Nickolov et al.	Oct 2013	A1
20130290468	Ramakrishnan et al.	Oct 2013	A1
20130298126	Nakagawa	Nov 2013	A1
20130304903	Mick et al.	Nov 2013	A1
20130305242	Wang et al.	Nov 2013	A1
20130305246	Goggin et al.	Nov 2013	A1
20130318515	Bowen	Nov 2013	A1
20130326173	Tsirkin	Dec 2013	A1
20130326174	Tsirkin	Dec 2013	A1
20130326175	Tsirkin	Dec 2013	A1
20130346613	Tarasuk-Levin et al.	Dec 2013	A1
20140019974	Siu et al.	Jan 2014	A1
20140032767	Nelson	Jan 2014	A1
20140040343	Nickolov et al.	Feb 2014	A1
20140068703	Balus et al.	Mar 2014	A1
20140081984	Sitsky et al.	Mar 2014	A1
20140101657	Bacher et al.	Apr 2014	A1
20140101754	McKenzie et al.	Apr 2014	A1
20140108854	Antony et al.	Apr 2014	A1
20140109180	McKenzie et al.	Apr 2014	A1
20140115162	Kalyanaraman et al.	Apr 2014	A1
20140115164	Kalyanaraman et al.	Apr 2014	A1
20140139869	Honma	May 2014	A1
20140149635	Bacher et al.	May 2014	A1
20140149984	Takahashi et al.	May 2014	A1
20140173600	Ramakrishnan Nair	Jun 2014	A1
20140173628	Ramakrishnan Nair	Jun 2014	A1
20140189041	Mashtizadeh et al.	Jul 2014	A1
20140189690	Ramakrishnan Nair	Jul 2014	A1
20140196037	Gopalan	Jul 2014	A1
20140201357	Tang et al.	Jul 2014	A1
20140215172	Tsirkin	Jul 2014	A1
20140245294	Kaul	Aug 2014	A1
20140297780	Zhou et al.	Oct 2014	A1
20140298338	Doi	Oct 2014	A1
20140325515	Salmela et al.	Oct 2014	A1
20140359607	Tsirkin et al.	Dec 2014	A1
20140372717	Ciu et al.	Dec 2014	A1
20150052282	Dong	Feb 2015	A1
20150063166	Sif et al.	Mar 2015	A1
20150066859	Blake	Mar 2015	A1
20150149999	Ramanathan et al.	May 2015	A1
20150160962	Borntraeger et al.	Jun 2015	A1
20150160963	Borntraeger et al.	Jun 2015	A1
20150160964	Nelson	Jun 2015	A1
20150163270	Lyons	Jun 2015	A1
20150186220	Serebrin et al.	Jul 2015	A1
20150193248	Noel et al.	Jul 2015	A1
20150205673	Bissett et al.	Jul 2015	A1
20150234684	Bacher et al.	Aug 2015	A1
20150235308	Mick et al.	Aug 2015	A1
20150242228	Voccio	Aug 2015	A1
20150277789	White et al.	Oct 2015	A1
20150277962	Tsirkin et al.	Oct 2015	A1
20150301931	Ahmad et al.	Oct 2015	A1
20150309839	Lu	Oct 2015	A1
20150324236	Gopalan et al.	Nov 2015	A1
20150324297	Tati et al.	Nov 2015	A1
20150331715	Sathyanarayana	Nov 2015	A1
20150339148	Zheng	Nov 2015	A1
20150370596	Fahs et al.	Dec 2015	A1
20150372862	Namihira	Dec 2015	A1
20150373096	Chandrasekaran et al.	Dec 2015	A1
20150378623	White	Dec 2015	A1
20150378766	Beveridge et al.	Dec 2015	A1
20150378767	Tarasuk-Levin et al.	Dec 2015	A1
20150378771	Tarasuk-Levin et al.	Dec 2015	A1
20150378783	Tarasuk-Levin et al.	Dec 2015	A1
20150378785	Tarasuk-Levin et al.	Dec 2015	A1
20150378831	Tarasuk-Levin et al.	Dec 2015	A1
20150378847	Tarasuk-Levin et al.	Dec 2015	A1
20150378931	White	Dec 2015	A1
20150381589	Tarasuk-Levin et al.	Dec 2015	A1
20160026489	Maislos et al.	Jan 2016	A1
20160028761	Sitsky et al.	Jan 2016	A1
20160034702	Sikka	Feb 2016	A1
20160098302	Ben-Yehuda et al.	Apr 2016	A1
20160103728	Shetty et al.	Apr 2016	A1
20160117122	Miwa et al.	Apr 2016	A1
20160132443	Davda et al.	May 2016	A1
20160139962	Tsirkin et al.	May 2016	A1
20160147556	Hu et al.	May 2016	A1
20160196449	Sikka	Jul 2016	A1
20160203015	Joshi et al.	Jul 2016	A1
20160224371	Ramanathan et al.	Aug 2016	A1
20160232023	Koladi et al.	Aug 2016	A1
20160239328	Kaplan et al.	Aug 2016	A1
20160239428	White et al.	Aug 2016	A1
20160246639	Voccio et al.	Aug 2016	A1
20160266940	Tsirkin et al.	Sep 2016	A1
20160274896	Bowen	Sep 2016	A1
20160274934	Nelson	Sep 2016	A1
20160283421	Wilson et al.	Sep 2016	A1
20160306862	Sitsky et al.	Oct 2016	A1
20160381125	Jagadeeshwaran et al.	Dec 2016	A1
20170005813	Sobh et al.	Jan 2017	A1
20170024246	Gopalan	Jan 2017	A1
20170075706	Apfelbaum et al.	Mar 2017	A1
20170090964	Tsirkin	Mar 2017	A1
20170090968	Tsirkin et al.	Mar 2017	A1
20170132062	Shetty et al.	May 2017	A1
20170132430	Sikka	May 2017	A1
20170139755	Tsirkin et al.	May 2017	A1
20170147501	Tsirkin et al.	May 2017	A1
20170199755	Voccio et al.	Jul 2017	A1
20170201574	Luo	Jul 2017	A1
20170220374	Beveridge et al.	Aug 2017	A1
20170270012	Tarasuk-Levin et al.	Sep 2017	A1
20170272515	Sanderson	Sep 2017	A1
20170288971	Jayaraman et al.	Oct 2017	A1
20170322815	Buswell	Nov 2017	A1
20170322828	Panicker	Nov 2017	A1
20170357527	Srinivas	Dec 2017	A1
20170371696	Prziborowski et al.	Dec 2017	A1
20180004679	Serebrin et al.	Jan 2018	A1
20180018135	Shen	Jan 2018	A1
20180024854	Wang et al.	Jan 2018	A1
20180041477	Shaposhnik	Feb 2018	A1
20180052701	Kaplan et al.	Feb 2018	A1
20180060107	Hu et al.	Mar 2018	A1
20180074842	Fu et al.	Mar 2018	A1
20180074908	Tarasuk-Levin et al.	Mar 2018	A1
20180074909	Bishop et al.	Mar 2018	A1
20180088975	Kaul	Mar 2018	A1
20180101452	Blake	Apr 2018	A1
20180113610	Gopalan et al.	Apr 2018	A1
20180121247	Kaul	May 2018	A1
20180129523	Bryant et al.	May 2018	A1
20180136838	White et al.	May 2018	A1
20180136963	Srinivas	May 2018	A1
20180173547	Stokes et al.	Jun 2018	A1
20180183869	Sanderson	Jun 2018	A1
20180203721	Fahs et al.	Jul 2018	A1
20180225221	Tsirkin et al.	Aug 2018	A1
20180232254	Mohan	Aug 2018	A1
20180276025	Kaul	Sep 2018	A1
20180329737	Dong et al.	Nov 2018	A1
20180349421	Sitsky et al.	Dec 2018	A1
20180357098	Dube et al.	Dec 2018	A1
20180373557	Dong	Dec 2018	A1
20190005576	Mick et al.	Jan 2019	A1
20190020732	Mosko	Jan 2019	A1
20190034112	Tsirkin et al.	Jan 2019	A1
20190050235	Buswell	Feb 2019	A1
20190087220	Turner	Mar 2019	A1
20190087244	Turner	Mar 2019	A1
20190095232	Okuno et al.	Mar 2019	A1
20190114197	Gong	Apr 2019	A1
20190129740	Hadas et al.	May 2019	A1
20190129814	Tati et al.	May 2019	A1
20190146853	Xu et al.	May 2019	A1
20190158588	Sanderson	May 2019	A1
20190179659	Herbert	Jun 2019	A1
20190205220	Zhang	Jul 2019	A1
20190207764	Maximov et al.	Jul 2019	A1
20190220364	Yu et al.	Jul 2019	A1
20190222522	Shih et al.	Jul 2019	A1
20190228557	Ma et al.	Jul 2019	A1
20190258503	Manco et al.	Aug 2019	A1
20190265904	Nakamura et al.	Aug 2019	A1
20190278624	Bade et al.	Sep 2019	A1
20190278632	Sakai	Sep 2019	A1
20190319896	Guo et al.	Oct 2019	A1
20190334909	Schmitt et al.	Oct 2019	A1
20190354378	Battaiola Kreling et al.	Nov 2019	A1
20190370049	Gopalan	Dec 2019	A1
20190384495	Balcha	Dec 2019	A1
20190384673	Bishop et al.	Dec 2019	A1
20190391835	Gowda et al.	Dec 2019	A1
20190391843	Franciosi et al.	Dec 2019	A1
20200019429	Kaplan et al.	Jan 2020	A1
20200019512	Serebrin et al.	Jan 2020	A1
20200026547	Bryant et al.	Jan 2020	A1
20200034176	Calciu et al.	Jan 2020	A1
20200034182	Kaul	Jan 2020	A1
20200034211	Kumar P et al.	Jan 2020	A1
20200073674	Tsirkin et al.	Mar 2020	A1
20200110629	Hu et al.	Apr 2020	A1
20200150997	Chang et al.	May 2020	A1
20200174817	Ding et al.	Jun 2020	A1
20200183729	Lu et al.	Jun 2020	A1
20200293697	Sion	Sep 2020	A1
20200319897	Turner	Oct 2020	A1
20200319904	Turner	Oct 2020	A1
20200334067	Liu et al.	Oct 2020	A1
20200387392	Sajjad et al.	Dec 2020	A1
20200389469	Litichever et al.	Dec 2020	A1
20200403940	Daly et al.	Dec 2020	A1
20210049079	Kumar et al.	Feb 2021	A1
20210055946	Golan et al.	Feb 2021	A1
20210058425	Sitsky et al.	Feb 2021	A1

Non-Patent Literature Citations (323)

Entry
Mallu et al., “Live Migration of Virtual Machines in Cloud Environment: A Survey”, May 2015, Indian Journal of Science and Technology, vol. 8(S9), pp. 326-332 (Year: 2015).
Siripoonya et al., “Thread-Based Live Checkpointing of Virtual Machines”, 2011 IEEE International Symposium on Network Computing and Applications, pp. 155-162 (Year: 2011).
Lee et al., “Hypervisor-Assisted Application Checkpointing in Virtualized Environments”, 2011, 2011 IEEE/IFIP 41st International Conference on Dependable Systems & Networks (DSN), Hong Kong, China, pp. 371-382 (Year: 2011).
U.S. Appl. No. 10/013,276, filed Jul. 3, 2018, Fahs et al.
U.S. Appl. No. 10/031,607, filed Jul. 24, 2018, Smith.
U.S. Appl. No. 10/033,595, filed Jul. 24, 2018, Sif et al.
U.S. Appl. No. 10/037,334, filed Jul. 31, 2018, Lunev et al.
U.S. Appl. No. 10/044,558, filed Aug. 7, 2018, Namihira.
U.S. Appl. No. 10/049,064, filed Aug. 14, 2018, Bonzini et al.
U.S. Appl. No. 10/061,622, filed Aug. 28, 2018, Tsirkin et al.
U.S. Appl. No. 10/073,713, filed Sep. 11, 2018, Nelson.
U.S. Appl. No. 10/078,529, filed Sep. 18, 2018, Voccio et al.
U.S. Appl. No. 10/083,053, filed Sep. 25, 2018, Dong.
U.S. Appl. No. 10/089,011, filed Oct. 2, 2018, McVey et al.
U.S. Appl. No. 10/114,662, filed Oct. 30, 2018, Tsirkin et al.
U.S. Appl. No. 10/120,480, filed Nov. 6, 2018, Smith.
U.S. Appl. No. 10/120,594, filed Nov. 6, 2018, Hardt et al.
U.S. Appl. No. 10/127,049, filed Nov. 13, 2018, Buswell.
U.S. Appl. No. 10/129,331, filed Nov. 13, 2018, Jagadeeshwaran et al.
U.S. Appl. No. 10/135,948, filed Nov. 20, 2018, Mosko.
U.S. Appl. No. 10/140,112, filed Nov. 27, 2018, Takano et al.
U.S. Appl. No. 10/146,353, filed Dec. 4, 2018, Smith.
U.S. Appl. No. 10/156,921, filed Dec. 18, 2018, Smith.
U.S. Appl. No. 10/156,986, filed Dec. 18, 2018, Gopalan et al.
U.S. Appl. No. 10/157,077, filed Dec. 18, 2018, Voccio et al.
U.S. Appl. No. 10/162,448, filed Dec. 25, 2018, Smith.
U.S. Appl. No. 10/169,068, filed Jan. 1, 2019, Brouwer et al.
U.S. Appl. No. 10/178,073, filed Jan. 8, 2019, Shaposhnik.
U.S. Appl. No. 10/187,466, filed Jan. 22, 2019, Sanderson.
U.S. Appl. No. 10/191,756, filed Jan. 29, 2019, Voccio.
U.S. Appl. No. 10/198,299, filed Feb. 5, 2019, Xu et al.
U.S. Appl. No. 10/198,377, filed Feb. 5, 2019, Wilson et al.
U.S. Appl. No. 10/203,794, filed Feb. 12, 2019, Smith.
U.S. Appl. No. 10/209,806, filed Feb. 19, 2019, Smith.
U.S. Appl. No. 10/209,807, filed Feb. 19, 2019, Smith.
U.S. Appl. No. 10/209,808, filed Feb. 19, 2019, Smith.
U.S. Appl. No. 10/209,809, filed Feb. 19, 2019, Smith.
U.S. Appl. No. 10/210,567, filed Feb. 19, 2019, Mick et al.
U.S. Appl. No. 10/222,891, filed Mar. 5, 2019, Smith.
U.S. Appl. No. 10/222,892, filed Mar. 5, 2019, Smith.
U.S. Appl. No. 10/222,893, filed Mar. 5, 2019, Smith.
U.S. Appl. No. 10/222,894, filed Mar. 5, 2019, Smith.
U.S. Appl. No. 10/222,895, filed Mar. 5, 2019, Smith.
U.S. Appl. No. 10/223,276, filed Mar. 5, 2019, Tsirkin.
U.S. Appl. No. 10/235,235, filed Mar. 19, 2019, Shetty et al.
U.S. Appl. No. 10/241,814, filed Mar. 26, 2019, Tsirkin.
U.S. Appl. No. 10/268,595, filed Apr. 23, 2019, Bonzini et al.
U.S. Appl. No. 10/275,086, filed Apr. 30, 2019, Smith.
U.S. Appl. No. 10/275,087, filed Apr. 30, 2019, Smith.
U.S. Appl. No. 10/289,435, filed May 14, 2019, Bond et al.
U.S. Appl. No. 10/289,436, filed May 14, 2019, Keagy et al.
U.S. Appl. No. 10/289,684, filed May 14, 2019, Mashtizadeh et al.
U.S. Appl. No. 10/310,977, filed Jun. 4, 2019, Mola.
U.S. Appl. No. 10/318,329, filed Jun. 11, 2019, Tsirkin et al.
U.S. Appl. No. 10/324,851, filed Jun. 18, 2019, Mola.
U.S. Appl. No. 10/338,736, filed Jul. 2, 2019, Smith.
U.S. Appl. No. 10/345,961, filed Jul. 9, 2019, Smith.
U.S. Appl. No. 10/365,758, filed Jul. 30, 2019, Smith.
U.S. Appl. No. 10/386,960, filed Aug. 20, 2019, Smith.
U.S. Appl. No. 10/394,656, filed Aug. 27, 2019, Tarasuk-Levin et al.
U.S. Appl. No. 10/394,668, filed Aug. 27, 2019, Tarasuk-Levin et al.
U.S. Appl. No. 10/423,437, filed Sep. 24, 2019, Kaplan et al.
U.S. Appl. No. 10/430,221, filed Oct. 1, 2019, Tsirkin.
U.S. Appl. No. 10/430,249, filed Oct. 1, 2019, Kaul.
U.S. Appl. No. 10/430,991, filed Oct. 1, 2019, Ma et al.
U.S. Appl. No. 10/437,627, filed Oct. 8, 2019, Hu et al.
U.S. Appl. No. 10/452,559, filed Oct. 22, 2019, Serebrin et al.
U.S. Appl. No. 10/466,999, filed Nov. 5, 2019, Fu et al.
U.S. Appl. No. 10/467,106, filed Nov. 5, 2019, Tsao et al.
U.S. Appl. No. 10/474,369, filed Nov. 12, 2019, Tati et al.
U.S. Appl. No. 10/474,489, filed Nov. 12, 2019, Dong et al.
U.S. Appl. No. 10/481,940, filed Nov. 19, 2019, Srinivas.
U.S. Appl. No. 10/481,942, filed Nov. 19, 2019, Kaul.
U.S. Appl. No. 10/503,551, filed Dec. 10, 2019, Dube et al.
U.S. Appl. No. 10/503,659, filed Dec. 10, 2019, Tsirkin et al.
U.S. Appl. No. 10/509,662, filed Dec. 17, 2019, Hardt et al.
U.S. Appl. No. 10/509,687, filed Dec. 17, 2019, Jain et al.
U.S. Appl. No. 10/514,938, filed Dec. 24, 2019, Ramakrishnan Nair.
U.S. Appl. No. 10/521,047, filed Dec. 31, 2019, Smith.
U.S. Appl. No. 10/530,822, filed Jan. 7, 2020, Lyons.
U.S. Appl. No. 10/534,474, filed Jan. 14, 2020, Smith.
U.S. Appl. No. 10/540,039, filed Jan. 21, 2020, Smith.
U.S. Appl. No. 10/541,928, filed Jan. 21, 2020, Shih et al.
U.S. Appl. No. 10/545,777, filed Jan. 28, 2020, Bryant et al.
U.S. Appl. No. 10/551,966, filed Feb. 4, 2020, Smith.
U.S. Appl. No. 10/552,080, filed Feb. 4, 2020, Tsirkin et al.
U.S. Appl. No. 10/552,230, filed Feb. 4, 2020, Tsirkin et al.
U.S. Appl. No. 10/552,267, filed Feb. 4, 2020, Bishop et al.
U.S. Appl. No. 10/572,271, filed Feb. 25, 2020, Tsirkin et al.
U.S. Appl. No. 10/579,409, filed Mar. 3, 2020, Beveridge et al.
U.S. Appl. No. 10/579,412, filed Mar. 3, 2020, Manco et al.
U.S. Appl. No. 10/592,039, filed Mar. 17, 2020, Smith.
U.S. Appl. No. 10/592,273, filed Mar. 17, 2020, Fahs et al.
U.S. Appl. No. 10/592,428, filed Mar. 17, 2020, Saidi.
U.S. Appl. No. 10/606,396, filed Mar. 31, 2020, Smith.
U.S. Appl. No. 10/628,200, filed Apr. 21, 2020, Hadas et al.
U.S. Appl. No. 10/635,474, filed Apr. 28, 2020, Panicker.
U.S. Appl. No. 10/635,477, filed Apr. 28, 2020, Kaul.
U.S. Appl. No. 10/642,413, filed May 5, 2020, Smith.
U.S. Appl. No. 10/642,539, filed May 5, 2020, Shen.
U.S. Appl. No. 10/649,571, filed May 12, 2020, Smith.
U.S. Appl. No. 10/649,578, filed May 12, 2020, Smith.
U.S. Appl. No. 10/649,579, filed May 12, 2020, Smith.
U.S. Appl. No. 10/649,580, filed May 12, 2020, Smith.
U.S. Appl. No. 10/649,581, filed May 12, 2020, Smith.
U.S. Appl. No. 10/656,752, filed May 19, 2020, Smith.
U.S. Appl. No. 10/656,753, filed May 19, 2020, Smith.
U.S. Appl. No. 10/656,754, filed May 19, 2020, Smith.
U.S. Appl. No. 10/656,755, filed May 19, 2020, Smith.
U.S. Appl. No. 10/656,756, filed May 19, 2020, Smith.
U.S. Appl. No. 10/656,757, filed May 19, 2020, Smith.
U.S. Appl. No. 10/656,758, filed May 19, 2020, Smith.
U.S. Appl. No. 10/656,759, filed May 19, 2020, Smith.
U.S. Appl. No. 10/664,097, filed May 26, 2020, Smith.
U.S. Appl. No. 10/666,516, filed May 26, 2020, Jayaraman et al.
U.S. Appl. No. 10/671,212, filed Jun. 2, 2020, Smith.
U.S. Appl. No. 10/671,213, filed Jun. 2, 2020, Smith.
U.S. Appl. No. 10/684,874, filed Jun. 16, 2020, Keagy et al.
U.S. Appl. No. 10/698,711, filed Jun. 30, 2020, Sobh et al.
U.S. Appl. No. 10/713,123, filed Jul. 14, 2020, Yu et al.
U.S. Appl. No. 10/725,581, filed Jul. 28, 2020, Smith.
U.S. Appl. No. 10/725,823, filed Jul. 28, 2020, Turull et al.
U.S. Appl. No. 10/740,016, filed Aug. 11, 2020, White et al.
U.S. Appl. No. 10/782,819, filed Sep. 22, 2020, Smith.
U.S. Appl. No. 10/788,931, filed Sep. 29, 2020, Smith.
U.S. Appl. No. 10/796,035, filed Oct. 6, 2020, Xia et al.
U.S. Appl. No. 10/810,037, filed Oct. 20, 2020, Liu et al.
U.S. Appl. No. 10/824,716, filed Nov. 3, 2020, Howell et al.
U.S. Appl. No. 10/826,930, filed Nov. 3, 2020, Sitsky et al.
U.S. Appl. No. 10/838,542, filed Nov. 17, 2020, Smith.
U.S. Appl. No. 10/853,119, filed Dec. 1, 2020, Ding et al.
U.S. Appl. No. 10/936,114, filed Mar. 2, 2021, Smith.
U.S. Appl. No. 10/936,330, filed Mar. 2, 2021, Battaiola Kreling et al.
U.S. Appl. No. 10/956,197, filed Mar. 23, 2021, Herbert.
U.S. Appl. No. 10/963,367, filed Mar. 30, 2021, Mola.
[1] A. Barak and R. Wheeler. Mosix: An integrated multiprocessor UNIX. In Mobility, pp. 41-53. ACM Press/Addison—Wesley Publishing Co., New York, Ny, USA, 1999.
[2] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. Xen and the art of virtualization. SIGOPS Oper. Syst. Rev., 37(5):164-177, Oct. 2003.
[3] F. Bellard. Qemu, a fast and portable dynamic translator. In Proc. of USENIX Annual Technical Conference, Apr. 2005.
[4] J. Bonwick and J. Adams. Magazines and vmem: Extending the slab allocator to many cpus and arbitrary resources. In USENIX Annual Technical Conference, pp. 15-33, 2001.
[5] T. C. Bressoud and F. B. Schneider. Hypervisor-based fault tolerance. In Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles, SOSP '95, pp. 1-11, New York, NY, USA, 1995. ACM.
[6] C. Clark, K. Fraser, S. Hand, J. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield. Live Migration of Virtual Machines. In USENIX Symposium on Networked Systems Design and Implementation (NSDI), pp. 273-286, 2005.
[7] B. Cully, G. Lefebvre, D. Meyer, M. Feeley, N. Hutchinson, and A. Warfield. Remus: High availability via asynchronous virtual machine replication. In USENIX Symposium on Networked Systems Design and Implementation (NSDI), pp. 161-174, 2008.
[8] U. Deshpande, D. Chan, S. Chan, K. Gopalan, and N. Bila. Scattergather live migration of virtual machines. IEEE Transactions on Cloud Computing, pp. (99):1-1, 2015.
[9] U. Deshpande, Y. You, D. Chan, N. Bila, and K. Gopalan. Fast server deprovisioning through scatter-gather live migration of virtual machines. In 2014 IEEE 7th International Conference on Cloud Computing, pp. 376-383, Jun. 2014.
[10] Y. Dong, W. Ye, Y. Jiang, I. Pratt, S. Ma, J. Li, and H. Guan. Colo: Coarse-grained lock-stepping virtual machines for non-stop service. In the 4th Annual Symposium on Cloud Computing, pp. 1-16, 2013.
[11] G. W. Dunlap, S. T. King, S. Cinar, M. A. Basrai, and P. M. Chen. Revirt: Enabling intrusion analysis through virtual-machine logging and replay. SIGOPS Oper. Syst. Rev., 36:211-224, Dec. 2002.
[12] G. W. Dunlap, III. Execution Replay for Intrusion Analysis. PhD thesis, Ann Arbor, MI, USA, 2006.
[13] C. Engelmann, G. R. Vallee, T. Naughton, and S. L. Scott. Proactive fault tolerance using preemptive migration. In 17th Euromicro International Conference on Parallel, Distributed and Network-based Processing, pp. 252-257, 2009.
[14] D. Fernando, Hardik Bagdi, Yaohui Hu, Ping Yang, Kartik Gopalan, Charles Kamhoua, and Kevin Kwiat. “Quick eviction of virtual machines through proactive live snapshots.” In 2016 IEEE/ACM 9th International Conference on Utility and Cloud Computing (UCC), pp. 99-107. IEEE, 2016.
[15] D. Fernando, H. Bagdi, Y. Hu, P. Yang, K. Gopalan, C. Kamhoua, and K. Kwiat. Quick eviction of virtual machines through proactive snapshots. In 2016 IEEE International Conference on Cluster Computing (CLUSTER), pp. 156-157, Sep. 2016.
[16] M. Hines, U. Deshpande, and K. Gopalan. Post live migration of virtual machines. In ACM SIGOPS Operating Systems Review, 43, No. 3: Jul. 14-26, 2009.
[17] M. R. Hines and Kartik Gopalan. “Post based live virtual machine migration using adaptive pre-paging and dynamic self-ballooning.” In Proceedings of the 2009 ACM SIGPLAN/SIGOPS international conference on Virtual execution environments (VEE), pp. 51-60. 2009.
[18] Iperf. The TCP/UDP Bandwidth Measurement Tool, dast.nlanr.net/Projects/Iperf/.
[19] R. Jhawar, V. Piuri, and M. Santambrogio. Fault tolerance management in cloud computing: A system-level perspective. IEEE Systems Journal, 7(2):288-297, Jun. 2013.
[20] A. Kangarlou, P. Eugster, and D. Xu. Vnsnap: Taking snapshots of virtual networked infrastructures in the cloud. IEEE Transactions on Services Computing, 5(4):484-496, 2012.
[21] S. T. King, G. W. Dunlap, and P. M. Chen. Debugging operating systems with time-traveling virtual machines. In Annual Conference on USENIX Annual Technical Conference, pp. 1-1, 2005.
[22] A. Kivity, Y. Kamay, D. Laor, U. Lublin, and A. Liguori. Kvm: The linux virtual machine monitor. In Proc. of Linux Symposium, Jun. 2007.
[23] Y. Kwon, M. Balazinska, and A. Greenberg. Fault-tolerant stream processing using a distributed, replicated file system. Proc. VLDB Endow., 1(1), Aug. 2008.
[24] M. Lu and T. c. Chiueh. Fast memory state synchronization for virtualization-based fault tolerance. In 2009 IEEE/IFIP International Conference on Dependable Systems Networks, 2009.
[25] D. Marques, G. Bronevetsky, R. Fernandes, K. Pingali, and P. Stodghil. Optimizing checkpoint sizes in the c3 system. In the 19th IEEE International Parallel and Distributed Processing Symposium, 2005.
[26] R. McDougall and J. Maura. Solaris internals. In Rachael Borden, 2001. [27] Micro Checkpointing. wiki.qemu.org/features/microcheckpointing. [28] S. J. Mullender, G. van Rossum, A. S. Tananbaum, R. van Renesse, and H. van Staveren. Amoeba: a distributed operating system for the 1990s. Computer, 23(5):44-53, May 1990.
[29] A. B. Nagarajan, F. Mueller, C. Engelmann, and S. L. Scott. Proactive fault tolerance for hpc with xen virtualization. In the 21st Annual International Conference on Supercomputing, pp. 23-32, 2007.
[30] M. Nelson, B. H. Lim, and G. Hutchins. Fast transparent migration for virtual machines. In Proc. of USENIX Annual Technical Conference, Apr. 2005.
[31] J. K. Ousterhout, A. R. Cherenson, F. Douglis, M. N. Nelson, and B. B. Welch. The sprite network operating system. Computer, 21(2):23-36, Feb. 1988.
[32] E. Park, B. Egger, and J. Lee. Fast and space-efficient virtual machine checkpointing. In the 7th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, pp. 75-86, 2011.
[33] R. F. Rashid and G. G. Robertson. Accent: A communication oriented network operating system kernel. In the Eighth ACM Symposium on Operating Systems Principles, pp. 64-75, 1981.
[34] Redis. In-memory data structure store, redis.io/.
[35] A. Ruprecht, D. Jones, D. Shiraev, G. Harmon, M. Spivak, M. Krebs, M. Baker-Harvey, and T. Sanderson. Vm live migration at scale. In the 14th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, pp. 45-56.
[36] Y. Tamura, K. Sato, S. Kihara, and S. Moriai. Kemari: Virtual machine synchronization for fault tolerance using domt, ntt cyber space labs. Technical report, Proc. USENIX Annual Technical Conference, 2008.
[37] VMWare Inc. VMWare IncArchitecture of VMware ESXi, www.vmware.com/files/pdf/esxiarchitecture.pdf.
[38] VMWare Inc. VMWare vMotion www.vmware.com/pdf/vmotion datasheet.pdf.
[39] VMWare Inc. VMWare vSphere 6 Fault Tolerance www.vmware.com/files/pdf/techpaper/vmware-vsphere6-ft-arch-perf.pdf. [40] C. Wang, X. Chen, Z. Wang, Y. Zhu, and H. Cui. A fast, general storage replication protocol for active-active virtual machine fault tolerance. In 2017 IEEE 23rd International Conference on Parallel and Distributed Systems (ICPADS), pp. 151-160, Dec. 2017.
[41] C. Wang, F. Mueller, C. Engelmann, and S. L. Scott. Proactive processlevel live migration in hpc environments. In International Conference for High Performance Computing, Networking, Storage and Analysis, 2008.
[42] M. Xu, R. Bodik, and M. D. Hill. A “flight data recorder” for enabling full-system multiprocessor deterministic replay. In the 30th International Symposium on Computer Architecture, pp. 122-135, 2003.
[43] Abe, Yoshihisa, Roxana Geambasu, Kaustubh Joshi, and Mahadev Satyanarayanan. “Urgent virtual machine eviction with enlightened post.” ACM SIGPLAN Notices 51, No. 7 (2016): 51-64.
[44] Asai, Hirochika. “P2V Migration with Post Hot Cloning for Service Downtime Reduction.” In 2013 International Conference on Cloud and Green Computing, pp. 1-8. IEEE, 2013.
[45] Ben-Yehuda, Muli, Rom Frieman, Abel Gordon, Benoit Hudzia, and Maor Vanmak. “Resilient post live migration using eviction to shared storage in a global memory architecture.” U.S. Appl. No. 14/588,424, filed Apr. 7, 2016.
[46] Chen, Xiao, Jian-Hui Jiang, and Qu Jiang. “A Method of Self-Adaptive Pre Container Checkpoint.” In 2015 IEEE 21st Pacific Rim International Symposium on Dependable Computing (PRDC), pp. 290-300. IEEE, 2015.
[47] Chou, Chih, Yuan Chen, Dejan Milojicic, Narasimha Reddy, and Paul Gratz. “Optimizing Post Live Migration with System-Level Checkpoint Using Fabric-Attached Memory.” In 2019 IEEE/ACM Workshop on Memory Centric High Performance Computing (MCHPC), pp. 16-24. IEEE, 2019.
[48] Dadrwal, Ashu, Suryaprakash Nehra, Ali Ahmad Khan, and Mohit Dua. “Checkpoint Based Live Migration of Virtual Machine.” In 2018 International Conference on Communication and Signal Processing (ICCSP), pp. 1083-1086. IEEE, 2018.
[49] Deshpande, Umesh, Danny Chan, Steven Chan, Kartik Gopalan, and Nilton Bila. “Scatter-gather live migration of virtual machines.” IEEE Transactions on Cloud Computing 6, No. 1 (2015): 196-208.
[50] Dhuria, Ritika, and Kiranbir Kaur. “A Hybrid Approach For Reducing Downtime, Migration Time And Energy Consumption of Live VM Migration.” International Journal of Advanced Research in Computer Science 8, No. 8 (2017).
[51] Fernando, Dinuni, Jonathan Terner, Kartik Gopalan, and Ping Yang. “Live Migration Ate My VM: Recovering a Virtual Machine after Failure of Post Live Migration.” In IEEE INFOCOM 2019—IEEE Conference on Computer Communications, pp. 343-351. IEEE, 2019.
[52] Gad, Ramy, Simon Pickartz, Tim Süβ, Lars Nagel, Stefan Lankes, Antonello Monti, and André Brinkmann. “Zeroing memory deallocator to reduce checkpoint sizes in virtualized HPC environments.” The Journal of Supercomputing 74, No. 11 (2018): 6236-6257.
[53] Gao, Xing, Jidong Xiao, Haining Wang, and Angelos Stavrou. “Understanding the Security Implication of Aborting Live Migration.” IEEE Transactions on Cloud Computing (2020).
[54] Hines, Michael R., G. Kartik, and D. Umesh. “Post Live Migration Of VM.” In Proc. of ACM International Conference on Virtual Execution Environments. 2009.
[55] Hirofuchi, Takahiro, Hidemoto Nakada, Satoshi Itoh, and Satoshi Sekiguchi. “Reactive consolidation of virtual machines enabled by postcopy live migration.” In Proceedings of the 5th international workshop on Virtualization technologies in distributed computing, pp. 11-18. 2011.
[56] Hirofuchi, Takahiro, Hidemoto Nakada, Satoshi Itoh, and Satoshi Sekiguchi. “Making vm consolidation more energy-efficient by postcopy live migration.” In Cloud Computing 2011, The Second International Conference on Cloud Computing, GRIDs, and Virtualization, pp. 195-204. 2011.
[57] Hirofuchi, Takahiro, Hidemoto Nakada, Satoshi Itoh, and Satoshi Sekiguchi. “Reactive cloud: Consolidating virtual machines with postcopy live migration.” Information and Media Technologies 7, No. 2 (2012): 614-626.
[58] Hirofuchi, Takahiro, Hidemoto Nakada, Satoshi Itoh, and Satoshi Sekiguchi. “Reactive cloud: Consolidating virtual machines with postcopy live migration.” Information and Media Technologies 7, No. 2 (2012): 614-626.
[59] Hirofuchi, Takahiro, Isaku Yamahata, and Satoshi Itoh. “Postcopy live migration with guest-cooperative page faults.” IEICE Transactions on Information and Systems 98, No. 12 (2015): 2159-2167.
[60] Hursey, Joshua. “A Transparent Process Migration Framework for Open MPI.” (2012).
[61] Lu, Peng, Antonio Barbalace, and Binoy Ravindran. “HSG-LM: hybrid speculative guest OS live migration without hypervisor.” In Proceedings of the 6th International Systems and Storage Conference, pp. 1-11. 2013.
[62] Mabe, Dusty. “The State of Virtual Machine Checkpointing, Live Migration and Replication.” (2013).
[63] Mosko, Marc. “Process Migration over CCNx.” arXiv preprint arXiv:1707.04835 (2017).
[64] Olivier, Pierre, AKM Fazla Mehrab, Stefan Lankes, Mohamed Lamine Karaoui, Robert Lyerly, and Binoy Ravindran. “HEXO: Offloading HPC Compute-Intensive Workloads on Low-Cost, Low-Power Embedded Systems.” In Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing, pp. 85-96. 2019.
[65] Rapoport, Mike, and Joel Nider. “User space memory management for post migration.” In Proceedings of the 10th ACM International Systems and Storage Conference, pp. 1-1. 2017.
[66] Rathi, Rakesh. “VM Migration Techniques in Cloud: A Review.” International Journal of Computer Applications 975: 8887. 2019.
[67] Sahni, Shashank, and Vasudeva Varma. “A hybrid approach to live migration of virtual machines.” In 2012 IEEE International Conference on Cloud Computing in Emerging Markets (CCEM), pp. 1-5. IEEE, 2012.
[68] Shah, Syed Asif Raza, Amol Hindurao Jaikar, and Seo-Young Noh. “A performance analysis of precopy, postcopy and hybrid live VM migration algorithms in scientific cloud computing environment.” In 2015 International Conference on High Performance Computing & Simulation (HPCS), pp. 229-236. IEEE, 2015.
[69] Shamsinezhad, Elham, Asadollah Shahbahrami, Alireza Hedayati, Ahmad Khadem Zadeh, and Hamid Banirostam. “Presentation methods for task migration in cloud computing by combination of yu router and post.” International Journal of Computer Science Issues (IJCSI) 10, No. 4 (2013): 98.
[70] Shan, Zhongyuan, Jianzhong Qiao, and Shukuan Lin. “Fix page fault in post live migration with RemotePF page table assistant.” In 2018 17th International Symposium on Distributed Computing and Applications for Business Engineering and Science (DCABES), pp. 40-43. IEEE, 2018.
[71] Shribman, Aidan, and Benoit Hudzia. “Pre and post vm live migration for memory intensive applications.” In European Conference on Parallel Processing, pp. 539-547. Springer, Berlin, Heidelberg, 2012.
[72] Sivagami, V. M. “Efficient Data Transmission during Virtual Machine Failures Using Hybrid Live Migration.” Proceedings of National Conference on Communication and Informatics—2016, Organized by Department of Information Technology, Sri Venkateswara College of Engineering, Sriperumbudur.
[73] Soni, Sushil Kumar, and Ravi Kant Kapoor. “Enhanced live migration of virtual machine using comparison of modified and unmodified pages.” IJCSMC 3, No. 2 (2014): 779-787.
[74] Stoyanov, Radostin, and Martin J. Kollingbaum. “Efficient live migration of Linux containers.” In International Conference on High Performance Computing, pp. 184-193. Springer, Cham, 2018.
[75] Su, Kui, Wenzhi Chen, Guoxi Li, and Zonghui Wang. “Rpff: A remote page-fault filter for post live migration.” In 2015 IEEE International Conference on Smart City/SocialCom/SustainCom (SmartCity), pp. 938-943. IEEE, 2015.
[76] Sun, Michael H., and Douglas M. Blough. Fast, lightweight virtual machine checkpointing. Georgia Institute of Technology, 2010.
[77] Svärd, Petter, B. Hudzia, S. Walsh, Johan Tordsson, and Erik Elmroth. “The Noble Art of Live VM Migration-Principles and performance of pre, post and hybrid migration of demanding workloads.” Technical Report, (2014).
[78] Takano, Ryousei, Hidemoto Nakada, Takahiro Hirofuchi, Yoshio Tanaka, and Tomohiro Kudoh. “Cooperative VM migration for a virtualized HPC cluster with VMM-bypass I/O devices.” In 2012 IEEE 8th International Conference on E-Science, pp. 1-8. IEEE, 2012.
[79] Fernando, Denuni, “Fast And Robust Live Virtual Machine Migration”, Ph.D. Dissertation, Binghamton University (2019).
[80] M. Nelson, B. Lim, and G. Hutchins. Fast transparent migration for virtual machines. In Proceedings of the annual conference on USENIX Annual Technical Conference, pp. 25-25, 2005.
[81] M. Nelson, B. H Lim, and G. Hutchins. Fast Transparent Migration for Virtual Machines. In USENIX Annual Technical Conference, 2005.
[82] John K. Ousterhout, Andrew R. Cherenson, Frederick Douglis, Michael N. Nelson, and Brent B. Welch. The sprite network operating system. Computer, 21(2):23-36, Feb. 1988.
[83] Eunbyung Park, Bernhard Egger, and Jaejin Lee. Fast and space-efficient virtual machine checkpointing. In the 7th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, pp. 75-86, 2011.
[84] Richard F. Rashid and George G. Robertson. Accent: A communication oriented network operating system kernel. In the Eighth ACM Symposium on Operating Systems Principles, pp. 64-75, 1981.
[85] Redis. Key-value Cache and Store, redis.io.
[86] Michael Richmond and Michael Hitchens. A new process migration algorithm. SIGOPS Oper. Syst. Rev., 31(1):31-42, 1997.
[87] P. Riteau, C. Morin, and T. Priol. Shrinker: Improving live migration of virtual clusters over wans with distributed data deduplication and content-based addressing. In Proc. of EURO-PAR, Sep. 2011.
[88] Adam Ruprecht, Danny Jones, Dmitry Shiraev, Greg Harmon, Maya Spivak, Michael Krebs, Miche Baker-Harvey, and Tyler Sanderson. Vm live migration at scale. In the 14th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, pp. 45-56.
[89] Jonathan M. Smith. A survey of process migration mechanisms. SIGOPS Oper. Syst. Rev., 22(3):28-40, Jul. 1988.
[90] Gang Sun, Dan Liao, Vishal Anand, Dongcheng Zhao, and Hongfang Yu. A new technique for efficient live migration of multiple virtual machines. Future Gener. Comput. Syst., 55:74-86, Feb. 2016.
[91] P. Svard, B. Hudzia, J. Tordsson, and E. Elmroth. Evaluation of delta compression techniques for efficient live migration of large virtual machines. In VEE, 2011.
[92] T. Hirofuchi and I. Yamahata. Postcopy live migration for qemu/kvm. grivon.apgrid.org/quick-kvm-migration.
[93] Yoshiaki Tamura, Koji Sato, Seiji Kihara, and Satoshi Moriai. Kemari: Virtual machine synchronization for fault tolerance using domt, ntt cyber space labs. Technical report, Proc. USENIX Annual Technical Conference, 2008.
[94] VMWare Inc. VMware DRS: Dynamic Scheduling of System Resources, www.vmware.com/files/pdf/drs datasheet.pdf.
[95] VMWare Inc. VMWare Inc. Architecture of VMware ESXi, www.vmware.com/files/pdf/esxi-architecture.pdf.
[96] VMWare Inc. VMWare vMotion www.vmware.com/pdf/vmotion datasheet.pdf.
[97] VMWare Inc. VMWare vSphere 6 Fault Tolerance www.vmware.com/files/pdf/techpaper/vmware-vsphere6-ft-arch-perf.pdf.
[98] VMWare Inc. VMware vSphere vMotion Architecture, Performance and Best Practices in VMware vSphere 5, www.vmware.com/files/pdf/vmotion-perf-vsphere5.pdf.
[99] VMWare Inc. Protecting Mission-Critical Workloads with VMware Fault Tolerance. White Paper, Revision: May 7, 2009 Item: WP-084-PRD-01-02, 2009.
[100] VMWare Knowledge Base. Virtual machine performance degrades while a vMotion is being performed, kb.vmware.com/kb/2007595.
[101] C. A. Waldspurger. Memory resource management in VMware ESX server. In Operating Systems Design and Implementation, Dec. 2002.
[102] C. Wang, F. Mueller, C. Engelmann, and S. L. Scott. Proactive process-level live migration in hpc environments. In International Conference for High Performance Computing, Networking, Storage and Analysis, 2008.
[103] H. Wang, Y. Li, Y. Zhang, and D. Jin. Virtual machine migration planning in software-defined networks. IEEE Transactions on Cloud Computing, pp. 1-1, 2017.
[104] Jian Wang, Kwame-Lante Wright, and Kartik Gopalan. Xenloop: A transparent high performance inter-vm network loopback. In Proceedings of the 17th International Symposium on High Performance Distributed Computing, HPDC '08, pp. 109-118, 2008.
[105] D. Williams, H. Jamjoom, Y.-H. Liu, and H. Weatherspoon. Overdriver: Handling memory overload in an oversubscribed cloud. In ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE), 2011.
[106] T. Wood, K. K. Ramakrishnan, P. Shenoy, and J. van der Merwe. Cloudnet: Dynamic pooling of cloud resources by live WAN migration of virtual machines. In ACM SIG-PLAN/SIGOPS International Conference on Virtual Execution Environments (VEE), 2011.
[107] T. Wood, P. Shenoy, A. Venkataramani, and M. Yousif. Sandpiper: Black-box and gray-box resource management for virtual machines. Intl. Journal of Computer and Telecommunications Networking, 53(17), 2009.
[108] Timothy Wood, Prashant Shenoy, Arun Venkataramani, and Mazin Yousif. Blackbox and gray-box strategies for virtual machine migration. In Proceedings of the 4th USENIX Conference on Networked Systems Design & Implementation, NSDI'07, pp. 17-17, 2007.
[191] N. Bila, E. de Lara, K. Joshi, H. A. Lagar-Cavilla, M. Hiltunen, and M. Satyanarayanan. Jettison: Efficient Idle Desktop Consolidation with Partial VM Migration. In Eurosys, Apr. 2012.
[110] I. Zhang, A. Garthwaite, Y. Baskakov, and K. C. Barr. Fast restore of checkpointed memory using working set estimation. In ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE), 2011.
[111] M. F. Zhani, Q. Zhang, G. Simona, and R. Boutaba. Vdc planner: Dynamic migration-aware virtual data center embedding for clouds. In 2013 IFIP/IEEE International Symposium on Integrated Network Management (IM 2013), pp. 18-25, 2013.
[112] M. Zhao and R. J. Figueiredo. Experimental study of virtual machine migration in support of reservation of cluster resources. In Virtualization Technology in Distributed Computing (VTDC), 2007 Second International Workshop on, pp. 1-8, 2007.
[113] Jeff Bonwick and Jonathan Adams. Magazines and vmem: Extending the slab allocator to many cpus and arbitrary resources. In USENIX Annual Technical Conference, pp. 15-33, 2001.
[114] T. C. Bressoud and F. B. Schneider. Hypervisor-based fault tolerance. In Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles, SOSP '95, pp. 1-11, New York, NY, USA, 1995. ACM.
[115] Martin Casado, Michael J. Freedman, Justin Pettit, Jianying Luo, Nick Mckeown, and Scott Shenker. Ethane: Taking control of the enterprise. In Proceedings of the 2007 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications, SIGCOMM '07, pp. 1-12, New York, NY, USA, 2007. ACM.
[116] J. Chiang, H. Li, and T. Chiueh. Working Set-based Physical Memory Ballooning.
In The International Conference on Autonomic Computing (ICAC), Jun. 2013.
[117] C. Clark, K. Fraser, S. Hand, J. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield. Live migration of virtual machines. In Symposium on Networked Systems Design & Implementation, pp. 273-286, 2005.
[118] C. Clark, K. Fraser, S. Hand, J.G. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield. Live Migration of Virtual Machines. In USENIX Symposium on Networked Systems Design and Implementation (NSDI), pp. 273-286, 2005.
[119] C. Clark, K. Fraser, S. Hand, J.G. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield. Live Migration of Virtual Machines. In USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2005.
[120] C. Clark, K. Fraser, S. Hand, J.G. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield. Live migration of virtual machines. In Network System Design and Implementation, 2005.
[121] Brian F. Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. Benchmarking cloud serving systems with ycsb. In Proceedings of the 1st ACM Symposium on Cloud Computing, SoCC '10, pp. 143-154. ACM, 2010.
[122] Brendan Cully, Geoffrey Lefebvre, Dutch Meyer, Mike Feeley, Norm Hutchinson, and Andrew Warfield. Remus: High availability via asynchronous virtual machine replication. In Proceedings of the 5th USENIX Symposium on Networked Systems Design and Implementation, NSDI'08, pp. 161-174, 2008.
[123] Tathagata Das, Pradeep Padala, Venkata N. Padmanabhan, Ramachandran Ramjee, and Kang G. Shin. Litegreen: Saving energy in networked desktops using virtualization. In Proceedings of the 2010 USENIX Conference on USENIX Annual Technical Conference, USENIXATC'10, pp. 3-3, 2010.
[124] U. Deshpande, D. Chan, T. Y. Guh, J. Edouard, K. Gopalan, and N. Bila. Agile live migration of virtual machines. In 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS), 2016.
[125] U. Deshpande, B. Schlinker, E. Adler, and K. Gopalan. Gang migration of virtual machines using cluster-wide deduplication. In IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, May 2013.
[126] U. Deshpande, B. Wang, S. Haque, M. Hines, and K. Gopalan. Memx: Virtualization of cluster-wide memory. In Proc. of International Conference on Parallel Processing, Sep. 2010.
[127] U. Deshpande, X. Wang, and K. Gopalan. Live gang migration of virtual machines. In Proc. of High Performance Distributed Computing, Jun. 2010.
[128] U. Deshpande, X. Wang, and K. Gopalan. Live gang migration of virtual machines. In ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC), 2011.
[129] U. Deshpande, Y. You, D. Chan, N. Bila, and K. Gopalan. Fast server deprovisioning through scatter-gather live migration of virtual machines. In 2014 IEEE 7th International Conference on Cloud Computing, pp. 376-383, Jun. 2014.
[130] Umesh Deshpande, Danny Chan, Steven Chan, Kartik Gopalan, and Nilton Bila. Scatter-gather live migration of virtual machines. IEEE Transactions on Cloud Computing, pp. (99):1-1, 2015.
[131] Umesh Deshpande, Danny Chan, Steven Chan, Kartik Gopalan, and Nilton Bila. Scatter-gather live migration of virtual machines. IEEE Transactions on Cloud Computing, 2015.
[132] Umesh Deshpande, Danny Chan, Ten-Young Guh, James Edouard, Kartik Gopalan, and Nilton Bila. Agile live migration of virtual machines. In IEEE International Parallel and Distributed Processing Symposium, Chicago, IL, USA, May 2016.
[133] YaoZu Dong, Wei Ye, YunHong Jiang, Ian Pratt, ShiQing Ma, Jian Li, and HaiBing Guan. Colo: Coarse-grained lock-stepping virtual machines for non-stop service. In the 4th Annual Symposium on Cloud Computing, pp. 1-16, 2013.
[134] George W. Dunlap, Samuel T. King, Sukru Cinar, Murtaza A. Basrai, and Peter M. Chen. Revirt: Enabling intrusion analysis through virtual-machine logging and replay. SIGOPS Oper. Syst. Rev., 36:211-224, Dec. 2002.
[135] George Washington Dunlap, III. Execution Replay for Intrusion Analysis. PhD thesis, Ann Arbor, MI, USA, 2006.
[136] C. Engelmann, G. R. Vallee, T. Naughton, and S. L. Scott. Proactive fault tolerance using preemptive migration. In 17th Euromicro International Conference on Parallel, Distributed and Network-based Processing, pp. 252-257, 2009.
[137] D. Fernando, H. Bagdi, Y. Hu, P. Yang, K. Gopalan, C. Kamhoua, and K. Kwiat. Quick eviction of virtual machines through proactive snapshots. In 2016 IEEE International Conference on Cluster Computing (CLUSTER), pp. 156-157, Sep. 2016.
[138] Dinuni Fernando, Hardik Bagdi, Yaohui Hu, Ping Yang, Kartik Gopalan, Charles Kamhoua, and Kevin Kwiat. Quick eviction of virtual machines through proactive live snapshots. In Proceedings of the 9th International Conference on Utility and Cloud Computing, UCC '16, pp. 99-107, 2016.
[139] Dinuni Fernando, Jonathan Terner, Ping Yang, and Kartik Gopalan. Live migration ate my vm: Recovering a virtual machine after failure of live migration. In Proceedings of IEEE International Conference on Computer Communications, INFOCOM, 2019.
[140] Google Cloud Platform for Data Center Professionals. cloud.google. com/docs/compare/data-centers/, 2017.
[141] Soudeh Ghorbani and Matthew Caesar. Walk the line: Consistent network updates with bandwidth guarantees. In Proceedings of the First Workshop on Hot Topics in Software Defined Networks, HotSDN '12, pp. 67-72, New York, NY, USA, 2012. ACM.
[142] Edward L. Haletky. VMware ESX Server in the Enterprise: Planning and Securing Virtualization Servers. 1 edition.
[143] F. Hermenier, J. Lawall, and G. Muller. Btrplace: A flexible consolidation manager for highly available applications. IEEE Transactions on Dependable and Secure Computing, 10(5):273-286, 2013.
[144] Fabien Hermenier, Xavier Lorca, Jean-Marc Menaud, Gilles Muller, and Julia Lawall. Entropy: A consolidation manager for clusters. In Proceedings of the 2009 ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, VEE '09, pp. 41-50, 2009.
[145] Fabien Hermenier, Xavier Lorca, Jean-Marc Menaud, Gilles Muller, and Julia Lawall. Entropy: A consolidation manager for clusters. In Proceedings of the 2009 ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, VEE '09, pp. 41-50, 2009.
[146] M. R. Hines, U. Deshpande, and K. Gopalan. Post live migration of virtual machines. SIGOPS Operating System Review, 43(3):14-26, 2009.
[147] Michael R. Hines, Umesh Deshpande, and Kartik Gopalan. Post live migration of virtual machines. SIGOPS Operating System Review, 43(3):14-26, 2009.
[148] Michael R. Hines and Kartik Gopalan. Post based live virtual machine migration using adaptive pre-paging and dynamic self-ballooning. In VEE, pp. 51-60, 2009.
[149] T. Hirofuchi, H. Nakada, S. Itoh, and S. Sekiguchi. Reactive cloud: Consolidating virtual machines with postcopy live migration. IPSJ Transactions on Advanced Computing Systems, pp. 86-98, Mar. 2012.
[150] R. Jhawar, V. Piuri, and M. Santambrogio. Fault tolerance management in cloud computing: A system-level perspective. IEEE Systems Journal, 7(2):288-297, Jun. 2013.
[151] H. Jin, L. Deng, S. Wu, X. Shi, and X. Pan. Live virtual machine migration with adaptive, memory compression. In Proc. of Cluster Computing and Workshops, Aug. 2009.
[152] C. Jo, E. Gustafsson, J. Son, and B. Egger. Efficient live migration of virtual machines using shared storage. In ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE), 2013.
[153] A. Kangarlou, P. Eugster, and D. Xu. Vnsnap: Taking snapshots of virtual networked infrastructures in the cloud. IEEE Transactions on Services Computing, 5(4):484-496, 2012.
[154] S. M. Kerner. Why facebook does sdn. www.enterprisenetworkingplanet.com/datacenter/why-facebook-does-sdn. html, 2014.
[155] Samuel T. King, George W. Dunlap, and Peter M. Chen. Debugging operating systems with time-traveling virtual machines. In Annual Conference on USENIX Annual Technical Conference, pp. 1-1, 2005.
[156] Avi Kivity, Yaniv Kamay, Dor Laor, Uri Lublin, and Anthony Liguori. Kvm: The linux virtual machine monitor. In Linux Symposium, Jun. 2007.
[157] Thomas Knauth and Christof Fetzer. Vecycle: Recycling VM checkpoints for faster migrations. In Proceedings of the 16th Annual Middleware Conference, pp. 210-221, 2015.
[158] C. Kolivas. Kernbench. ck.kolivas.org/apps/kernbench/ kernbench-0.50/.
[159] YongChul Kwon, Magdalena Balazinska, and Albert Greenberg. Fault-tolerant stream processing using a distributed, replicated file system. Proc. VLDB Endow., 1(1), Aug. 2008.
[160] Linux Foundation. openvswitch.org/.
[161] H. Liu and B. He. Vmbuddies: Coordinating live migration of multi-tier applications in cloud environments. IEEE Transactions on Parallel and Distributed Systems, 26(4):1192-1205, 2015.
[162] H. Lu, C. Xu, C. Cheng, R. Kompella, and D. Xu. vhaul: Towards optimal scheduling of live multi-vm migration for multi-tier applications. In 2015 IEEE 8th International Conference on Cloud Computing, pp. 453-460, Jun. 2015.
[163] M. Lu and T. c. Chiueh. Fast memory state synchronization for virtualization-based fault tolerance. In 2009 IEEE/IFIP International Conference on Dependable Systems Networks, 2009.
[164] V. Mann, A. Vishnoi, A. Iyer, and P. Bhattacharya. Vmpatrol: Dynamic and automated qos for virtual machine migrations. In 2012 8th international conference on network and service management (cnsm) and 2012 workshop on systems virtualiztion management (svm), pp. 174-178, Oct. 2012.
[165] Vijay Mann, Akanksha Gupta, Partha Dutta, Anilkumar Vishnoi, Parantapa Bhattacharya, Rishabh Poddar, and Aakash Iyer. Remedy: Network-aware steady state vm management for data centers. In Proceedings of the 11th International IFIP TC 6 Conference on Networking, vol. Part I, IFIP' 12, pp. 190-204. Springer Verlag, 2012.
[166] Daniel Marques, Greg Bronevetsky, Rohit Fernandes, Keshav Pingali, and Paul Stodghil. Optimizing checkpoint sizes in the c3 system. In the 19th IEEE International Parallel and Distributed Processing Symposium, 2005.
[167] Ali Mashtizadeh, Emre' Celebi, Tal Garfinkel, and Min Cai. The design and evolution of live storage migration in vmware esx. In Proceedings of the 2011 USENIX Conference on USENIX Annual Technical Conference, USENIXATC'11, pp. 14-14, 2011.
[168] John D. McCalpin. Memory bandwidth and machine balance in current high performance computers. IEEE Computer Society Technical Committee on Computer Architecture (TCCA) Newsletter, pp. 19-25, Dec. 1995.
[169] Richard McDougall and Jim Maura. Solaris internals. In Rachael Borden, 2001.
[170] Nick Mckeown, Tom Anderson, Hari Balakrishnan, Guru Parulkar, Larry Peterson, Jennifer Rexford, Scott Shenker, and Jonathan Turner. Openflow: Enabling innovation in campus networks. SIGCOMM Comput. Commun. Rev., 38(2):69-74, Mar. 2008.
[171] Micro Checkpointing. wiki.qemu.org/features/microcheckpointing.
[172] Microsoft Corp. Hyper-v server 2008 r2. www.microsoft.com/ hyper-v-server/en/us/overview.aspx.
[173] R. Miller. Failure rates in google data centers www.datacenterknowledge.com/archives/2008/05/30/ failure-rates-in-google-data-centers, 2008.
[174] Dejan S. Milo'ic^ic', Fred Douglis, Yves Paindaveine, Richard Wheeler, and Songnian Zhou. Process migration. ACM Comput. Surv., 32(3):241-299, Sep. 2000.
[175] David Mosberger and Tai Jin. Httperf—a tool for measuring web server performance. SIGMETRICS Perform. Eval. Rev., 26(3):31-37, Dec. 1998.
[176] MOSIX. www.mosix.org.
[177] S. J. Mullender, G. van Rossum, A. S. Tananbaum, R. van Renesse, and H. van Staveren. Amoeba: a distributed operating system for the 1990s. Computer, 23(5):44-53, May 1990.
[178] Arun Babu Nagarajan, Frank Mueller, Christian Engelmann, and Stephen L. Scott. Proactive Fault Tolerance for HPC with Xen Virtualization. In Proceedings of the 21st Annual International Conference on Supercomputing, pp. 23-32, 2007.
[179] Arun Babu Nagarajan, Frank Mueller, Christian Engelmann, and Stephen L. Scott. Proactive fault tolerance for hpc with xen virtualization. In the 21st Annual International Conference on Supercomputing, pp. 23-32, 2007.
[180] Iperf—the TCP/UDP bandwidth measurement tool. dast.nlanr.net/ Projects/Iperf/.
[181] S. Al-Haj and E. Al-Shaer. A formal approach for virtual machine migration planning. In Proceedings of the 9th International Conference on Network and Service Management (CNSM 2013), pp. 51-58, 2013.
[182] Samer Al-Kiswany, Dinesh Subhraveti, Prasenjit Sarkar, and Matei Ripeanu. Vm-flock: Virtual machine co-migration for the cloud. In Proceedings of the 20th International Symposium on High Performance Distributed Computing, HPDC '11, pp. 159-170, 2011.
[183] Amazon Corporation. Amazon Elastic Compute Cloud (EC2), aws.amazon.com/ec2.
[184] Amazon Inc. AWS Server Migration Service. docs.aws.amazon. com/server-migration-service/latest/userguide/server-migration-ug.pdf.
[185] I. Banerjee, P. Moltmann, K. Tati, and R. Venkatasubramanian. VMware ESX Memory Resource Management: Swap. In VMWare Technical Journal, 2014.
[186] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. Xen and the art of virtualization. SIGOPS Operating Systems Review, 37(5):164-177, 2003.
[187] Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, and Andrew Warfield. Xen and the art of virtualization. In Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles, SOSP '03, 2003.
[188] M. F. Bari, M. F. Zhani, Q. Zhang, R. Ahmed, and R. Boutaba. Cqncr: Optimal vm migration planning in cloud data centers. In 2014 IFIP Networking Conference, pp. 1-9, Jun. 2014.
[189] F. Bellard. Qemu, a fast and portable dynamic translator. In In ATEC 05: Proceedings of the annual conference on USENIX Annual Technical Conference, 2005.
[190] SPEC CPU 2017 benchmark suite. www.spec.org/cpu2017/.

Related Publications (1)

	Number	Date	Country
	20220382574 A1	Dec 2022	US

Provisional Applications (1)

	Number	Date	Country
	63017320	Apr 2020	US

Continuations (1)

	Number	Date	Country
Parent	17242508	Apr 2021	US
Child	17818234		US

Recovering a virtual machine after failure of post-copy live migration

Information

Patent Number

Date Filed

Date Issued

Inventors

Original Assignees

Examiners

Agents

CPC

Field of Search

CPC

International Classifications

Abstract