Survey on Mechanisms for Live Virtual Machine Migration and Its Improvements

Information and Media Technologies 11: 101-115 (2016) reprinted from: Computer Software 33(2): 101-115 (2016) © Japan Society for Software Science and Technology

特集●サーベイ論文解説

Survey on Mechanisms for Live Virtual Machine Migration and its Improvements

Hiroshi Yamada

Live virtual machine (VM) migration (simply live migration) is a powerful tool for managing data center resources. Live migration moves a running VM between diﬀerent physical machines without losing any states such as network conditions and CPU status. Live migration has attracted the attention of academic and industrial researchers since replacing running VMs inside data centers by live migration makes it easier to manage data center resources. This paper summarizes live migration basics and techniques for improving them. Speciﬁcally, this survey focuses on software mechanisms for realizing basic live migration, improving its performance, and expanding its applicability. Also, this paper shows research opportunities that the state-of-the-art live migration techniques have not covered yet.

tualization software such as Xen [5],KVM[31],Vir- 1 Introduction tualBox [46], VMware ESXi [55], and Hyper-V [41] One of the innovative technologies in our com- is widely available. puter systems for the last decade is system virtual- Live VM migration (simply live migration) is a ization, which allows us to run multiple operating powerful tool for managing data center resources. systems (OSes) on a physical machine. In system Live migration moves a running VM between dif- virtualization, the virtual machine monitor (VMM) ferent physical machines without losing any states is a primary software layer that directly manages such as network connections and CPU status. Re- the underlying hardware, instead of the OS. The placing running VMs inside the data centers by live VMM provides virtual machines (VMs) on which migration makes it easier to manage data center OSes are running as if they are running on physi- resources. For example, the availability of services cal machines. System virtualization brings several can be improved by migrating less loaded VMs to benefits. For example, we can reduce the number of another host to assign resources to more loaded running physical machines by consolidating the VM VMs. Another typical example is to support phys- running server software into one physical machine. ical machine maintenance; a physical machine can This leads to improvements in physical resource uti- be maintained with much less service downtime by lization and reduction in power consumption. An migrating all the VMs running on the target ma- AFCOM survey [1] reports that 72.9% of data cen- chine to other machines. Policies for VM replace- ters in the world are virtualized in 2010. Also, vir- ment using live migration including load balanc- ing [21][57][62] and power saving [19][40][54][58] have 仮想マシンライブマイグレーション機構およびその効率化 been widely studied in research communities. に関するサーベイ Exploring ways to design and implement effective Hiroshi Yamada, 東京農工大学工学部情報工学科, Dept. and/or efficient live migration is still a hot topic in of Information and Computer Sciences, Tokyo Uni- the system research community. This paper de- versity of Agriculture and Technology. scribes a survey on live migration basics and tech- コンピュータソフトウェア ,Vol.33, No.2 (2016),pp.101–115. niques for improving them. Specifically, this survey [解説論文] 2015 年 10 月 2 日受付.

101 Information and Media Technologies 11: 101-115 (2016) reprinted from: Computer Software 33(2): 101-115 (2016) © Japan Society for Software Science and Technology focuses on software mechanisms for realizing ba- In virtualized environments, an OS is running on sic live migration, improving its performance, and a VM created by the VMM. We refer to an OS run- expanding its applicability. We believe that this ning on the VM as the guest OS. The VMM pro- survey helps researchers learn about existing live vides an illusion that the guest OSes are running migration techniques, helps administrators judge as if they are running on the physical hardware. which live migration technique is suitable for their The VMM multiplexes the underlying hardware to services and data centers, and sheds light on the create virtual hardware such as virtual CPUs and research directions of live migration. virtual devices. The VMM assigns part of the un- Numerous researches have implicitly assumed derlying hardware to VMs running on top of it. In that live migration is done on a local area network addition, the VMM achieves isolation between run- (LAN); the source and destination are connected in ning VMs; even if a guest OS crashes or is hijacked, the same network. The focus of this paper is live the other guest OSes are not affected. migration mechanisms that are used in intra data The VMM runs in the privileged mode to manage centers. Some efforts extend live migration to apply and multiplex the underlying physical hardware de- it to wide area networks [9][23][38][66]. Surveying vices whereas guest OSes run in the non-privileged these techniques is out of the scope of this paper. mode. When a guest OS executes a privileged in- We also note that exploring VM replacement poli- struction, such as access to MMU or I/O peripher- cies [19][21][40][54][57][58][62] is an important topic als, software interrupts occur, and control is trans- of live migration research, but this paper focuses ferred to the VMM. At this point, the VMM can on the software mechanisms for realizing live mi- capture and regulate all resources because it pro- gration. cesses the interrupts before delivering them to the The contributions of this paper are as follows: guest OS. • We describe mechanisms of live migration and its improvements. Note that the previous sur- 2. 2 Live Migration vey of live migration [39] describes basic mech- The goal of live migration is to move a running anisms of live migration; our survey is different VM between physical machines without disrupting from it. The previous survey mainly focuses on its services. To achieve this goal, minimizing the the difference between process migration and downtime during which the VM is stopped is im- VM migration while our main focus is on the portant, and is the most different point from a sus- difference between live migration mechanisms pend & resume scheme that stops a VM, extracts (Sec.3,4,and5). its memory image, and restores it on the destina- • We classify live migration mechanisms into tion machine. The details of algorithms to mini- two categories: performance and applicability. mize downtime in moving the VM are described in We discuss the state-of-the-art live migration Sec. 3. mechanisms in terms of the two aspects (Sec. 4 To move a running VM from a physical machine and 5). to another one, the live migration mechanism run- • We compare the mechanisms with each other ning inside the VMM transfers the target VM hard- and show some research directions of live mi- ware states to the destination machine. At the des- gration (Sec. 6). tination machine, the VMM builds the target VM from the received states and runs it after the state 2 Background restoration. The live migration mechanism typically transfers memory contents, CPU states, CPU 2. 1 Virtualization register values, and device states. Live migration System virtualization is commonplace in comput- is supported by open-source virtualization software ing environments including high-end data centers, such as Xen [5] and KVM [31]. laptops, and embedded systems. To support sys- The concept of migration is not new. In the tem virtualization, CPU vendors offer CPU exten- system research community, process-level migration sions for virtualization. Typical examples are Intel- approaches have been studied widely [42].Com- VTx [27], AMD SVM [3], and ARM TrustZone [4]. pared to this approach, VM-level migration has the

102 Information and Media Technologies 11: 101-115 (2016) reprinted from: Computer Software 33(2): 101-115 (2016) © Japan Society for Software Science and Technology following advantages, as described in [11]． Sec. 1. • VM states on source are eliminated completely: The narrow interface between 3. 1 Pre-copy Approach a guest OS and the VMM makes it easy to Pre-copy [11] is a widely used approach to trans- avoid the problem of residual dependencies in fer VM resources. The basic idea of pre-copy is to which the source machine must remain avail- iteratively copy from the source to the destination able and network-accessible to service certain the VM’s pages that have been dirtied during live system calls or even memory accesses on behalf migration execution. This idea is used by other sys- of migrated processes. Avoiding this problem tems such as VM fault-tolerant systems [13][43][49]. is valuable when we conduct live migration for Figure 1 shows the execution flow of the pre-copy maintenance of the source machine. migration. The pre-copy consists of two phases: • Entire VM memory can be migrated: push phase and stop-and-copy phase. When the Live migration transfers all of the in-memory live migration starts, it first starts the push phase. state of the VM in a consistent and effi- The VMM copies all pages of a VM from the source cient fashion. This applies to the kernel- to the destination during the first iteration. Subse- internal state (e.g. the TCP control block quent iterations only copy those pages dirtied dur- for a currently active connection) as well as ing the previous iteration. To detect dirty pages, the application-level state, even when this is the VMM maintains the dirty bitmap that describes shared between multiple cooperating processes. which page becomes dirtied. When the number of In practical terms, for example, this means dirty pages is under a predefined threshold, the pre- that we can migrate an on-line game server copy live migration starts the stop & copy phase. or streaming media server without requiring In the stop & copy phase, the VMM suspends the clients to reconnect. VM and then transfers any remaining pages and • Guest OS involvements are not needed: processor states such as register values. After that, VM-level live migration allows a separation of the original VM on the source is discarded and the concerns between the users and operator of a VM on the destination is resumed. data center or cluster. We do not need to pro- To keep the same IP address of the VM across vide the data center operator with any OS-level the migration, the VMM advertises that the IP has access at all (e.g. a root login to quiesce pro- moved to a new location by generating an unso- cesses or I/O prior to migration). Similarly licited ARP reply from the destination machine. the operator need not be concerned with the This will reconfigure the same machines to send details of what is occurring within the VM. packets to the new physical address, and the VM Live migration is a powerful tool and attracts connections are going with almost no observable not only system community but also database one. interference. In fact, some research literatures report efficient The paper [11] also describes some optimization database live migration schemes [6][14][16][44]. techniques for the pre-copy live migration. An example is to transfer only pages dirtied in the previ- 3 Basic Strategies ous iteration that have not been dirtied again. This is based on the observation that rapidly modified This section describes the three strategies to real- pages are very likely to be dirtied again. Another ize live migration: pre-copy, post-copy,andhybrid. example is to throttle network bandwidth usage for Typical live migration mechanisms, supported in copying memory pages to avoid network contention Xen and KVM, transfer three hardware states: with the other traffic including service-related net- memory, CPU, and NIC. These mechanisms do works. not transfer block device states. The assumption The paper [11] introduces Self-Migration pre- behind them is that the source and destination copy live migration that transfers VM states by the machines connect to a networked shared storage, guest OS itself, not the VMM. The self-migration which is reasonable since the target of live mi- mechanism first executes a stub on the destination gration is in a LAN environment, as described in to buffer transferred states. And then, the mecha-

103 Information and Media Technologies 11: 101-115 (2016) reprinted from: Computer Software 33(2): 101-115 (2016) © Japan Society for Software Science and Technology

… …

(1) First Iteration (Push Phase) … …

(2) Second/later Iterations (Push Phase)

… …

(3) Stop & Copy Phase

Fig. 1 Pre-copy Strategy. nism starts the push phase in which the guest OS fetches the VM’s pages from the source. Specif- sets all pages to be write-protected and iteratively ically, the post-copy mechanism transfers CPU transfers dirtied pages by managing its own dirty states and then starts the target VM on the desti- bitmap. In the stop-and-copy phase, the guest OS nation. When the VM accesses a memory page that disables all its activities except for the migration is not in the destination, the post-copy mechanism and then performs a final scan of the dirty bitmap, handles the page fault and fetches the page from clearing the appropriate bit as each page is trans- the source. Also, the post-copy mechanism simul- ferred. Any pages that are dirtied during the final taneously transfers the VM’s pages from the source scan and are still marked as dirty in the bitmap are to the destination. The original implementation is copied to a buffer. The second and final stage then based on Xen, but this approach is integrated into transfers the contents of the buffer. KVM [24]. Also, the post-copy mechanism [22] offers a func- 3. 2 Post-copy Approach tion to reduce the number of pages to be trans- A major drawback of pre-copy live migration ferred. It avoids transferring unused pages with is that the push phase becomes much longer as the balloon driver [56]. The balloon driver is an numerous pages become dirtied in each iteration. OS kernel-level module that reclaims unused pages Post-copy live migration [22] overcomesthisdraw- of the guest OS and returns them to the VMM. back. Its execution flow is shown in Fig. 2. The It pins down pages inside the guest OS and noti- post-copy first starts the migrating VM on the des- fies the VMM of their page frame numbers. The tination by transferring the minimum number of post-copy mechanism inflates the balloon driver to states that is enough to start the VM and then reclaim unused pages on the source and skips trans-

104 Information and Media Technologies 11: 101-115 (2016) reprinted from: Computer Software 33(2): 101-115 (2016) © Japan Society for Software Science and Technology

… …

(1) Stop & Copy Phase ! # " " … …

(2) Memory Fetch

… …

(3) Final Phase

Fig. 2 Post-copy Strategy. ferring the reclaimed pages. On the destination, it source, which happens on the post-copy. deflates the balloon driver to reassign the unused pages. 3. 4 Comparison of Three Strategies The post-copy approach inherently suffers from Although total memory transfers of the post-copy performance degradation when there is a large andHGS-LMtendtobesmallerthanthatofthe working set workload. Since bursty page faults are pre-copy, the pre-copy is still widely used due to its caused by accessing numerous pages just after the reliability. When a network connection between the VM starts on the destination, the system frequently source and destination is down during live migra- waits for its page fetch completion. tion, the VM crashes in the post-copy and HGS- LM. In the pre-copy, the live migration fails but 3. 3 Hybrid Approach theVMcanstillrunonthesource. HGS-LM [36] isahybridapproachofbothpre- Table 1 summarizes the live migration strategies copy and post-copy. Figure 3 shows the execution described above and relatively evaluates them. We flow of HGS-LM. HGS-LM first transfers all pages review them in terms of three aspects: migration to the destination, like the pre-copy’s first iteration. time, memory transfer, and reliability. Migration And, it starts the target VM on the destination af- time is the time during which the live migration is ter copying its CPU states and fetches VM pages being executed. Pre-copy is the most reliable since from the source in the same way as the post-copy. the VM can service while the network fails during By doing so, HGS-LM avoids severe performance the migration. But its migration time and mem- degradation caused by bursty page fetches from the ory transfers are much worse due to a workload

105 Information and Media Technologies 11: 101-115 (2016) reprinted from: Computer Software 33(2): 101-115 (2016) © Japan Society for Software Science and Technology

… …

(1) All Memory Copy " !!

" $ $ " ! … … !

! # ! ! # !! !! !$! "! % $& (2) Stop & Copy Phase "!# $ ! ! … …

(3) Memory Fetch

Fig. 3 HGS-LM.

Table 1 Comparison of three live migration strategies Migration Total memory Migration Strategy Reliability Description time transfers Migration time and memory transfer Pre-copy [11] × × become worse as dirty page rate of workload is higher. VM performance is severely degraded Post-copy [22] × by bursty page faults caused by large working set workload. Performance penalty just after the Hybrid [36] × VM starts on destination can be less. whose dirty page rate is high. Post-copy’s migra- strategy performs more memory transfers than the tion time and memory transfer are the best out of post-copy one, but it mitigates such page fetches all the strategies but the reliability is worse than by transferring all memory at the ﬁrst step. that of the pre-copy. In addition, the performance penalty after the control is transferred to the des- 4 Performance Improvements tination can be high when the large working set Numerous researches to improve the live migra- workload runs; the workload causes bursty page tion have been conducted so far. We classify the faults and page fetches from the source. The hybrid existing eﬀorts into two categories: performance

106 Information and Media Technologies 11: 101-115 (2016) reprinted from: Computer Software 33(2): 101-115 (2016) © Japan Society for Software Science and Technology and applicability. Techniques categorized as per- Naoi and Yamada present case studies of ac- formance address issues of how we make live mi- celerating compression-based live migration with gration execution more efficient. Ones categorized GPGPU [45]. They apply GPGPU programming into the applicability address issues of how live mi- techniques such as pipelining and sampling sleeps gration is applicable to variously configured VMs. to live migration to asynchronously execute CPU This section describes techniques for improving the and GPU. The execution of the live migration con- performance of live migration. sists of three stages: page fetching, page compres- We note that most of the efforts that have been sion on GPU, and page transfer. These are exe- conducted so far are based on the pre-copy strat- cuted in a pipelined manner. egy. If a proposed method is designed for the other strategies, we clearly describe it. 4. 2 Memory Deduplication This technique deduplicates the same contents 4. 1 Memory Compression in pages to reduce network traffic. The dedu- Live migration transfers at least all pages of plication is used by several live migration ap- the VM. Since the current trend is that as VM proaches [10][12][28][48][60][64][65]. For example, the memory size becomes bigger and bigger, migration mechanism of CloudNet [60] splits each memory time becomes much longer and more data trans- page into fixed sized chunks and generates a hash fers are required. To address this issue, some table based on their contents. If a hash matches techniques use data compression and decompres- an entry in FIFO caches maintained at the source sion [10][12][20][28][48][52][60][64]. These techniques and destination, a block with the same contents compress VM’s page contents on the source and was sent previously. After verifying that the pages decompresses the compressed data on the destina- match, the mechanism sends a 32 bit index to the tion. By doing so, we can reduce the amount of the cache entry instead of the full chunk. network messages and shorten the migration time. Jo et al. [29] deduplicate the same contents be- KVM uses a simple technique that compresses and tween in the buffer cache and in the disk. Their in- decompresses zero pages [31]. sight is that the guest OS can reconstruct the buffer The mechanisms perform content-based compres- cache contents since the VM on the source and des- sion on all the VM memory. They typically com- tination host uses the same shared disk. This dedu- press the VM memory with run length encoding plication mechanism first replaces the buffer cache (RLE), which is a simple but powerful compression contents with the sector number whose sector con- algorithm. RLE compresses repeated ‘runs’ of the tains the original buffer cache contents. The kernel same number by storing the length of that run. For mechanism restores the buffer caches by fetching example, RLE changes “00001” into “041”. the memorized sectors from the shared disk just Delta compression [20][52] is relatively effective after the VM starts on the destination. for the second and later iteration where updated pages in the previous phase are transferred. Delta 4. 3 Leveraging Log & Replay compression only transfers changes of memory To reduce network traffic for the live migration, pages instead of the full page contents, based on the CR/TR-Motion [35] transfers an execution log in- fact that the updated part in a dirty page is typ- stead of the VM memory and replays the log on ically small. Specifically, the technique computes the destination to create the same state of the tar- the delta page by applying XOR to the current get VM. It first extracts the memory image of the and previous version of a page. To compose the target VM and transfers it to the destination. And new page at the destination, this process is sim- then, the VMM-level mechanism logs the execution ply reversed to produce a copy of the source’s cur- of the VM and iteratively transfers the log to the rent version from the previous version and the delta destination. The logging mechanism is based on the page. The mechanisms also compress the delta to log/replay system [15], which records and replays reduce the amount of transferred data as much as all non-deterministic events that can affect the ex- possible. The papers reveal that the delta pages ecution of the VM, such as virtual interrupts. The can be efficiently compressed by binary RLE. replay mechanism on the destination replays the

107 Information and Media Technologies 11: 101-115 (2016) reprinted from: Computer Software 33(2): 101-115 (2016) © Japan Society for Software Science and Technology

VM execution from the transferred log. CR/TR- contents pages of the VM exist on the destina- Motion starts the stop-and-copy phase when the tion. When the same contents pages are detected, log size is smaller than the predefined size. the mechanism does not transfer the correspond- ing pages on the source and assigns the detected 4. 4 Parallelizing Migration pages on the destination in a copy-on-write manner. PMigrate [51] shortens the migration time by Miyakodori [2] reuses memory pages on the desti- executing the live migration in parallel on mod- nation on which the VM ran before. Miyakodori ern multi-core platforms. PMigrate leverages both preservesasmanymemorypagesaspossibleonthe data and pipeline parallelism to parallelize live mi- source after migrating the VM. When the VM is gration. For example, VM page fetch and page migrated to the source, Miyakodori reassigns pre- transfer are executed in a pipelined manner. In served pages again if they have not been updated. addition, PMigrate introduces the range lock to en- hance the parallelism of live migration. Range lock 4. 7 Application Assists is a kernel-level function that enables threads to Hou et al. present a live migration technique that fetch different parts of the VM memory in paral- cooperates with application-level memory manage- lel, which is different from the default single lock ment [25]. JAVMMselectsmemorypagestobe offered by mmap(). transferred with information on the JVM’s memory object management. Specifically, JVAVMM 4. 5 Migration Noise Avoidance skips the memory region where young generation The research literature [32][33] shows that live objects reside during the push phase and sends migration has a side-effect named migration noise. its pages to the destination at the stop-and-copy Live migration significantly interferes with not only phase. This is based on the JVM’s garbage col- migrating VM but also collocating VM perfor- lection management knowledge that young objects mance because the resource consumption, such as tend to be free soon, which means that the memory CPU and network, for live migration causes re- region for young objects is updated frequently. In source contention between the migration process JAVMM, the JVM notifies the VMM of the guest and VMs. physical address of the memory region for young SonicMigration [33] mitigates migration noise by objects with the kernel-level mechanism, and the making live migration lightweight. It significantly VMM does not transfer the pages in the region un- reduces the number of VM pages to be trans- til the stop-and-copy phase. ferred, shortening the migration time and resource usage. Specifically, SonicMigration transfers only 4. 8 Other Optimizations hard state pages that are for the VM execution; it Some approaches improve live migration perfor- avoids transferring soft state pages such as buffer mance in a different way than described so far. VM cache pages and free pages. By doing so, SonicMi- Scatter [12] efficiently migrates multiple VMs at the graion reduces opportunities to occur resource con- same time with deduplication and multicast fea- tentionwiththeotherVMs. tures. It extracts the same pages of the target VMs iAware [63] is a strategy that minimizes migra- and multicasts them to the destinations. tion noise during the VM replacement. It models Jettison [8] uses a technique named partial migra- resource usage for live migration and executes live tion, which transfers a part of the VM memory and migration with as little resource contention as pos- control to the destination. Jettison transfers only sible, based on the model. memory pages of the working set and then starts the VM on the destination. The rest pages remain 4. 6 Destination Page Reuse in the source. If the VM accesses a page that is Some approaches make use of the same contents not on the destination, Jettison handles the page pages of the target VM on the destination to skip fault and fetches the page from the source. Jetti- their transfers. Memory Buddies [61] detects the son is designed for reducing the power consumption same contents pages of the VM on the destina- in offices by consolidating idle PCs into a physical tion. Its mechanism first checks whether the same server.

108 Information and Media Technologies 11: 101-115 (2016) reprinted from: Computer Software 33(2): 101-115 (2016) © Japan Society for Software Science and Technology

tion. Taking a snapshot of a virtual disk produces 5 Applicability Improvements a new snapshot file that preserves writes to the disk 5. 1 Managing I/O Devices after taking it while the original virtual disk file re- Some approaches allows us to migrate a VM that mains. Snapshotting sends the original virtual disk is attached to pass-through devices. Nomad [26] file to the destination. After the transfer of the offers a mechanism for migrating a VM attached original virtual disk is completed, the snapshotting to pass-through Infiniband. In Nomad, the device mechanism takes a snapshot of the virtual disk on driver-level module on the source wakes up the In- the source. And then, it sends the previous snap- finiband device, suspends all the connections, and shot file and merges it with the original virtual disk sends the client status. After starting the VM on file on the destination. This process is repeated un- the destination, the attached Infiniband device re- til the snapshot file size is smaller than the prede- connects to the client to restart the network com- fined threshold. When the snapshot file size is small munication. enough, the mechanism suspends the VM, switches CompSC [47] migrates a VM attached to a NIC to the virtual disk on the destination, and resumes virtualized by using the the single root I/O virtu- it. alization feature. CompSC’s kernel module records The second technique, named dirty block track- instructions that change the NIC state. It trans- ing, is similar to the pre-copy-based memory transfers the record log and register values of the NIC fer. The technique first copies the target virtual when the VM migration starts. The kernel mod- disk to the destination. It next sends updated ule builds the same states of the NIC using the blocks during the virtual disk copy and merges transferred information after the VM starts on the them with the virtual disk on the destination. It it- destination. Kadav et al. present a technique for eratively copies and merges updated blocks during restoring the same state of the NIC on the des- the previous data block transfer until the updated tination with the shadow driver [30].Theshadow block number is under the threshold. If it is un- driver [53] intermediates between the OS kernel and der the threshold, the technique suspends the VM, a device driver and thus can monitor all their inter- sends the updated blocks, and resumes the VM af- actions. The technique is similar to the CompSC’s ter merging the updated blocks to the virtual disk. one. The shadow driver records the kernel requests The third technique is I/O mirroring, which that change the device driver. The shadow driver transfers the virtual disk and updated blocks in reloads the device driver and replays the recorded parallel. The VM accesses the original virtual disk requests on the destination during the stop-and- file on the source during the migration. When the copy phase. VM updates a block sent to the destination, the LoGV [18] enables us to migrate a VM attached I/O mirroring mechanism propagates the update to a virtual GPU. LoGV stops the GPU accesses to both the source and destination. When the VM of the VM just after the push phase. When the updates a block that is being copied to the desti- physical GPU becomes idle, LoGV transfers the nation, the update propagation is delayed until the contents of the GPU memory and then executes block transfer completes. The mechanism switches the stop-and-copy phase. It permits the VM to ac- the source to the destination after copying the vir- cess the virtual GPU after starting the VM on the tual disk file. destination. Zhou et al. present techniques for migrating virtual storages between SSDs [67]. They take the 5. 2 Storage Migration I/O penalty and SSD lifetime into account in vir- Several work focuses on migration of a virtual tual storage migration. These techniques differ- disk between different disk volumes. Mashtizahe et ently handle VM’s disk I/O requests while copy- al. summarize the characteristics of three typical ing data blocks. An example is low redundancy storage migration techniques [37]. First is a tech- (LR), which aims at avoiding redundant writes. LR nique called snapshotting, which makes use of vir- redirects disk I/O of the guest VM to the destina- tual disk snapshots to move the disk to the destination except for disk reads to not-yet-copied blocks. Another example is source-based, low redundancy

109 Information and Media Technologies 11: 101-115 (2016) reprinted from: Computer Software 33(2): 101-115 (2016) © Japan Society for Software Science and Technology

Table 2 Mechanisms for Improving Live Migration (a) Performance Migration Network CPU Description Time Traﬃc usage Compression/decompression × Compression and decompression [10][12][20][28][48][52][60][64] of pages consume CPU time. Deduplication × Deduplication of memory [10][12][28][29][48][60][64] contents consumes CPU time. Replaying log Log&replay [35] × consumes CPU time. Parallelization [51] − − Resource utilization is bursty. Pruning soft-state Losing buﬀer caches kernel objects [33] degrades VM’s performance. Destination pages reuse − − [2][61] Application assists [25] − − Packet multicasts [12] − − Partial migration [8] Entire VM is not migrated.

(b) Applicability Target Resource Nomad [26] Pass-throughed Inﬁniband CompSC [47], Kadav et al. [30] Pass-throughed SR-IOV NIC Snapshotting, Dirty block tracking, Virtual disks I/O mirroring [37] LR and SLR [67] Virtual disks LoGV [18] Para-virtualized GPUs LIME [17] Virtual private network

(SLR), which aims to maximize the performance of theVMbyissuingasmanydiskrequestsaspos- 6 Summary sible to the source. SLR passes VM’s disk I/O to 6. 1 Classification the source and tracks updated blocks to retransfer Table 2 and 3 highlight mechanisms for improv- them to the destination. ing live migration, described in Sec. 4, and 5. Ta- ble 2 exhibits migration mechanisms and their ben- 5. 3 Software-defined Network efits. Table 2-(a) lists techniques categorized as the The software-defined network is widely used in performance and shows which aspects of live migra- data centers to manage its network. Virtual tion execution they contribute to and deteriorate switches can assign the VMs of each tenant its own compared to the default live migration. All the virtual private network, which is built on the lo- techniques contribute to shortening migration time, cal area network on the data center. LIME [17] and some techniques do so at the expense of more allows us to migrate a VM, keeping the virtual pri- resource utilization. For example, the compres- vate network state. To minimize packet loss dur- sion/decompression and deduplication utilize more ing migration, LIME copies the state of the virtual CPU usage to execute their tasks such as checking switch connected to the target VM to the destina- memory contents and compressing/deduplicating tion. After the virtual switch copying, LIME starts the pages. The log & replay requires more CPU migrating the VM. When the VM starts on the des- time to replay the execution log. Parallelization can tination, LIME tunnels packets to the destination shorten migration time without more resources but to redirect them to the VM if necessary. the resource usage becomes bursty since it is done in parallel. Techniques for partial migration and

110 Information and Media Technologies 11: 101-115 (2016) reprinted from: Computer Software 33(2): 101-115 (2016) © Japan Society for Software Science and Technology

Table 3 Requirements for software stack modiﬁcation

VMM OS Kernel Application Compression/decompression − − [10][12][20][28][48][52][60][64] Deduplication − − [10][12][28][29][48][60][64] Log&replay [35] − − Parallelization [51] − Pruning soft-state kernel objects [33] − Destination pages reuse [2][61] − − Application assists [25] Packet multicasts [12] − Partial migration [8] − − Nomad [26] CompSC [47], Kadav et al. [30] − LoGV [18] − LIME [17] − − pruning soft-state kernel objects look effective, but ties are summarized as follows. they have limitations including the severe perfor- • Live migration for high speed LAN: Ex- mance penalty and the untransferring of the entire isting migration techniques implicitly assume VM’s memory. The rest of the techniques (desti- that network bandwidth is 1 Gbps. In the re- nation page reuse, application assists, and packet cent data centers, machines are connected with multicasts) can contribute to live migration execu- higher speed networks including 10 Gbps and tion without special resource usage. 40 Gbps. Since we can transfer more memory Table 2-(b) lists techniques categorized in the ap- during migration at a time, bottlenecks of the plicability and shows their target resources. We migration mechanisms and the trade-offs be- overview that the existing efforts make live migra- tween CPU utilization and network utilization tion applicable to various resources. We believe will be different from the 1 Gbps network situa- that researches for expanding the target resources tion. Exploring mechanisms for live migration of the live migration continue to be conducted since suitable to the high speed networks contributes brand-new hardware such as accelerators including to more efficient data center management. FPGA and non-volatile memory is becoming pop- • Live migration of nested VMM: The ular. nested VMM, which is a VMM running on Table 3 summarizes software stack modification the VMM, is an emerging application for sys- required by each technique. This point is important tem virtualization. The mechanisms to real- for administrators to judge which techniques are ize the nested VMM have been studied [7][59], usable in their own data centers. The techniques and some systems leverage the nested VMM categorized in the performance tend to be applica- feature [34][50]. Since the mechanisms for the ble with only VMM modification, while most of the nested VMM are tightly coupled with the un- techniques categorized in the applicability require derlying VMM, the existing migration mecha- not only the VMM- but also kernel-level supports. nisms are difficult to use to migrate the nested Application assist and Nomad require all the layers’ VMM. Designing and implementing mecha- supports to enjoy their benefits. nisms to enable us to migrate nested VMMs is one of the important topics. 6. 2 Research Opportunities • Live migration of VM attached to pass- From the survey on the existing live migration throughed accelerators: There is increas- techniques, we reveal research issues that these ing attention to accelerators such as FPGA, techniques do not cover yet. Research opportuni- Xeon-Phi, and GPUs. One of their unique fea-

111 Information and Media Technologies 11: 101-115 (2016) reprinted from: Computer Software 33(2): 101-115 (2016) © Japan Society for Software Science and Technology

tures compared to other devices is that the ac- [ 8 ] Bila, N., de Lara, E., Joshi, K., Lagar-Cavilla, celerators have complicated states and are not H. A., Hiltunen, M. and Satyanarayanan, M.: Jetti- easy to restore to the same state. The targets son: Efficient Idle Desktop Consolidation with Par- tial Migration, in Proc. of the 7th ACM European of the existing efforts are relatively simple de- Conference on Computer Systems (EuroSys ’12), vices and thus new mechanisms are required 2012, pp. 211–224. to realize live migration of the VMs with the [ 9 ] Bradford, R., Kotsovinos, E., Feldmann, A. and pass-throughed modern accelerators. Schiöberg, H.: Live Wide-Area Migration of Vir- tual Machines Including Local Persistent State, in Proc. of the 3rd ACM International Conference on 7 Conclusion Virtual Execution Environments (VEE ’07), 2007, This paper surveyed the state-of-the-art live mi- pp. 169–179. [10] Chiang, J.-H., Li, H.-L. and cker Chiueh, T.: gration techniques. Specifically, we summarized Introspection-based Memory De-duplication and basic live migration strategies and techniques with Migration, in Proc. of the 9th ACM International a view to improvement. We compared these tech- Conference on Virtual Execution Environments niques in terms of their benefits by classifying the (VEE ’13), 2013, pp. 51–62. techniques into two categories: performance and [11] Clark, C., Fraser, K., Hand, S., Hansen, J. G., Jul, E., Limpach, C., Pratt, I. and Warfield, A.: applicability. This paper also describes research Live Migration of Virtual Machines, in Proc. of opportunities to evolve live migration for more ef- the 2nd USENIX Symposium on Networked Sys- fective and efficient data center administration. We tems Design and Implementation (NSDI ’05), 2005, believe that this survey is useful to cover the basic pp. 273–286. [12] Cui,L.,Li,J.,BoLi,J.H.,Hu,C.,Wo,T.,Al- techniques for live migration. Aqrabi, H. and Liu, L.: VMScatter: Migrate Virtual Machines to Many Hosts, in Proc. of the 9th ACM References International Conference on Virtual Execution En- [ 1 ] AFCOM: 2009/2010 AFCOM Data Center vironments (VEE ’13), 2013, pp. 63–72. Trends Survey Results & Analysis, 2010. [13] Cully, B., Lefebvre, G., Meyer, D., Feeley, [ 2 ] Akiyama, S., Hirofuchi, T., Takano, R. and M., Hutchinson, N. and Warfield, A.: Remus: Honiden, S.: Miyakodori: A Memory Reusing High Availability via Asynchronous Virtual Ma- Mechanism for Dynamic VM Consolidation, in chine Replication, in Proc. of the 5th USENIX Sym- Proc. of the 2012 IEEE 5th International Con- posium on Networked Systems Design and imple- ference on Cloud Computing (CLOUD ’12), 2012, mentation (NSDI ’08), 2008, pp. 161–174. pp. 606–613. [14] Das, S., Nishimura, S., Agrawal, D. and El Ab- [3] AMD:AMD64 Programmer’s Manual Volume2: badi, A.: Albatross: Lightweight Elasticity in System Programming, Sep. 2007, http://www.amd. Shared Storage Databases for the Cloud using Live com/us-en/assets/content type/white papers and Data Migration, Proc. of the VLDB Endowment, tech docs/24593.pdf. Vol. 4, No. 8(2011), pp. 494–505. [ 4 ] ARM: ARM TrustZone, http://www.arm.com/ [15] Dunlap, G. W., King, S. T., Cinar, S., Bas- products/processors/technologies/trustzone/. rai, M. A. and Chen, P. M.: Revirt: Enabling In- [ 5 ] Barham, P., Dragovic, B., Fraser, K., Hand, S., trusion Analysis Through Virtual-Machine Logging Harris, T., Ho, A., Neugebauer, R., Pratt, I. and and Replay, in Proc. of the 5th USENIX Symposium Warfield, A.: Xen and Art of Virtualization, in on Operating System Design and Implementation Proc. of the 19th ACM Symposium on Operating (OSDI ’02), 2002, pp. 211–224. Systems Principles (SOSP ’03), 2003, pp. 164–177. [16] Elmore, A. J., Das, S., Agrawal, D. and Abbadi, [ 6 ] Barker, S., Chi, Y., Hacigümüs, H., Shenoy, P. A. E.: Zephyr: Live Migration in Shared Nothing and Cecchet, E.: ShuttleDB: Database-Aware Elas- Databases for Elastic Cloud Platforms, in Proc. of ticity in the Cloud, in Proc. of the 11th Interna- the 2011 ACM International Conference on Man- tional Conference on Autonomic Computing (ICAC agement of Data (SIGMOD ’11), 2011, pp. 301–312. ’14), 2014, pp. 33–43. [17] Ghorbani, S., Schlesinger, C., Monaco, M., [ 7 ] Ben-Yehuda, M., Day, M. D., Dubitzky, Z., Keller, E., Caesar, M., Rexford, J. and Walker, D.: Factor, M., Har’El, N., Gordon, A., Liguori, A., Transparent, Live Migration of a Software-Defined Wasserman, O. and Yassour, B.-A.: The Turtles Network, in Proc. of the ACM Symposium on Cloud Project: Design and Implementation of Nested Vir- Computing (SoCC ’14), 2014, pp. 3:1–3:14. tualization, in Proc. of the 9th USENIX Symposium [18] Gottschlag, M., Hillenbrand, M., Kehne, J., on Operating Systems Design and Implementation Stoess, J. and Bellosa, F.: LoGV: Low-overhead (OSDI ’10), 2010, pp. 423–436. GPGPU Virtualization, in Proc. of the 4th Inter-

112 Information and Media Technologies 11: 101-115 (2016) reprinted from: Computer Software 33(2): 101-115 (2016) © Japan Society for Software Science and Technology

national Workshop on Frontiers of Heterogeneous pp. 41–50. Computing, IEEE, 2013, pp. 1721–1726. [30] Kadav, A. and Swift, M. M.: Live Migration of [19] Govindan, S., Wang, D., Sivasubramaniam, A. Direct-access Devices, in Proc. of the 1st Workshop and Urgaonkar, B.: Leveraging Stored Energy for on I/O Virtualization (WIOV ’08), 2008, pp. 7–13. Handling Power Emergencies in Aggressively Pro- [31] Kivity, A., Kamay, Y., Laor, D., Lublin, U. visioned Datacenters, in Proc. of the 17th ACM and Liguori, A.: kvm: the Linux Virtual Machine International Conference on Architectural Support Monitor, in Proc. of the Linux Symposium, 2007, for Programming Languages and Operating Systems pp. 225–230. (ASPLOS’12), 2012, pp. 75–86. [32] Koto, A., Kono, K. and Yamada, H.: A Guidline [20] Hacking, S. and Hudzia, B.: Improving the Live for Selecting Live Migration Policies and Implemen- Migration Process of Large Enterprise Applications, tations in Clouds, in Proc. of the 2014 IEEE 6th In- in Proc. of the 3rd International Workshop on Vir- ternational Conference on Cloud Computing Tech- tualization Technologies in Distributed Computing nology and Science (CloudCom14), 2014, pp. 226– (VTDC ’09), 2009, pp. 51–58. 233. [21] Hermenier, F., Lorca, X., Menaud, J.-M., [33] Koto, A., Yamada, H., Ohmura, K. and Kono, Muller, G. and Lawall, J.: Entropy: a Consolidation K.: Towards Unobtrusive VM Live Migration for Manager for Clusters, in Proc. of the 2009 ACM In- Cloud Computing Platforms, in Proc. of the 3rd ternational Conference on Virtual Execution Envi- ACM Asia-Pacific Conference on Systems (APSys ronments (VEE ’09), 2009, pp. 41–50. ’12), 2012, pp. 7:1–7:6. [22] Hines, M. R. and Gopalan, K.: Post-Copy Based [34] Kourai, K. and Ooba, H.: Zero-copy Migration Live Virtual Macine Migration Using Adaptie Pre- for Lightweight Software Rejuvenation of Virtual- Paging and Dynamic Self-Ballooning, in Proc.ofthe ized Systems, in Proc. of the 6th ACM Asia-Pacific 2009 ACM International Conference on Virtual Ex- Workshop on Systems (APSys ’15), 2015, pp. 7:1– ecution Environments (VEE’09), 2009, pp. 51–60. 7:8. [23] Hirofuchi, T., Nakada, H., Ogawa, H., Itoh, S. [35] Liu, H., Jin, H., Liao, X., Hu, L. and Yu, C.: and Sekiguchi, S.: A Live Storage Migration Mech- Live Migration of Virtual Machine Based on Full anism over WAN and its Performance Evaluation, System Trace and Replay, in Proc. of the 18th in Proc. of the 3rd International Workshop on Vir- ACM International Symposium on High Perfor- tualization Technologies in Distributed Computing mance Distributed Computing (HPDC ’09), 2009, (VTDC ’09), 2009, pp. 67–74. pp. 101–110. [24] Hirofuchi, T. and Yamahata, I.: Yabusame: [36] Lu, P., Barbalace, A. and Ravindran, B.: HSG- Postcopy Live Migration for Qemu/KVM, in KVM LM: Hybrid-Copy Speculative Guest OS Live Mi- Forum 2011, 2011. gration without Hypervisor, in Proc. of the 6th In- [25] Hou, K.-Y., Shin, K. and Sung, J.-L.: ternational Systems and Storage Conference (SYS- Application-assisted Live Migration of Virtual Ma- TOR ’13), 2013, pp. 2:1–2:11. chines with Java Applications, in Proc. of the 10th [37] Mashtizadeh, A., Emré, Garfinkel, T. and Cai, ACM SIGOPS European Conf. on Computer Sys- M.: The Design and Evolution of Live Storage tems (EuroSys ’15), 2015, pp. 15:1–15:15. Migration in VMware ESX, in Proc. of the 2011 [26] Huang, W., Liu, J., Koop, M., Abali, B. and USENIX Conference on USENIX Annual Technical Panda, D.: Nomad: Migrating OS-bypass Networks Conference (USENIX ATC’11), 2011, pp. 187–200. in Virtual Machines, in Proce. of the 3rd Interna- [38] Mashtizadeh, A. J., Cai, M., Tarasuk-Levin, G., tional Conference on Virtual Execution Environ- Koller, R., Garfinkel, T. and Setty, S.: XvMotion: ments (VEE ’07), 2007, pp. 158–168. Unified Virtual Machine Migration over Long Dis- [27] Intel Corporation: Intel64 and IA-32 Ar- tance, in Proc. of the 2014 USENIX Conference on chitechtures Software Developer’s Manual Vol- Annual Technical Conference (USENIX ATC ’14), ume 3B: System Programming Guide, Part 2, 2014, pp. 97–108. 2008, http://download.intel.com/design/processor/ [39] Medina, V. and Garc´ıa, J. M.: A Survey of manuals/25 3669.pdf. Migration Mechanisms of Virtual Machines, ACM [28] Jin, H., Deng, L., Wu, S., Shi, X. and Pan, Computing Survey, Vol. 46, No. 3(2014), pp. 30:1– X.: Live Virtual Machine Migration with Adaptive 30:33. Memory Compression, in Proc. of 2009 IEEE Inter- [40] Meng, X., Pappas, V. and Zhang, L.: Im- national Conference on Cluster Computing (Clus- proving the Scalability of Data Center Networks ter’09), 2009, pp. 1–10. with Traffic-aware Virtual Machine Placement, in [29] Jo, C., Gustafsson, E., Son, J. and Egger, Proc. of the 29th IEEE International Conference on B.: Efficient Live Migration of Virtual Machines Computer Communications (INFOCOM ’10), 2010, Using Shared Storage, in Proc. of the 9th ACM pp. 1–9. SIGPLAN/SIGOPS International Conference on [41] Microsoft: Windows Server Hyper-V, https:// Virtual Execution Environments (VEE ’13), 2013, www.microsoft.com/en-us/server-cloud/solutions/

113 Information and Media Technologies 11: 101-115 (2016) reprinted from: Computer Software 33(2): 101-115 (2016) © Japan Society for Software Science and Technology

virtualization.aspx. per: Power and Migration Cost Aware Application [42] Milo´jiˇcić, D. S., Douglis, F., Paindaveine, Y., Placement in Virtualized Systems, in Proc. of the Wheeler, R. and Zhou, S.: Process Migration, ACM 9th ACM/IFIP/USENIX International Conference Computer Survey, Vol. 32, No. 3(2000), pp. 241–299. on Middleware (Middleware ’08), 2008, pp. 243– [43] Minhas, U. F., Rajagopalan, S., Cully, B., 264. Aboulnaga, A., Salem, K. and Warfield, A.: Re- [55] VMware Inc.: VMware vSphere ESXi, http: musDB: Transparent High Availability for Database //http://www.vmware.com/products/vsphere. Systems, The International Journal on Very Large [56] Waldspurger, C. A.: Memory Resource Manage- Data Bases, Vol. 22, No. 1(2013), pp. 29–45. ment in VMware ESX Server, in Proc. of the 5th [44] Mishima, T. and Fujiwara, Y.: Madeus: USENIX Symposium on Operating System Design Database Live Migration Middleware Under Heavy and Implementation (OSDI ’02), 2002, pp. 181–194. Workload for Cloud Environment, in Proc. of the [57] Wang, H., Isci, C., Subramanian, L., Choi, J., 2015 ACM International Conference on Magement Qian, D. and Mutlu, O.: A-DRM: Architecture- of Data (SIGMOD ’15), 2015, pp. 315–329. aware Distributed Resource Management of Virtu- [45] Naoi, Y. and Yamada, H.: A GPU-Accelerated alized Clusters, in Proc. of the 11th ACM SIG- VM Live Migration for Big Memory Workloads, in PLAN/SIGOPS International Conference on Vir- Poster Session at the ACM Symposium on Cloud tual Execution Environments (VEE ’15), 2015, Computing (SoCC ’14), 2014. pp. 93–106. [46] ORACLE: VirtualBox, 2007, http://www. [58] Wang, M., Meng, X. and Zhang, L.: Consoli- virualbox.org. dating virtual machines with dynamic bandwidth [47] Pan, Z., Dong, Y., Chen, Y., Zhang, L. and demand in data centers, in Proc. of the 30th IEEE Zhang, Z.: CompSC: Live Migration with Pass- International Conference on Computer Communi- through Devices, in Proc. of the 8th ACM SIG- cations (INFOCOM ’11), 2011, pp. 71 –75. PLAN/SIGOPS Conference on Virtual Execution [59] Williams, D., Jamjoom, H. and Weatherspoon, Environments (VEE ’12), 2012, pp. 109–120. H.: The Xen-Blanket: Virtualize Once, Run Ev- [48] Rai, A., Ramjee, R., Anand, A., Padmanabhan, erywhere, in Proc. of the 7th ACM European Con- V. N. and Varghese, G.: MiG: Efficient Migration ference on Computer Systems (EuroSys ’12), 2012, of Desktop VMs Using Semantic Compression, in pp. 113–126. Proc. of the 2013 USENIX Conference on Annual [60] Wood, T., Ramakrishnan, K. K., Shenoy, P. and Technical Conference (USENIX ATC ’13), 2013, van der Merwe, J.: CloudNet: Dynamic Pooling of pp. 25–36. Cloud Resources by Live WAN Migration of Vir- [49] Rajagopalan, S., Cully, B., O’Connorand, R. tual Machines, in Proc. of the 7th ACM Interna- and Warfield, A.: SecondSite: Disaster Tolerance tional Conference on Virtual Execution Environ- as a Service, in Proc. of the 8th ACM Interna- ments (VEE’11), 2011, pp. 121–132. tional Conference on Virtual Excution Environ- [61] Wood, T., Tarasuk-Levin, G., Shenoy, P., ments (VEE ’12), 2012, pp. 97–108. Desnoyers, P., Cecchet, E. and Corner, M. D.: [50] Sharma, P., Lee, S., Guo, T., Irwin, D. and Memory Buddies: Exploiting Page Sharing for Shenoy, P.: SpotCheck: Designing a Derivative IaaS Smart Colocation in Virtualized Data Centers, in Cloud on the Spot Market, in Proc. of the Tenth Eu- Proc. of the 2009 ACM SIGPLAN/SIGOPS Inter- ropean Conference on Computer Systems (EuroSys national Conference on Virtual Execution Environ- ’15), 2015, pp. 16:1–16:15. ments (VEE ’09), 2009, pp. 31–40. [51] Song, X., Shi, J., Liu, R., Yang, J. and Chen, H.: [62] Wood, T., Venkataramani, A. and Yousif, M.: Parallelizing Live Migration of Virtual Machines, in Black-box and Gray-box Strategies for Virtual Ma- Proc. of the 9th ACM International Conference on chine Migration, in Proc. of the 4th USENIX Sym- Virtual Execution Environments (VEE ’13), 2013, posium on Networked Systems Design and Imple- pp. 85–96. mentation (NSDI ’07), 2007, pp. 229–242. [52] Sväard, P., Hudzia, B., Tordsson, J. and Elm- [63] Xu, F., Liu, F., Liu, L., Jin, H., Li, B. and roth, E.: Evaluation of Delta Compression Tech- Li, B.: iAware: Making Live Migration of Virtual niques for Efficient Live Migration of Large Vir- Machines Interference-Aware in the Cloud, IEEE tual Machines, in Proc. of the 7th ACM Inter- Transactions on Computers, Vol. 63, No. 12(2014), national Conference on Virtual Excution Environ- pp. 3012–3025. ments (VEE ’11), 2011, pp. 111–120. [64] Zhang, X., Huo, Z., Ma, J. and Meng, D.: Ex- [53] Swift, M. M., Annamalai, M., Bershad, B. N. ploiting Data Deduplication to Accelerate Live Vir- and Levy, H. M.: Recoverying Device Drivers, in tual Machine Migration, in Proc. of the IEEE Inter- Proc. of the 6th USENIX Symposium on Operating national Conference on Cluster Computing (CLUS- Systems Design and Implementation (OSDI ’04), TER ’10), 2010, pp. 88–96. 2004, pp. 1–16. [65] Zheng, J., Ng, T. S. E. and Sripanidkulchai, K.: [54] Verma, A., Ahuja, P. and Neogi, A.: pMap- Workload-aware Live Storage Migration for Clouds,

114 Information and Media Technologies 11: 101-115 (2016) reprinted from: Computer Software 33(2): 101-115 (2016) © Japan Society for Software Science and Technology

in Proc. of the 7th ACM SIGPLAN/SIGOPS In- Hiroshi Yamada ternational Conference on Virtual Execution Envi- ronments (VEE ’11), 2011, pp. 133–144. received his B.E. and M.E. degrees [66] Zheng, J., Ng, T. S. E., Sripanidkulchai, K. and from the University of Electro- Liu, Z.: COMMA: Coordinating the Migration of communications in 2004 and 2006, Multi-tier Applications, in Proc. of the 10th ACM SIGPLAN/SIGOPS International Conference on respectively. He received his Ph.D. Virtual Execution Environments (VEE ’14), 2014, degree from Keio University in pp. 153–164. 2009. He is currently an associate professor of the [67] Zhou, R., Liu, F., Li, C. and Li, T.: Optimiz- Division of Advanced Information Technology and ing Virtual Machine Live Storage Migration in Het- Computer Science at Tokyo University of Agricul- erogeneous Storage Environment, in Proc. of the 9th ACM SIGPLAN/SIGOPS International Con- ture and Technology. His research interests include ference on Virtual Execution Environments (VEE operating systems, virtualization, dependable sys- ’13), 2013, pp. 73–84. tems, and cloud computing. He is a member of ACM, USENIX, and IEEE/CS.

115