A Transparent Remote Paging Model for Virtual Machines Haogang Chen, Yingwei Luo, Xiaolin Wang, Binbin Zhang, Yifeng Sun Department of Computer Science and Technology, Peking University, P.R.China, 100871 Zhenlin Wang Department of Computer Science, Michigan Technological University, Houghton, MI 49931, USA Abstract oped to improve memory resource efficiency in vir- tual machine systems, including memory sharing[2], In virtual machine systems, with the increase in the num- ballooning[19] and demand paging. These techniques ber of VMs and the demands of the applications, the can either reduce memory footprint or adjust memory al- main memory is becoming a bottleneck for the applica- location among VMs within the same physical machine. tion performance. To improve paging performance and But none of them helps when all the VMs used up their alleviating thrashing behavior for memory-intensive or memory. I/O-intensive virtual machine workloads, we proposed Other existing work also aims at integrating hard- hypervisor based remote paging, which allows a virtual ware resources in a distributed environment using virtu- machine to transparently use the memory resource on alization. For example, vNUMA[3] can create a virtual other physical machines as a cache between its virtual NUMA multiprocessor system over cluster of worksta- memory and virtual disk device. tions. Taking the approach of distributed shared memory The goal of remote paging is to reduce disk accesses, (DSM), vNUMA makes the memory resource on each which is much slower than transferring memory pages node accessible for other nodes through a shared address over modern interconnect networks. As a result, the im- space. Unfortunately, simulating shared address space pact of thrashing behavior can be alleviated since the av- in the system level imposed a significant overhead. The erage disk I/O latency is reduced. Remote paging also virtual machine monitor (VMM) has to maintain mem- benefits some I/O intensive applications, resulting in a ory coherence among nodes, which is very expensive for better-than-native performance. write-shared pages[14]. More importantly, the unaware- Our remote paging model is totally transparent to the ness of the underlying distributed architecture from the guest operating system as well as its applications, and is guest operating system (guest OS) can magnify the over- compatible with existing techniques like ballooning, hy- head, since it does not know how to exploit data locality. pervisor disk cache and page sharing. A combination of It has been observed that performance thrashing on them can provide a more flexible resource management many server systems comes from bursty requests[7, 17], policy. that is, memory requirements proliferate only for a short period of time. Generally, it is unnecessary to span the 1 Introduction entire system over several machines, but better to tem- porarily “borrow” some memory from other physical In recent years, virtual machine(VM)[1, 2, 9] technolo- machines to overcome the burst. In this way, most com- gies are becoming more and more popular due to its sup- putation remains on the original machine, so locality is port for resource encapsulation, hardware independency maintained. and easy manageability[5]. Most of current research fo- This paper proposes hypervisor based remote paging. cuses on how to multiplex hardware resources so that It allows a virtual machine to transparently use the mem- many VMs can run simultaneously on a single physical ory resource on other physical machines as a cache be- machine. As a result, resources available for a VM is tween its virtual memory and virtual disk device. A pag- usually restricted within a physical machine boundary. ing request from a VM is intercepted by the hypervisor, In a virtual machine system, the memory resource and is preferably satisfied from remote machines (called is virtualized in the manner of partitioning. With the memory servers). The goal of remote paging is to reduce increase in the number of VMs and the demands of disk accesses, which is much slower than transferring the applications, the main memory will become an ex- memory pages over modern interconnect networks. tremely limited resource. If the memory requirements Virtual machine systems can benefit from the remote of a virtual machine exceeds the available amount, the paging in many ways. First, this model speeds up VM VM may suffer severe performance degradation due disk I/O transparently so that the impact of thrashing be- to thrashing[13]. Various techniques have been devel- havior can be alleviated. Second, some I/O intensive ap- 1 plications also benefit from remote paging, resulting in page that has been paged out by VMM. This will cause a better-than-native performance. Third, it can be used the page to be read in just for writing back to the VM’s together with existing techniques like ballooning, hyper- swapping area again. The third one is redundant write visor disk cache and page sharing, to provide a more back. If an unallocated page is selected by VMM as a flexible resource management policy. Additionally, our victim page, it will unnecessarily write back the page approach is more cost efficient than other methods, such content[10]. Even worse, when an access time-based as migrating and vNUMA, since it respects the princi- (such as Least Recently Used, LRU) algorithm is used ple of locality and does not require the remote memory in hypervisor, such choice is very likely to happen be- server to have strong computation power. cause unallocated pages are usually inactive. Our work inherits the ideas of previous studies on re- Some artifices can be used to walk around certain mote paging in multicomputer systems[6, 12]. Remote problems. For example, VMware ESX Server[19] uses paging allows a node to use the memory on other nodes a randomized page replacement algorithm to alleviate as its fast backing storage. When a node is out of mem- bad interaction with guest OS; Cellular Disco[10] avoids ory, a set of pages will be sent and cached on remote double paging by remapping guest disk blocks to hyper- memory servers, rather than directly going to the disk visor’s swap area. However, these methods do not fun- storage. The new caching layer introduced by remote damentally solve the above problems. These counter- paging architecturally lies between a node’s local mem- effects in all can likely offset the performance benefits ory and its disk storage (usually on I/O nodes). gained by remote paging. Our hypervisor-based remote paging can be entirely implemented within the VMM, without any modifica- 2.1 Basic Design tion to the guest OS. Moreover, we do not introduce an- other level of paging in the hypervisor. Victim pages Rather than introducing another level of paging in the are selected by the guest OS itself. A page swap in hypervisor, our design let the guest OS select victim guest OS will result in disk I/O that can be intercepted pages itself, according to its native replacement poli- by the VMM. Under the intervention of VMM, the pag- cies. This idea is common with ballooning[19], which ing request is transparently extended to remote memory can eliminate the anomaly described earlier. servers. In other words, the remote cache can be con- A page swap in the guest OS will result in disk I/O sidered as a second level cache of VM’s virtual stor- that can be intercepted by the VMM. The VMM main- age. Section 2 describes the characteristics and design tains a fast cache on remote machines, called memory choices of our hypervisor-based remote paging model, servers. A disk request that hits a cached block is di- and investigates some cache policies for the remote rectly satisfied from the remote memory server, which cache. A hash-comparing based technique is proposed to provides a lower latency than a real disk access. In this support eviction-based remote cache in our work. Sec- way, a new level of caching is transparently added into tion 3 details our implementation of a prototype of our virtual machine’s memory hierarchy. Figure 1 illustrates remote paging model in Xen hypervisor. this design. 2 The Remote Paging Model Local Machine Remote Machine (Memory Server) Guest OS In most virtual memory systems, victim pages (i.e. the VM Memory pages to be evicted) are selected by the paging routine itself. Therefore, an intuitive way to implement remote Disk I/O paging in virtual machine system is to initiate paging Interconnect Remote VMM Disk in the hypervisor. Because VMM controls the trans- Cache Memory lation from guest-physical-address to the correspond- Disk I/O ing machine-address, this method will automatically ex- tends VM’s memory space to a remote machine, giving guest OS the illusion of a larger physical address space. Disk Storage However, in virtual machine environment, introduc- ing another level of paging raises several unique perfor- Figure 1: Basic design of hypervsior-based remote paging. Block No. Hash code mance problems: First, VMM cannot wisely choose vic- In this paper,page we assume the virtual machine uses the local tim pages due to lack of information on page usages[19]. storages on(pfn) the same physical machine. Admit (page, old_bno) The second problem is double paging[9]. Because page I/O access replacement in the hypervisor is transparent to the guest Our remote paging model lies between the virtual ma- (pfn, bno) … …… Y OS, it is likely that the guest OS will choose to reclaim a chine’s memorypfn and its virtual storage devices. More old_bno 0x1ac93...2d = 2 New hash code 0x1ac93...2d specifically, if the guest OS also has its own buffer cache, cache, since it can provide a better hit rate.