A Case for Virtualizing Persistent Memory
Total Page:16
File Type:pdf, Size:1020Kb
A Case for Virtualizing Persistent Memory Liang Liang, Rong Chen, Haibo Chen, Yubin Xia, †KwanJong Park, Binyu Zang, Haibing Guan Shanghai Key Laboratory of Scalable Computing and Systems, Shanghai Jiao Tong University †Samsung Electronics Contact: [email protected] Abstract Keywords Persistent Memory, Virtualization, Para- With the proliferation of software and hardware support virtualization for persistent memory (PM) like PCM and NV-DIMM, we envision that PM will soon become a standard component of 1. Introduction commodity cloud, especially for those applications demand- Computer systems have long faced a tension between ing high performance and low latency. Yet, current virtual- performance and persistence, which poses a challenge to ization software lacks support to efficiently virtualize and place data in volatile and non-volatile storage to ensure manage PM to improve cost-effectiveness, performance, and proper crash recovery under power failures. Persistent mem- endurance. ory (PM), either in the form of battery-backed memory In this paper, we make the first case study on extending cards (e.g., NVDIMM) or emerging non-volatile memory commodity hypervisors to virtualize PM. We explore design like PCM [47], Memristor [65], STT-MRAM [3], is promis- spaces to abstract PM, including load/store accessible guest- ing to relax this tension by providing high-performance re- physical memory and a block device. We design and im- coverable systems. plement a system, namely VPM, which provides both full- Because of its intriguing promise, PM has gained signif- virtualization as well as a para-virtualization interface that icant interests from both hardware and software sides. Ma- provide persistence hints to the hypervisor. By leveraging jor memory vendors like Micron, Viking, JEDEC, Fusion-IO the fact that PM has similar characteristics with DRAM ex- have provided commercial NVDIMM to the mass market. cept for persistence, VPM supports transparent data migra- SNIA has also recently formed the NVDIMM Special In- tion by leveraging the two-dimensional paging (e.g., EPT) to terest Group [49] to accelerate the adoption of NVDIMM. adjust the mapping between guest PM to host physical mem- For emerging non-volatile memory, Micron has announced ory (DRAM or PM). Finally, VPM provides efficient crash its 1Gb PCM available to volume production [38] in 2012. In recovery by properly bookkeeping guest PM’s states as well response of this, researchers have investigated a number of as key hypervisor-based states into PM in an epoch-based ways of utilizing PM [10, 14, 20, 31, 53, 62, 64, 68]. Linux consistency approach. Experimental results with VPM im- and Windows developers have started to provide support for plemented on KVM and Linux using simulated PCM and PM-aware file systems [15, 16]. NVDIMM show that VPM achieves a proportional consoli- With the proliferation of hardware and software systems dation of PM with graceful degradation of performance. Our for PM, as well as the emerging in-memory computing in- para-virtualized interface further improves the consolidation stances for big data from major cloud providers [2], we en- ratio with less overhead for some workloads. vision that PM will soon be an indispensable part of cloud Categories and Subject Descriptors D.4.2 [Storage Man- computing systems. Actually, recent industry talks [1, 45] in- agement ]: Virtual memory dicated that cloud will likely incorporate non-volatile mem- ory like NVDIMM to increase the services reliability, scal- General Terms Design, Performance ability, and availability. However, commodity hypervisors still lack support to virtualize and efficiently manage PM in Permission to make digital or hard copies of all or part of this work for personal or a cloud environment. classroom use is granted without fee provided that copies are not made or distributed In this paper, we make a case study on how to effi- for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the ciently virtualize PM to improve cost-effectiveness and per- author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or formance, which are important for several reasons. First, as republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. new technologies, PM is still with a relatively high price. SoCC ’16, October 05 - 07, 2016, Santa Clara, CA, USA. For example, currently NV-DIMM’s price is at least 8-10X c 2016 Copyright held by the owner/author(s). Publication rights licensed to ACM. ISBN 978-1-4503-4525-5/16/10. $15.00. higher than DRAM according to the price quote from NV- DOI: http://dx.doi.org/10.1145/2987550.2987551 DIMM providers; efficiently virtualizing and consolidating 126 PM may lead to significant cost-effectiveness. Second, some solidation rate by allowing more workloads to be executed emerging memory technologies like PCM or Memristor still concurrently with low performance degradation. have inferior write performance than DRAM, virtualizing This paper makes the following contributions: them may lead to performance improvement. By virtualiz- • A case for virtualizing PM (§3) and a set of ways to ing PM, we provide VM with an abstraction of sufficient virtualize PMs (§4). amount of PM, through emulating it with multiple types of media, without any relax to the persistency that PM offers. • A set of techniques to efficiently consolidate PM for There are several technical challenges in providing effi- better cost-efficiency and performance (§5). cient virtualization for PM and VPM addresses them in a • An implementation on KVM (§6) set of evaluations that framework called VPM. First, current virtualization lacks confirms the effectiveness of VPM (§7). support for an appropriate abstraction for PM, which may present as a block device or directly as a piece of mem- 2. Background and Motivation ory in guest virtual machines (VMs). In this paper, we ex- 2.1 Persistent Memory plore the design spaces of providing appropriate abstrac- tions as well as support from hypervisors to different forms Persistent memory (PM), also known as non-volatile of PM. Specifically, VPM virtualizes PM as a block device memory (NVRAM), embraces the high performance byte and a special piece of memory and provides a few para- addressability of DRAM and the persistency of disk and virtualization interfaces to bridge the semantic gap of per- flash devices. They can be attached to a DRAM-like mem- sistency between guest VMs and the hypervisor. ory bus and thus can be accessed through load and store Second, as PM may exhibit different performance and instructions [23]. Currently, there are several technologies price characteristics from DRAM, it is critical to consoli- providing PM potentially as a main-memory replacement: date PM to maximize cost-efficiency, performance, and en- PCM (Phase-change Memory) [29], STT-MRAM (Spin- durance. As a PM has similar characteristics (except some Transfer-Torque MRAM) [27], battery-backed DRAM (i.e., performance gap between DRAM and some emerging mem- NVDIMM) [52], and memristors [51]. With the advances of ory like PCM) with DRAM when serving read accesses from semiconductor techniques, PM is now commercially avail- guest VM, it has no impact on persistence whether data is able with sizes of 8GB [52]. In this paper, we do not con- placed in DRAM or PM. To this end, VPM leverages DRAM sider flash-based storage (e.g., solid state disk) since they to place read-mostly data in a transparent way by leverag- are mostly accessed as block device through I/O bus. ing the two-dimensional paging (e.g., extended page table While it is still unclear which emerging technology would (EPT)) to dynamically migrating data between DRAM and likely be a replacement of DRAM, we envision that there PM. will be a long period where DRAM will co-exist with emerg- To make an optimal decision on data placement, VPM ing PM, for the sake of performance, cost, and endurance. needs to efficiently track PM accesses to predict future ac- A co-existence of DRAM with PM would lead to hetero- cesses. VPM leverages the newly introduced dirty bit in the geneity in performance and cost-effectiveness. For example, hypervisor-managed page table (e.g., EPT) to transparently NVDIMM is with 8-10X price compared to DRAM. Other tracking writing working set of guest VMs. emerging PM would still have a relatively higher price until Finally, crash recovery is a challenge for VPM under a massive production and wide adoption. Besides, PM tends to hybrid memory architecture. To ensure proper and efficient have two types: symmetric read/write performance or asym- crash recovery under a power failure, VPM only needs to metric read/write performance. Finally, some PM devices preserve key data structures in the hypervisor to be consis- have endurance issues such that they have a limited lifetime. tent but also need to efficiently bookkeep virtualized PMs In this paper, we mainly consider two types of PM: 1) PM for proper crash recovery. VPM address this challenge by with symmetric read/write performance but higher price; 2) making the part of hypervisors leverage PM to store its key PM with asymmetric read/write performance. Specifically, data structures for crash recovery and using a portion of PM we consider PCM and NVDIMM as their representatives. to bookkeep guest VM’s PM structures. PM in Use: PM is now being used in three categories. We have built a working prototype based on KVM, by First, as many applications and file systems are still not PM simulating PM using DRAM and treating PM as a portion aware, PM is used through the block device interface (e..g, of the physical address space in the hypervisor.