Towards a Flexible, Lightweight Virtualization Alternative David C. van Moolenbroek Raja Appuswamy Andrew S. Tanenbaum VU University Amsterdam VU University Amsterdam VU University Amsterdam [email protected] [email protected] [email protected] ABSTRACT level. For OS-level virtualization it is relatively high, at the In recent times, two virtualization approaches have become operating system application interface level. This boundary dominant: hardware-level and operating system-level virtu- determines important properties of the virtualization sys- alization. They differ by where they draw the virtualization tem: the former is generally thought of as more flexible and boundary between the virtualizing and the virtualized part better isolated; the latter as faster and more lightweight. of the system, resulting in vastly different properties. We ar- In this paper, we take a top-down look at the virtualiza- gue that these two approaches are extremes in a continuum, tion boundary. We argue that the two existing approaches and that boundaries in between the extremes may combine are extremes in a continuum with a relatively unexplored yet several good properties of both. We propose abstractions promising middle ground. This middle ground offers the po- to make up one such new virtualization boundary, which tential to combine several of the good properties from both combines hardware-level flexibility with OS-level resource sides. We propose a set of \mid-level" abstractions to form a sharing. We implement and evaluate a first prototype. new virtualization boundary, where the virtualizing infras- tructure provides object-based storage, page caching and mapping, memory management, and scheduling, whereas Categories and Subject Descriptors any higher-level abstractions are implemented within each D.4.0 [Operating Systems]: General virtual environment. We argue that this approach offers a viable alternative, providing flexible, lightweight virtual en- General Terms vironments for users and applications. We implement the core of our ideas in a microkernel-based prototype. Design, Performance, Reliability, Security The rest of the paper is laid out as follows. In Sec. 2 we describe virtualization boundaries as a continuum. In Sec. 3 1. INTRODUCTION we propose and discuss a new alternative. We describe our The concept of virtualization in computer systems has prototype in Sec. 4 and its evaluation in Sec. 5. Sec. 6 lists been around for a long time, but it has gained widespread related work. We conclude in Sec. 7. adoption only in the last fifteen years. It is used to save on hardware and energy costs by consolidating multiple work- 2. VIRTUALIZATION AS A CONTINUUM loads onto a single system without compromising on isola- tion, as well as to create host-independent environments for Reduced to its most basic form, any virtualization system users and applications. consists of two parts. The domains are the environments In these recent times, two virtualization approaches have being virtualized, with (at least) user applications in them. established themselves as dominant: hardware-level and op- The host comprises all system components facilitating the erating system-level (OS-level) virtualization. The two are virtualization of such domains. We refer to the abstraction fundamentally different in where they draw the virtualiza- level defining the separation between these parts as the vir- tion boundary: the abstraction level at which the virtualized tualization boundary. part of the system is separated from the virtualizing infras- In this paper, we consider only virtualization in which un- tructure. The boundary for hardware-level virtualization is changed applications can run using machine-native instruc- low in the system stack, at the machine hardware interface tions (as opposed to e.g. Java [19] and NaCl [60]), and in which domains are considered untrusted. In this space, two Permission to make digital or hard copies of all or part of this work for per- approaches have become dominant: hardware-level virtual- sonal or classroom use is granted without fee provided that copies are not ization (Sec. 2.1) and OS-level virtualization (Sec. 2.2). We made or distributed for profit or commercial advantage and that copies bear argue that these are extremes in a continuum of virtualiza- this notice and the full citation on the first page. Copyrights for components tion boundaries, in which new alternatives have the potential of this work owned by others than the author(s) must be honored. Abstract- to combine good properties of both (Sec. 2.3). ing with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. 2.1 Hardware-level virtualization ACM SYSTOR ’14, June 10-12 2014, Haifa, Israel Hardware-level virtualization [4,6,17,58] places the virtu- Copyright is held by the owner/author(s). Publication rights licensed to alization boundary as low in the system stack as practically ACM. ACM 978-1-4503-2920-0/14/06 ...$15.00. feasible. A host layer (virtual machine monitor or hypervi- DOI 10.1145/2611354.2611369 sor) provides its domains (virtual machines) with abstrac- tions that are either equal to a real machine or very close to stractions to its domains that are higher-level than those it: privileged CPU operations, memory page tables, virtual of hardware, and yet lower-level than those of OSes. Each storage and network devices, etc. The low boundary allows domain then contains a system layer which uses those \mid- a full software stack (OS and applications) to run inside a level" abstractions to construct the desired interface for its domain with minimal or no changes. The result is strong applications, as illustrated in Fig. 1. This way, we have the isolation, and full freedom for the OS in each domain to potential to reduce the footprint of the domains while retain- implement arbitrary high-level abstractions. ing much of their flexibility and isolation. The only point However, the OS adds to the domain's footprint, while that we must compromise on, is the ability to run existing typically exploiting only a part of its flexibility. Several fun- operating systems. damental abstractions are common across OSes: processes, storage caches, memory regions, etc. Reimplementing these 3. A NEW VIRTUALIZATION DESIGN in isolation leads to duplication and missed opportunities In this section, we present the design of a new point in for global optimization. Only the host side can solve these the virtualization continuum. We first state our design goals issues, but the low boundary creates a semantic gap [8]: the (Sec. 3.1). We then present the abstractions making up the host lacks necessary knowledge about the higher-level ab- new virtualization boundary (Sec. 3.2). Finally, we discuss stractions within the domains. Many ad-hoc techniques have the properties of our design (Sec. 3.3). been proposed to work around this gap [14,26,35,39,40,55]. 3.1 Design goals 2.2 Operating system-level virtualization Our goal is to establish a new set of abstractions imple- In contrast, with OS-level virtualization [28,42,44,48,61], mented in the host system and exposed to the domains. the operating system itself has been modified to be the vir- Each domain's system layer may use those abstractions to tualization host. The domains (containers) consist of appli- construct an interface for its applications. In principle, the cation processes only{all system functionality is in the OS. domain should be able to achieve binary compatibility with Each domain gets a virtualized view of the OS resources: the existing applications. With that constraint as a given, we file system hierarchy, process identifiers, network addresses, set the following subgoals. etc. Since the OS doubles as the host, there is no redundancy First, the abstraction level should be high enough to bridge between domains and resources can be optimized globally. the semantic gap. In particular, the new abstractions should Thus, OS-level virtualization is relatively lightweight. allow for lightweight domains, mainly by reducing re- However, merging the host role into the OS has down- source duplication. In many cases, the domains running sides as well. First, it eliminates all the flexibility found in on a single host will bear great similarity, because they im- hardware-level virtualization: the domains have to make do plement the same application interface, and thus can share with the abstractions offered by the OS. Second, the merge programs, libraries, and data, both in memory and on disk. removes an isolation boundary; failures and security prob- The abstractions should make sharing of such resources a lems in the OS may now affect the entire system rather than natural result rather than an afterthought, but with copy- a single domain. on-write (CoW) semantics to retain full isolation. Bridging the semantic gap should also allow for other global opti- 2.3 The case for new alternatives mizations generally found in OS-level virtualization only. The placement of the virtualization boundary in the soft- Second, the new abstractions should be sufficiently low- ware stack clearly has important consequences. However, we level to give the system layer in each domain substantial argue that the two described approaches are extremes in a flexibility in implementing (or omitting) high-level abstrac- continuum. With the boundary as low in the software stack tions for its applications. The host system should expose as possible, hardware-level virtualization represents one end only abstractions that are well established as fundamental of the spectrum. OS-level virtualization represents the other building blocks for operating systems and thus practically end, with the boundary as high as possible without affecting used in all domains. applications. That leaves a wide range of choices in between Third, it should be possible to provide the new abstrac- these extremes.