Container-based Operating System Virtualization: A Scalable, High-performance Alternative to Hypervisors Stephen Soltesz Herbert Pötzl Marc E. Fiuczynski Dept. of Computer Science Linux-VServer Maintainer Dept. of Computer Science Princeton University Laaben, Austria Princeton University Princeton, New Jersey 08540 herbert@13thfloor.at Princeton, New Jersey 08540 [email protected] [email protected] Andy Bavier Larry Peterson Dept. of Computer Science Dept. of Computer Science Princeton University Princeton University Princeton, New Jersey 08540 Princeton, New Jersey 08540 [email protected] [email protected] ABSTRACT 1. INTRODUCTION Hypervisors, popularized by Xen and VMware, are quickly Operating system designers face a fundamental tension be- becoming commodity. They are appropriate for many us- tween isolating applications and enabling sharing among age scenarios, but there are scenarios that require system them—to simultaneously support the illusion that each ap- virtualization with high degrees of both isolation and effi- plication has the physical machine to itself, yet let applica- ciency. Examples include HPC clusters, the Grid, hosting tions share objects (e.g., files, pipes) with each other. To- centers, and PlanetLab. We present an alternative to hy- day’s commodity operating systems, designed for personal pervisors that is better suited to such scenarios. The ap- computers and adapted from earlier time-sharing systems, proach is a synthesis of prior work on resource containers typically provide a relatively weak form of isolation (the pro- and security containers applied to general-purpose, time- cess abstraction) with generous facilities for sharing (e.g., shared operating systems. Examples of such container-based a global file system and global process ids). In contrast, systems include Solaris 10, Virtuozzo for Linux, and Linux- hypervisors strive to provide full isolation between virtual VServer. As a representative instance of container-based machines (VMs), providing no more support for sharing be- systems, this paper describes the design and implementa- tween VMs than the network provides between physical ma- tion of Linux-VServer. In addition, it contrasts the archi- chines. tecture of Linux-VServer with current generations of Xen, and shows how Linux-VServer provides comparable support The workload requirements for a given system will direct for isolation and superior system efficiency. users to the point in the design space that requires the least trade-off. For instance, workstation operating systems gen- erally run multiple applications on behalf of a single user, Categories and Subject Descriptors making it natural to favor sharing over isolation. On the D.2.11 [Software Engineering]: Software Architectures; other hand, hypervisors are often deployed to let a single D.4.8 [Operating Systems]: Performance—Measurements, machine host multiple, unrelated applications, which may Operational analysis run on behalf of independent organizations, as is common when a data center consolidates multiple physical servers. General Terms The applications in such a scenario have no need to share information. Indeed, it is important they have no impact Performance Measurement Design on each other. For this reason, hypervisors heavily favor full isolation over sharing. However, when each virtual ma- Keywords chine is running the same kernel and similar operating sys- Linux-VServer Xen virtualization container hypervisor op- tem distributions, the degree of isolation offered by hyper- erating system alternative visors comes at the cost of efficiency relative to running all applications on a single kernel. A number of emerging usage scenarios—such as HPC clus- Permission to make digital or hard copies of all or part of this work for ters, Grid, web/db/game hosting organizations, distributed personal or classroom use is granted without fee provided that copies are hosting (e.g., PlanetLab, Akamai, Amazon EC2)—benefit not made or distributed for profit or commercial advantage and that copies from virtualization techniques that isolate different groups bear this notice and the full citation on the first page. To copy otherwise, or of users and their applications from one another. What these republish, to post on servers or to redistribute to lists, requires prior specific usage scenarios share is the need for efficient use of system permission and/or a fee. resources, either in terms of raw performance for a single EuroSys’07, March 21–23, 2007, Lisboa, Portugal. Copyright 2007 ACM 978-1-59593-636-3/07/0003 $5.00. or small number of VMs, or in terms of sheer scalability of 275 concurrently active VMs. ios of VMs to set the context within which we compare and contrast the different approaches to virtualization. We then This paper describes a virtualization approach designed to make a case for container-based virtualization with these enforce a high degree of isolation between VMs while main- usage scenarios. taining efficient use of system resources. The approach syn- thesizes ideas from prior work on resource containers [2, 2.1 Usage Scenarios 14] and security containers [7, 19, 12, 25] and applies it There are many innovative ideas that exploit VMs to secure to general-purpose, time-shared operating systems. Indeed, work environments on laptops, detect virus attacks in real- variants of such container-based operating systems are in time, determine the cause of computer break-ins, and debug production use today—e.g., Solaris 10 [19], Virtuozzo [23], difficult to track down system failures. Today, VMs are and Linux-VServer [11]. predominantly used by programmers to ease software devel- opment and testing, by IT centers to consolidate dedicated The paper makes two contributions. First, this is the first servers onto more cost effective hardware, and by traditional thorough description of the techniques used by Linux-VServer hosting organizations to sell virtual private servers. Other for an academic audience (henceforth referred to as just emerging, real-world scenarios for which people are either “VServer”). We choose VServer as the representative in- considering, evaluating, or actively using VM technologies stance of the container-based system for several reasons: 1) include HPC clusters, the Grid, and distributed hosting or- it is open source, 2) it is in production use, and 3) because we ganizations like PlanetLab and Amazon EC2. This paper have real data and experience from operating 700+ VServer- focuses on these three emerging scenarios, for which effi- enabled machines on PlanetLab [17]. ciency is paramount. Second, we contrast the architecture of VServer with a re- Compute farms, as idealized by the Grid vision and typically cent generation of Xen, which has changed drastically since realized by HPC clusters, try to support many different users its original design was described by Barham et al. [3]. In (and their application’s specific software configurations) in a terms of performance, the two solutions are equal for CPU batch-scheduled manner. While compute farms do not need bound workloads, whereas for I/O centric (server) work- to run many concurrent VMs (often just one per physical loads VServer makes more efficient use of system resources machine at a time), they are nonetheless very sensitive to and thereby achieves better overall performance. In terms raw performance issues as they try to maximize the number of scalability, VServer far surpasses Xen in usage scenar- of jobs they can push through the overall system per day. ios where overbooking of system resources is required (e.g., As well, experience shows that most software configuration PlanetLab, managed web hosting, etc), whereas for reser- problems encountered on compute farms are due to incom- vation based usage scenarios involving a small number of patibilities of the system software provided by a specific OS VMs VServer retains an advantage as it inherently avoids distribution, as opposed to the kernel itself. Therefore, giv- duplicating operating system state. ing users the ability to use their own distribution or special- ized versions of system libraries in a VM would resolve this The next section presents a motivating case for container point of pain. based systems. Section 3 presents container-based tech- niques in further detail, and describes the design and im- In contrast to compute farms, hosting organizations tend plementation of VServer. Section 4 reproduces benchmarks to run many copies of the same server software, operating that have become familiar metrics for Xen and contrasts system distribution, and kernels in their mix of VMs. In for- those with what can be achieved by VServer. Section 5 profit scenarios, hosting organizations seek to benefit from describes the kinds of interference observed between VMs. an economy of scale and need to reduce the marginal cost Finally, Section 6 offers some concluding remarks. per customer VM. Such hosting organizations are sensitive to issues of efficiency as they try to carefully oversubscribe 2. MOTIVATION their physical infrastructure with as many VMs as possible, Virtual machine technologies are the product of diverse groups without reducing overall quality of service. Unfortunately, with different terminology. To ease the prose, we settle on companies are reluctant to release just how many VMs they referring to the isolated execution context running on top operate on their hardware. of the underlying system providing virtualization as a vir- tual machine (VM), rather than a domain, container, applet, Fortunately,