Xen and the Art of Virtualization Paul Barham∗, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer†, Ian Pratt, Andrew Warfield University of Cambridge Computer Laboratory 15 JJ Thomson Avenue, Cambridge, UK, CB3 0FD {firstname.lastname}@cl.cam.ac.uk ABSTRACT 1. INTRODUCTION Numerous systems have been designed which use virtualization to Modern computers are sufficiently powerful to use virtualization subdivide the ample resources of a modern computer. Some require to present the illusion of many smaller virtual machines (VMs), specialized hardware, or cannot support commodity operating sys- each running a separate operating system instance. This has led to tems. Some target 100% binary compatibility at the expense of a resurgence of interest in VM technology. In this paper we present performance. Others sacrifice security or functionality for speed. Xen, a high performance resource-managed virtual machine mon- Few offer resource isolation or performance guarantees; most pro- itor (VMM) which enables applications such as server consolida- vide only best-effort provisioning, risking denial of service. tion [42, 8], co-located hosting facilities [14], distributed web ser- This paper presents Xen, an x86 virtual machine monitor which vices [43], secure computing platforms [12, 16] and application allows multiple commodity operating systems to share conventional mobility [26, 37]. hardware in a safe and resource managed fashion, but without sac- Successful partitioning of a machine to support the concurrent rificing either performance or functionality. This is achieved by execution of multiple operating systems poses several challenges. providing an idealized virtual machine abstraction to which oper- Firstly, virtual machines must be isolated from one another: it is not ating systems such as Linux, BSD and Windows XP, can be ported acceptable for the execution of one to adversely affect the perfor- with minimal effort. mance of another. This is particularly true when virtual machines Our design is targeted at hosting up to 100 virtual machine in- are owned by mutually untrusting users. Secondly, it is necessary stances simultaneously on a modern server. The virtualization ap- to support a variety of different operating systems to accommodate proach taken by Xen is extremely efficient: we allow operating sys- the heterogeneity of popular applications. Thirdly, the performance tems such as Linux and Windows XP to be hosted simultaneously overhead introduced by virtualization should be small. for a negligible performance overhead — at most a few percent Xen hosts commodity operating systems, albeit with some source compared with the unvirtualized case. We considerably outperform modifications. The prototype described and evaluated in this paper competing commercial and freely available solutions in a range of can support multiple concurrent instances of our XenoLinux guest microbenchmarks and system-wide tests. operating system; each instance exports an application binary inter- face identical to a non-virtualized Linux 2.4. Our port of Windows Categories and Subject Descriptors XP to Xen is not yet complete but is capable of running simple user-space processes. Work is also progressing in porting NetBSD. D.4.1 [Operating Systems]: Process Management; D.4.2 [Opera- Xen enables users to dynamically instantiate an operating sys- ting Systems]: Storage Management; D.4.8 [Operating Systems]: tem to execute whatever they desire. In the XenoServer project [15, Performance 35] we are deploying Xen on standard server hardware at econom- ically strategic locations within ISPs or at Internet exchanges. We General Terms perform admission control when starting new virtual machines and expect each VM to pay in some fashion for the resources it requires. Design, Measurement, Performance We discuss our ideas and approach in this direction elsewhere [21]; this paper focuses on the VMM. Keywords There are a number of ways to build a system to host multiple Virtual Machine Monitors, Hypervisors, Paravirtualization applications and servers on a shared machine. Perhaps the simplest is to deploy one or more hosts running a standard operating sys- ∗ Microsoft Research Cambridge, UK tem such as Linux or Windows, and then to allow users to install † Intel Research Cambridge, UK files and start processes — protection between applications being provided by conventional OS techniques. Experience shows that system administration can quickly become a time-consuming task due to complex configuration interactions between supposedly dis- Permission to make digital or hard copies of all or part of this work for joint applications. personal or classroom use is granted without fee provided that copies are More importantly, such systems do not adequately support per- not made or distributed for profit or commercial advantage and that copies formance isolation; the scheduling priority, memory demand, net- bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific work traffic and disk accesses of one process impact the perfor- permission and/or a fee. mance of others. This may be acceptable when there is adequate SOSP’03, October 19–22, 2003, Bolton Landing, New York, USA. provisioning and a closed user group (such as in the case of com- Copyright 2003 ACM 1-58113-757-5/03/0010 ...$5.00. putational grids, or the experimental PlanetLab platform [33]), but Notwithstanding the intricacies of the x86, there are other argu- not when resources are oversubscribed, or users uncooperative. ments against full virtualization. In particular, there are situations One way to address this problem is to retrofit support for per- in which it is desirable for the hosted operating systems to see real formance isolation to the operating system. This has been demon- as well as virtual resources: providing both real and virtual time strated to a greater or lesser degree with resource containers [3], allows a guest OS to better support time-sensitive tasks, and to cor- Linux/RK [32], QLinux [40] and SILK [4]. One difficulty with rectly handle TCP timeouts and RTT estimates, while exposing real such approaches is ensuring that all resource usage is accounted to machine addresses allows a guest OS to improve performance by the correct process — consider, for example, the complex interac- using superpages [30] or page coloring [24]. tions between applications due to buffer cache or page replacement We avoid the drawbacks of full virtualization by presenting a vir- algorithms. This is effectively the problem of “QoS crosstalk” [41] tual machine abstraction that is similar but not identical to the un- within the operating system. Performing multiplexing at a low level derlying hardware — an approach which has been dubbed paravir- can mitigate this problem, as demonstrated by the Exokernel [23] tualization [43]. This promises improved performance, although and Nemesis [27] operating systems. Unintentional or undesired it does require modifications to the guest operating system. It is interactions between tasks are minimized. important to note, however, that we do not require changes to the We use this same basic approach to build Xen, which multiplexes application binary interface (ABI), and hence no modifications are physical resources at the granularity of an entire operating system required to guest applications. and is able to provide performance isolation between them. In con- We distill the discussion so far into a set of design principles: trast to process-level multiplexing this also allows a range of guest operating systems to gracefully coexist rather than mandating a 1. Support for unmodified application binaries is essential, or specific application binary interface. There is a price to pay for this users will not transition to Xen. Hence we must virtualize all flexibility — running a full OS is more heavyweight than running architectural features required by existing standard ABIs. a process, both in terms of initialization (e.g. booting or resuming versus fork and exec), and in terms of resource consumption. 2. Supporting full multi-application operating systems is im- portant, as this allows complex server configurations to be For our target of up to 100 hosted OS instances, we believe this virtualized within a single guest OS instance. price is worth paying; it allows individual users to run unmodified binaries, or collections of binaries, in a resource controlled fashion 3. Paravirtualization is necessary to obtain high performance (for instance an Apache server along with a PostgreSQL backend). and strong resource isolation on uncooperative machine ar- Furthermore it provides an extremely high level of flexibility since chitectures such as x86. the user can dynamically create the precise execution environment their software requires. Unfortunate configuration interactions be- 4. Even on cooperative machine architectures, completely hid- tween various services and applications are avoided (for example, ing the effects of resource virtualization from guest OSes each Windows instance maintains its own registry). risks both correctness and performance. The remainder of this paper is structured as follows: in Section 2 we explain our approach towards virtualization and outline how Note that our paravirtualized x86 abstraction is quite different Xen works. Section 3 describes key aspects of our design and im- from that proposed by the recent Denali project [44]. Denali