The Ideal Versus the Real: Revisiting the History of Virtual Machines and Containers Allison Randal, University of Cambridge Abstract also have greater access to the host’s privileged software (kernel, operating system) than a physically distinct ma- The common perception in both academic literature and chine would have. the industry today is that virtual machines offer better se- curity, while containers offer better performance. How- Ideally, multitenant environments would offer strong ever, a detailed review of the history of these technolo- isolation of the guest from the host, and between guests gies and the current threats they face reveals a different on the same host, but reality falls short of the ideal. The story. This survey covers key developments in the evo- approaches that various implementations have taken to lution of virtual machines and containers from the 1950s isolating guests have different strengths and weaknesses. to today, with an emphasis on countering modern misper- For example, containers share a kernel with the host, ceptions with accurate historical details and providing a while virtual machines may run as a process in the host solid foundation for ongoing research into the future of operating system or a module in the host kernel, so they secure isolation for multitenant infrastructures, such as expose different attack surfaces through different code cloud and container deployments. paths in the host operating system. Fundamentally, how- ever, all existing implementations of virtual machines and containers are leaky abstractions, exposing more of 1 Introduction the underlying software and hardware than is necessary, useful, or desirable. New security research in 2018 deliv- Many modern computing workloads run in multitenant ered a further blow to the ideal of isolation in multitenant environments, such as cloud or containers, where each environments, demonstrating that certain hardware vul- physical machine is split into hundreds or thousands of nerabilities related to speculative execution—including smaller units of computing, called virtual machines, con- Spectre, Meltdown, Foreshadow, L1TF, and variants— tainers, cloud instances, or more generically guests. Typ- can easily bypass the software isolation of guests. ically, a single tenant (a user or group of users) is granted Because multitenancy has proven to be useful and access to deploy guests in an orchestrated fashion across profitable for a large sector of the computing industry, a cloud or cluster made up of hundreds or thousands it is likely that a significant percentage of computing arXiv:1904.12226v1 [cs.NI] 27 Apr 2019 of physical machines located in the same data center or workloads will continue to run in multitenant environ- across multiple data centers, to facilitate operational flex- ments for the foreseeable future. This is not a matter ibility in areas such as capacity planning, resiliency, and of na¨ıvete,´ but of pragmatism: these days, the compa- reliable performance under variable load. Each guest nies who provide and make use of multitenant environ- runs its own (often minimal) operating system and ap- ments are generally fully aware of the security risks, but plication workloads, and maintains the illusion of being they do so anyway because the benefits—such as flexi- a physical machine, both to the end users who interact bility, resiliency, reliability, performance, cost, or any of with the services running in the guests, and to develop- a dozen other factors—outweigh the risks for their par- ers who are able to build those services using familiar ticular use cases and business needs. That being the case, abstractions, such as programming languages, libraries, it is worthwhile to take a step back and examine how the and operating system features. The illusion, however, is past sixty years of evolution led to the current tension be- not perfect, because ultimately the guests do share the tween secure ideals and flawed reality, and what lessons hardware resources (CPU, memory, cache, devices) of from the past might help us build more secure software the underlying physical host machine, and consequently and hardware for the next sixty years. 1 Capsicum Influence BSD jails direct chroot indirect SunOS Solaris Zones Multics UNIX Borg Kubernetes POSIX POSIX.1e Chicago Magic Number Machine MINIX Linux LXC Docker OCI VServer CTSS CAL-TSS OpenVZ Plessey System 250 UML Kata capabilities B5000 QEMU NEMU CAP crosvm Firecracker KVM iAPX 432 Denali ukvm hvt System/38 AS/400 AWS multiprogramming CP-40/CMS CP-67/CMS VM/370 Disco VMware Xen LightVM M44/44X 1950 1960 1970 1980 1990 2000 2010 today Figure 1: The evolution of virtual machines and containers. This survey is divided into sections following the evo- use are Banga et al. [25] in 1999, Lottiaux and lutionary paths of the technologies behind virtual ma- Morin [125] in 2001, Morin et al. [143] in 2002, chines and containers, generally in chronological order, and Price and Tucker [162] in 2004. Early litera- as illustrated in Figure 1. Section 3 explores the com- ture on containers confusingly referred to them as a mon origins of virtual machines and containers in the kind of virtualization [162; 180; 140; 102; 45; 48], late 1950s and early 1960s, driven by the architectural or even called them virtual machines [180]. As con- shift toward multitasking and multiprocessing, and moti- tainers grew more popular, the confusion shifted to vated by a desire to securely isolate processes, efficiently virtual machines being called containers [37; 217]. utilize shared resources, improve portability, and mini- This survey uses the term “container” for mul- mize complexity. Section 4 examines the first virtual ma- titenant deployment techniques involving process chines in the mid-1960s to 1970s, which primarily aimed isolation on a shared kernel (in contrast with virtual to improve resource utilization in time-sharing systems. machine, as defined below). However, in practice Section 5 delves into the capability systems of the early the distinction between containers and virtual ma- 1960s to 1970s—the precursors of modern containers— chines is more of a spectrum than a binary divide. which evolved along a parallel track to virtual machines, Techniques common to one can be effectively ap- with similar motivations but different implementations. plied to the other, such as using system call filter- Section 6 outlines the resurgence of virtual machines in ing with containers, or using seccomp sandboxing the late 1990s and 2000s. Section 7 traces the emergence or user namespaces with virtual machines. of containers in the 2000s and 2010s. Section 8 investi- gates the impact of recent security research on both vir- • complexity: There are many dimensions to com- tual machines and containers. Section 9 briefly looks at plexity in computing, but in the context of mul- the relationship between virtual machines and containers titenant infrastructures some uniquely relevant di- and the related terms “cloud”, “serverless”, and “uniker- mensions are keeping each guest, the interactions nels”. between guests, and the host’s management of the guests as small and simple as possible. The im- 2 Terminology plementation technique of isolation supports mini- mizing complexity by restricting access to internal For the sake of clarity, this survey consistently uses cer- knowledge of the guests and host, and providing tain modern or common terms, even when discussing lit- well-defined interfaces to reduce the complexity of erature that used various other terms for the same con- interactions between them. cepts. • guest: The term “guest” had some early usage in • container: The term “container” does not have a the 1980s for the operating system image running single origin, but some early relevant examples of inside a virtual machine [145], but was not com- 2 mon until the early 2000s [194; 26]. This survey • process: The early literature tended to use the terms uses “guest” as a general term for operating sys- “job” [169] or “program” [52; 151; 20], and “pro- tem images hosted on multitenant infrastructures, cess” only appeared around the mid-1960s [64; 14]. but occasionally distinguishes between virtual ma- This survey uses the modern term “process”. The chine guests and container guests. early use of “multiprogramming” meaning “multi- processing” was derived from the early use of “pro- • kernel: A variety of different terms appear in the gram” meaning “process”. early literature, including “supervisory program” [52], “supervisor program” [20], “control program” • security: There are many dimensions to security [147; 151; 15], “coordinating program” [151], “nu- in computing, but in the context of multitenant in- cleus” [43; 1], “monitor” [206], and ultimately “ker- frastructures some uniquely relevant dimensions are nel” around the mid-1970s [121; 159]. This survey limiting access between guests, from guests to the uses the modern term “kernel”. host, and from the host to the guests. The imple- mentation technique of isolation supports security, • performance: There are many dimensions to per- at both the software level and the hardware level, by formance in computing, but in the context of mul- reducing the likelihood of a breach and limiting the titenant infrastructures some uniquely relevant di- scope of damage when a breach occurs. mensions are the performance impact of added lay- ers of abstraction separating the guest application workload from the host, balanced against the perfor- • virtual machine: This survey uses the term “virtual mance benefits of sharing resources between guests machine” for multitenant deployment techniques and reducing wasted resources from unused capac- involving the replication/emulation of real hard- ity. At the level of a single machine, this involves ware architectures in software (in contrast with con- running multiple guests on the same machine at tainer, as defined above). The code responsible for the same time, with potential for intelligent, dy- managing virtual machine guests on a physical host namic scheduling to extract more work from the machine is often called a “hypervisor” or “virtual same resource pool.