Containers and Virtual Machines at Scale: A Comparative Study Experimentation and Deployment Track Submission Lucas Chaufournier Prateek Sharma Prashant Shenoy [email protected] [email protected] [email protected] University of Massachusetts University of Massachusetts University of Massachusetts Amherst Amherst Amherst Y.C. Tay [email protected] National University of Singapore ABSTRACT allocation of physical resources to virtualized applications where Virtualization is used in data center and cloud environments to de- the mapping of virtual to physical resources as well as the amount couple applications from the hardware they run on. Hardware vir- of resources to each application can be varied dynamically to ad- tualization and operating system level virtualization are two promi- just to changing application workloads. Furthermore, virtualiza- nent technologies that enable this. Containers, which use OS virtu- tion enables multi-tenancy, which allows multiple instances of vir- alization, have recently surged in interest and deployment. In this tualized applications (“tenants”) to share a physical server. Multi- paper, we study the differences between the two virtualization tech- tenancy allows data centers to consolidate and pack applications nologies. We compare containers and virtual machines in large data into a smaller set of servers and reduce operating costs. Virtualiza- center environments along the dimensions of performance, man- tion also simplifies replication and scaling of applications. ageability and software development. There are two types of server virtualization technologies that We evaluate the performance differences caused by the different are common in data center environments—hardware-level virtual- virtualization technologies in data center environments where mul- ization and operating system level virtualization. Hardware level tiple applications are running on the same servers (multi-tenancy). virtualization involves running a hypervisor which virtualizes the Our results show that co-located applications can cause performance server’s resources across multiple virtual machines. Each hardware interference, and the degree of interference is higher in the case of virtual machine (VM) runs its own operating system and applica- containers for certain types of workloads. We also evaluate differ- tions. By contrast, operating system virtualization virtualizes re- ences in the management frameworks which control deployment sources at the OS level. OS-level virtualization encapsulates stan- and orchestration of containers and VMs. We show how the differ- dard OS processes and their dependencies to create “containers”, ent capabilities exposed by the two virtualization technologies can which are collectively managed by the underlying OS kernel. Ex- affect the management and development of applications. Lastly, we amples of hardware virtualization include Xen [21], KVM [34], evaluate novel approaches which combine hardware and OS virtu- and VMware ESX [18]. Operating system virtualization is used alization. by Linux containers (LXC [7]), Docker [2], BSD Jails [32], and Solaris Zones [23]. Both types of virtualization technologies also have management 1. INTRODUCTION frameworks that enable VMs and applications to be deployed and Modern enterprises increasingly rely on IT applications for their managed at data center scale. Examples of VM management frame- business needs. Today’s enterprise IT applications are hosted in works include commercial offerings like vCenter [19] and open data centers—servers and storage that provide compute, storage source frameworks like OpenStack [8], CloudStack [11]. Kuber- and network resources to these applications. Modern data centers netes [5] and Docker Swarm [12] are recent container management are increasingly virtualized where applications are hosted on one or frameworks. more virtual machines that are then mapped onto physical servers While hardware virtualization has been the predominant virtual- in the data center. ization technology for deploying, packaging, and managing appli- Virtualization provides a number of benefits. It enables a flexible cations, containers (which use operating system virtualization) are increasingly filling that role due to the popularity of systems like Docker [2]. Containers promise low-overhead virtualization and improved performance when compared to VMs. Despite the surge of interest in containers in enterprise environments, there is a dis- tinct lack of performance comparison studies which quantify and compare the performance benefits of containers and VMs. Previ- ous research [20, 26] has compared the two technologies, and our work expands on them and provides a multi-dimensional perfor- ACM ISBN 978-1-4503-2138-9. mance comparison of containers and VMs. DOI: 10.1145/1235 1 Given these trends, in this paper we ask the following questions: Virtual Machine-1 Virtual Machine-2 1. From a data center operator’s perspective, what are the ad- Application Application Container-1 Container-2 vantages and disadvantages of each virtualization technol- Libraries Libraries ogy from the perspective of application performance, man- Application Application ageability and deployment at scale? Guest OS Guest OS Libraries Libraries 2. Under what scenarios is one technology more suitable than Virtual H/W Virtual H/W Operating System Kernel the other? Hypervisor To answer these questions, we conduct a detailed comparison Hardware Hardware of hardware and OS virtualization. While some of our results and (a) Virtual Machines (b) Containers observations are specific to the idiosyncrasies of the platforms we chose for our experimental evaluation, our goal is to derive gen- Figure 1: Hardware and operating system virtualization. eral results that are broadly applicable to the two types of vir- tualization technologies. We choose open source platforms for our evaluation—Linux containers (LXC) and KVM (a Linux-based effort manner. The hypervisor is also responsible for isolation. Iso- type-2 hypervisor) , and our method involves comparing four con- lation among VMs is provided by trapping privileged hardware ac- figurations that are common in data center environments: bare- cess by guest operating systems and performing those operations metal, containers, virtual machines, and containers inside VMs. in the hypervisor on behalf of the guest OS. Examples of hardware Our comparative study asks these specific questions: virtualization platforms include VMware ESXi, Linux KVM and Xen. 1. How do these two virtualization approaches compare from a resource isolation and overcommitment perspective? 2.2 Operating System Virtualization 2. How do these approaches compare from the perspective of Operating system virtualization involves virtualizing the OS ker- deploying many applications in VMs/containers at scale? nel rather than the physical hardware (Figure 1). OS-level virtual machines are referred to as containers. Each container encapsu- 3. How do these approaches compare from the application life- lates a group of processes that are isolated from other containers cycle perspective and how it affects how developers interact or processes in the system. The OS kernel is responsible for imple- with them? menting the container abstraction. It allocates CPU shares, memory and network I/O to each container and can also provide file system 4. Can approaches which combine these two technologies (con- isolation. tainers inside VMs and lightweight VMs) enable the best of Similar to hardware virtualization, different allocation strategies both technologies to be reached? may be supported such as dedicated, shared and best effort. Con- tainers provide lightweight virtualization since they do not run their Our results show that co-located applications can cause perfor- own OS kernels, but instead rely on the underlying kernel for OS mance interference, and the degree of interference is higher in the services. In some cases, the underlying OS kernel may emulate a case of containers for certain types of workloads (Section 4). We different OS kernel version to processes within a container, a fea- also evaluate differences in the management frameworks which ture often used to support backward OS compatibility or emulating control deployment and orchestration of containers and VMs (Sec- different OS APIs. tion 5). We show how the different capabilities exposed by the Many OS virtualization techniques exist including Solaris Zones, two virtualization technologies can affect the management and de- BSD-jails and Linux LXC. The recent emergence of Docker, a velopment of applications (Section 6). Lastly, we evaluate novel container platform similar to LXC but with a layered filesystem approaches which combine hardware and OS virtualization (Sec- and added software engineering benefits, has renewed interest in tion 7). container-based virtualization for data centers and the cloud. Linux containers in particular employ two key features: 2. BACKGROUND Cgroups. Control groups [6] are a kernel mechanism for control- In this section we provide some background on the two types of ling the resource allocation to process groups. Cgroups exist for virtualization technologies that we study in this paper. each major resource type: CPU, memory, network, block-IO, and devices. The resource allocation for each of these can be controlled 2.1 Hardware Virtualization individually, allowing the complete resource limits for a process or Hardware virtualization involves virtualizing the hardware on a a process