Virtualization and Containerization of Application Infrastructure: A Comparison Mathijs Jeroen Scheepers University of Twente P.O. Box 217, 7500AE Enschede The Netherlands [email protected] ABSTRACT Modern cloud infrastructure uses virtualization to isolate applications, optimize the utilization of hardware resources and provide operational flexibility. However, conventional virtualization comes at the cost of resource overhead. Container-based virtualization could be an alternative as it potentially reduces overhead and thus improves the uti- lization of datacenters. This paper presents the results of a marco-benchmark performance comparison between the two implementations of these technologies, namely Xen Figure 1. A schematic overview of virtual ma- and LXC, as well as a discussion on their operational flex- chines in a datacenter. ibility. Keywords et al. [7], expects hypervisors to provide isolation and portability. The Xen [4] hypervisor is a popular technol- Hypervisor, Virtualization, Cloud computing, Application ogy and widely used at the moment. infrastructure, LXC, Xen, Container-based virtualization With recent developments around Docker [2] and LXC [3] there now seems to be a viable alternative to the hyper- 1. INTRODUCTION visor and traditional virtualization for application infras- According to Zhang et al. [20] virtualization technology tructures. Linux Containers (LXC) is a kernel technol- is an essential part of modern cloud infrastructure, such ogy that is able to run a multitude of processes, each in as Amazon's Elastic Compute Cloud (EC2) and Google's their own isolated environment. This technique is called App Engine. These days, most cloud computing datacen- container-based virtualization. Docker is a tool that makes ters run hypervisors on top of their physical machines. A it easy to package an application and all of its depen- hypervisor is a piece of computer software that creates dencies into such containers. Merkel [13] explains that and runs virtual machines. With these hypervisors, and \Docker is . the lightweight and nimble cousin of virtual the virtual machines that run on them, system adminis- machines". trators are able to optimize the use of available physical There is a school of thought, popular within the Linux resources and confine individual parts of application in- community, that claims that hypervisors originally were frastructure. A typical setup is displayed schematically in developed due to the Linux kernel's inability to provide Figure 1. With the use of virtualization, resources can be superior resource isolation and effective scalability [11]. consumed more effectively than conventional bare-metal The container could be the solution. setups, which use physical machines for isolating different parts of application infrastructure. Still efficiency could be The multiple kernels running on a hypervisor use a rather increased even further. A hypervisor will run multiple ker- large fraction of the machines physical resources. LXC nels on a single physical machine, therefore the isolation of does not seem to have this problem. Combined with the applications and processes is expensive. Mills [14] stated tooling Docker provides, they provide the flexibility a mod- that 1,500 terawatt-hours of power per year is used to ern system administrator expects, like easy provisioning power cloud computing datacenters, that is about 10% of and image construction. The way LXC isolates processes the worlds energy consumption, and this number is climb- could reduce overhead on major software deployments in ing. If compute resources could be used more efficiently deployment time, application portability as well as physi- that could have a big impact. cal resource usage. With a kernel feature, LXC is able to isolate processes and allocate resources without the use of The cloud computing paradigm, as described by Buyya hardware emulation. The technology is leveraged by the Docker and CoreOS [1] software, which enables the cre- Permission to make digital or hard copies of all or part of this work for ation of complex and portable application infrastructures. personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that Where Docker provides LXC with the deployment tooling copies bear this notice and the full citation on the first page. To copy oth- it needs, CoreOS provides the underlying host operating erwise, or republish, to post on servers or to redistribute to lists, requires system and makes it possible to setup a cluster of machines prior specific permission and/or a fee. on which containers can be managed and migrated. 21st Twente Student Conference on IT June 23rd, 2014, Enschede, The Netherlands. By using a single kernel per bare-metal machine, container- Copyright 2014, University of Twente, Faculty of Electrical Engineer- based virtualization could shift the cloud paradigm away ing, Mathematics and Computer Science. from hypervisor-based virtual machines. 1 Table 1. Virtualization technologies Full Para Container-based KVM Xen LXC VMWare OpenVZ VirtualBox VServer UML LXC differs in a lot of ways from the traditional hyper- visor. This paper will focus on two differences: physical resource impact and operational flexibility. This paper is structured as follows: In Section 2 we will elaborate on the working of the Xen hypervisor, LXC, Docker and CoreOS. Next, in Section 3 we discuss related work and the contribution of this paper. Figure 2. A schematic overview of a machine run- We will compare the physical performance of a single ma- ning the Xen hypervisor. chine, running the same application using two different isolation techniques. Namely, isolation through virtual- virtual machine on the Xen hypervisor could run a mod- ization and through containerization. Do containers re- ified kernel in order to provide better performance and ally have a performance benefit, and if so, how significant reduce overhead. The hypervisor is installed directly into is that benefit? These questions will be answered in Sec- the bootloader. tion 4 by analyzing the results of several benchmarks. The virtual machines, running on top of the hypervisor, There are still several research challenges when talking are called domains or guests. A special domain, called about cloud computing. Among others, improving auto- domain0, controls the system (Dom0). This domain has mated service provisioning, machine migrations and server the capability to setup the environment. It could contain consolidation [20]. In Section 5, we discuss these and show tools for the setup of networking, provisioning of new vir- how Xen and Docker are able to help with these challenges tual machines and migrating them. and how their solutions differ. The other domains are what is called underprivileged to Finally, in Section 6 the results of the performance com- domain0. Therefore they are called DomU. These DomU parison as well as the operational comparison will be dis- domains can either be para-virtualized (PV) or hardware- cussed and related. assisted (HVM). The PV-domains require a optimized ker- nel, whereas the HVM-domains require no kernel modifi- 2. BACKGROUND cation but do require x86's virtualization support (Intel Modern application infrastructure techniques and method- VT-X, AMD-SVM). This architecture support is not re- ologies incentivize an accelerated adoption of cloud com- quired when running a PV virtual machine. puting technologies as well as various virtualization tech- Since Xen only provides the hypervisor technology, we nologies. For example the DevOps [10] software develop- still need a management operating system to be installed ment methodology and techniques that require scriptable on Dom0. The XenServer is an implementation for the infrastructure. Dom0 management system. It provides extended tooling The virtualization technologies that have emerged mostly to provision, manage, monitor and migrate virtual ma- focus on the Linux kernel and can be split up into three chines. This is the domain on which XenServer could be categories: full-virtualization, para-virtualization and installed. With domain0 being a virtual environment, a container-based virtualization. Para-virtualization modi- XenServer installation is itself running on a virtual ma- fies the kernel of virtual machines slightly to optimize for chine. Figure 2 shows a schematic overview of a machine performance in the virtual environment. Full-virtualization running the Xen hypervisor with XenServer installed on does not require kernel adjustments. Container-based vir- Dom0. tualization does not use a kernel at all. Xen has been in development for more than 12 years and Table 1 shows a selection of various technologies and their thus can be considered a mature technology. Xen technol- categorization. These are the kind of technologies used in ogy is widely used, for example by Amazone Web Services, IaaS (infrastructure as a service) solutions and PaaS (plat- Google, Rackspace, Oracle, Cisco and Citrix [5]. form as a service) solutions like Amazone Elastic Compute Cloud, Google App Engine, DotCloud and Open Shift. 2.2 LXC Since application infrastructure can be diverse, there is no Linux Containers (LXC) provides lightweight operating single best solution for all of these services. Rather, each system virtualization and is relatively new to the other service or application has its own specific requirements. technologies listed in Table 1. Unlike Xen, LXC does not We will be comparing two