Performance Exploration of Virtualization Systems Joel Mandebi Mbongue Danielle Tchuinkou Kwadjo Christophe Bobda University of Florida University of Florida University of Florida Gainesville, Florida Gainesville, Florida Gainesville, Florida [email protected] [email protected] [email protected] ABSTRACT 3 User App 3 Guest App 3 Guest App Virtualization has gained astonishing popularity in recent decades. 2 2 2 It is applied in several application domains, including mainframes, 1 1 1 VMM Kernel VMM 0 Host Kernel Privileges Privileges personal computers, data centers, and embedded systems. While 0 0 Privileges the benefits of virtualization are no longer to be demonstrated, it Hardware Hardware Hardware often comes at the price of performance degradation compared to (a) (b) (c) native execution. In this work, we conduct a comparative study on the performance outcome of VMWare, KVM, and Docker against Figure 1: x86 Privilege Ring and Virtualization. (a) Typical compute-intensive, IO-intensive, and system benchmarks. The ex- configuration in environment with no virtualization. The periments reveal that containers are the way-to-go for the fast kernel runs at level 0 and applications run at level 3. (b) execution of applications. It also shows that VMWare and KVM Corresponds to bare-metal virtualization stacks. There is no perform similarly on most of the benchmarks. host operating system, the virtual machine monitor (VMM) runs at level 0 and guest applications are at level 3. (c) De- KEYWORDS ployment of hosted VMMs. The host kernel runs at level 0, Virtualization, Containers, KVM, VMware, Docker the VMM at level 1, and the guests at level 3. 1 INTRODUCTION the performance that can be achieved against IO-intensive (such Virtual machines (VM) have been introduced early in the 1960s by as applications intensively accessing the disk), memory-intensive IBM to consolidate the hardware and decrease exploitation costs [7]. (such as matrix-based applications), and compute-intensive bench- The mainframes were sold at about $2.9 million (equivalent to about marks (such as high-performance applications). We also evaluate $25 million in 2020) and rented for $63,500 (about $553,417 in 2020) the overhead introduced by virtualization technologies against na- per month in a typical configuration, making computing systems tive executions. only accessible to a small range of customers [12, 23]. A VM could be seen as an instance of the physical machine in which the users had 2 BACKGROUND the illusion of fully owning the hardware. In reality, it was just a way to transparently share resources and run workloads from different 2.1 Type of Virtual Machine Monitors users in an isolated way on the same hardware. A few decades VMs have several advantages among which easy maintenance, fast later, researchers investigated models, challenges, and solutions recovery from fault, rapid provisioning and domain isolation [2]. to efficiently implement “virtual sub-environments” in physical They allow running multiple operating systems simultaneously machines [4]. The VM abstraction then provided concurrent and on the same machine. Furthermore, they support the execution of interactive access to the underlying hardware. systems with entirely different instruction set architectures than The continuous innovation in virtualization technology has led that of the underlying hardware. VMs typically run above a software to the emergence of an ecosystem of products ranging from VMs called "Virtual Machine Monitor" (VMM) or simply hypervisor. It running on personal computers to enterprise and commercial sys- controls the run-time resources of the VMs and ensures proper arXiv:2103.07092v1 [cs.DC] 12 Mar 2021 tems running in the cloud. Virtualization concepts are also applied execution of privileged instructions. beyond traditional hardware devices such as processors, memory, The x86 architecture separates processor privileges with a pro- disk, and network cards. As example, some research propose to vir- tection ring or levels [3]. It is a mechanism that protects data and tualize Field-Programmable Gate Arrays (FPGA) for cloud and data restricts operations that programs can run. Each program that ex- center applications [14–16]. Graphic Processing Units (GPU) are ecutes in an x86 system is assigned to a specific ring or level that also provisioned as part of virtual resource pools [10, 11]. Among defines the access privileges on system resources. Figure 1shows the most common virtualization softwares are VirtualBox, KVM, the different privilege levels available in x86 architectures. Typi- QEMU, Xen, VMware workstation, and container engines such as cally, level 0 is reversed for the operating system (OS) services that Docker and LXD. The emergence of multiple virtualization sys- directly interface with the hardware (kernel mode). Levels 1 and 2 tems supporting hardware consolidation in personal computers, are mostly unused and are reserved for some drivers and middle- embedded systems, and cloud-scale deployments raise the need ware. User applications run at level 3 (user mode) [3]. In Figure 1(a), for architecture classification and performance evaluation. In the no virtualization is implemented. The user applications run at level context of this work, we study the architectures of state-of-the- 3 and the kernel of the OS handles privileged instructions at level 0. art virtualization systems and provide a quantitative evaluation of Executing at level 0 allow the kernel to directly access and control VM Host World VMM World Guest App VMM (VMware) Guest OS VM QEMU Guest App Virt. Device POSIX vcpu1 ... vcpuN Thread Guest OS Front. Driver Apps VM App irtIO V Host OS VM Driver Control Transfer KVM (KVM.ko) Host OS Kernel Back. Driver Hardware Hardware Figure 2: VMware Workstation Architecture Figure 3: Overview of the KVM-QEMU Virtualization Archi- tecture the hardware. Depending on how far apart the VMM is from the actual hardware in the x86 privilege levels, we consider two types of processors with virtualization extensions such as Intel VT or AMD- hypervisors [8, 9]: (1) Type-1 hypervisors (bare metal): the VMM V. To emulate processors and IO devices, KVM is combined with is installed directly above the hardware (see Figure 1(b)). Examples QEMU (Quick Emulator) [3]. IO communication between the virtual of such VMMs include Xen and Linux enabled by Kernel-based and physical system is done through VirtIO. VirtIO is an abstrac- (KVM) [19]. The VMM is responsible from emulating the privileged tion of IO devices implemented by Rusty Russel for communication instructions launched in the guest space. (2) Type-2 hypervisor interfaces between guests and host in paravirtualized architectures. (hosted): in this configuration, the VMM is installed in the host KVM uses VirtIO as paravirtualized device drivers since kernel OS (see Figure 1(c)). An example of this category is VMware Work- version 2.6.25 [3, 20]. Figure 3 highlights the key components of the station. Privileged instructions in the guest space typically cause a KVM-QEMU virtualization. To execute guest applications on the "world switch" to the host kernel under the supervision of the VMM. physical hardware, QEMU creates POSIX threads that represent the In general, a set of applications or/and drivers implemented in the virtual CPUs. It has the advantage of making virtual applications VMM are used to access kernel privileged instructions. appear as processes in the host environment. The guest applications are run via KVM kernel modules that provide extension support for 2.2 VMware Workstation hardware virtualization such as Intel VMX [21]. Specifically, QEMU VMware Workstation is a Type-2 hypervisor that runs on x86 pro- opens the device file /dev/kvm exposed by KVM kernel module cessors. It supports Windows and Linux hosts, and allows users and runs a set of ioctls() functions. These functions allow setting to run multiple VMs on a single machine [1]. It virtualizes IO de- and updating the state of the registers of each virtual CPU in the vices using a hosted IO model which consists in taking advantage QEMU internal data structure, thus ensuring a smooth execution of pre-existing support in the host OS. This approach has several of guest applications [3]. This whole emulation however comes advantages among which application portability and consistency. with a considerable overhead. In a comparative study, Weber et .al It also delivers near native performance for CPU-intensive work- reported that QEMU was up to 5× slower than native environment loads. Figure 2 summarizes the architecture of VMware workstation. on some compute-intensive applications [22]. Non-privileged instructions from the guest can run natively on the hardware without interference from the VMM. On the other hand, 2.4 Containers: Docker when guest applications issue privileged instructions, the VMM 2.4.1 Containers. Containers are virtualization technologies in traps and emulates. Specifically, the VMM requests a "world switch" which the virtual environment directly runs above the host OS. from the VM Driver. Next, the VMM provides data to the VM App. They run within a container engine instead of an hypervisor. They The VM App is then in charge of mapping the virtual requests to are not designed to run a complete systems, but focus at the ap- host system calls [13]. After completing the system calls, the VM plication level. Containers are developed to reduce the footprint Driver returns the control to the VMM. The VMM collects the re- of systems, especially those that do not need heavy virtualization sults from the VM App and passes them to the VM. The VM can infrastructures. Figure 4(a) and (b) show the typical virtualization then resume its normal execution. stacks for VMs. Next, Figure 4(c) illustrate the key difference be- tween container and VM stacks. It resides in that containers only 2.3 Kernel-based Virtual Machine run applications on top of a container engine instead of a hypervisor. Kernel-based Virtual Machine (KVM) is a virtualization module Containers only need application binaries and a run-time engine, present in Linux releases since kernel version 2.6.20.