KVM/ARM: Experiences Building the Linux ARM Hypervisor Christoffer Dall and Jason Nieh fcdall, [email protected] Department of Computer Science, Columbia University Technical Report CUCS-010-13 April 2013 Abstract ization. To address this problem, ARM has introduced hardware virtualization extensions in the newest ARM As ARM CPUs become increasingly common in mo- CPU architectures. ARM has benefited from the hind- bile devices and servers, there is a growing demand sight of x86 in its design. For example, nested page for providing the benefits of virtualization for ARM- tables, not part of the original x86 virtualization hard- based devices. We present our experiences building the ware, are standard in ARM. However, there are impor- Linux ARM hypervisor, KVM/ARM, the first full sys- tant differences between ARM and x86 virtualization ex- tem ARM virtualization solution that can run unmodified tensions such that x86 hypervisor designs may not be guest operating systems on ARM multicore hardware. directly amenable to ARM. These differences may also KVM/ARM introduces split-mode virtualization, allow- impact hypervisor performance, especially for multicore ing a hypervisor to split its execution across CPU modes systems, but have not been evaluated with real hardware. to take advantage of CPU mode-specific features. This allows KVM/ARM to leverage Linux kernel services and We describe our experiences building KVM/ARM, functionality to simplify hypervisor development and the ARM hypervisor in the mainline Linux kernel. maintainability while utilizing recent ARM hardware KVM/ARM is the first hypervisor to leverage ARM virtualization extensions to run application workloads in hardware virtualization support to run unmodified guest guest operating systems with comparable performance operating systems (OSes) on ARM multicore hardware. to native execution. KVM/ARM has been successfully Our work makes four main contributions. First, we in- merged into the mainline Linux 3.9 kernel, ensuring that troduce split-mode virtualization, a new approach to hy- it will gain wide adoption as the virtualization platform pervisor design that splits the core hypervisor so that it of choice for ARM. We provide the first measurements runs across different CPU modes to take advantage of the on real hardware of a complete hypervisor using ARM specific benefits and functionality offered by each CPU hardware virtualization support. Our results demonstrate mode. This approach provides key benefits in the con- that KVM/ARM has modest virtualization performance text of ARM virtualization. ARM introduces a new CPU and power costs, and can achieve lower performance and mode for running hypervisors called Hyp mode, but Hyp power costs compared to x86-based Linux virtualization mode has a different feature set from other CPU modes. on multicore hardware. Hypervisors provide many aspects of OS functionality, but standard OS mechanisms in Linux would have to be significantly redesigned to run in Hyp mode. Our split- 1 Introduction mode virtualization mechanism allows a hypervisor to use Hyp mode to leverage ARM hardware virtualization ARM-based devices are seeing tremendous growth features, but also run in normal privileged CPU modes, across smartphones, netbooks, and embedded comput- allowing it to coexist with other OS functionality. ers. While ARM CPUs have benefited from their advan- Second, we designed and implemented KVM/ARM tages in power efficiency in these markets, ARM CPUs from the ground up as an open source project that would also continue to increase in performance such that they be easy to maintain and integrate into the Linux ker- are now within the range of x86 CPUs for many classes nel. For example, by using our split-mode virtualiza- of applications. This is spurring the development of new tion, we can leverage the existing KVM hypervisor in- ARM-based microservers and an upward push of ARM terface in Linux and can reuse substantial pieces of ex- CPUs into traditional server and PC systems. isting kernel code and interfaces to reduce code duplica- Unlike x86-based systems, a key limitation of ARM- tion. KVM/ARM was accepted as the ARM hypervisor based systems has been the lack of support for virtual- of the mainline Linux kernel as of the Linux 3.9 kernel, 1 ensuring its wide adoption and use given the dominance 64-bit ARMv8 architecture. The Cortex-A15 [2] is an of Linux on ARM platforms. Based on our open source examples of current ARMv7 CPUs including hardware experiences, we offer some useful hints on transferring virtualization extensions. We present a brief overview of research ideas into implementations likely to be adopted the ARM virtualization extensions. by existing open source communities. Third, we demonstrate the effectiveness of 2.1 CPU Virtualization KVM/ARM on real multicore ARM hardware. Our results are the first measurements of a hypervisor using Figure 1 shows the CPU modes on the ARMv7 archi- ARM virtualization support on real hardware. We tecture, including TrustZone (Security Extensions) and compare against the standard widely-used Linux x86 a new CPU mode called Hyp mode. TrustZone splits the KVM hypervisor and evaluate its performance overhead modes into two worlds, secure and non-secure, which are for running application workloads in virtual machines orthogonal to the CPU modes. A special mode, monitor (VMs) versus native non-virtualized execution. Our mode, is provided to switch between the secure and non- results show that KVM/ARM achieves comparable secure worlds. Although ARM CPUs always power up performance overhead in most cases, and significantly starting in the secure world, ARM bootloaders typically lower performance overhead for two important appli- transition to the non-secure world at an early stage and cations, Apache and MySQL, on multicore platforms. secure world is only used for specialized use cases such These results provide the first comparison of ARM as digital rights management. TrustZone may appear and x86 virtualization extensions on real hardware to useful for virtualization by using the secure world for hy- quantitatively demonstrate how the different design pervisor execution, but this does not work because there choices affect virtualization performance. We show is no support for trap-and-emulate. There is no means to that KVM/ARM also provides power efficiency benefits trap operations executed in the non-secure world to the over Linux x86 KVM. secure world. Non-secure software can therefore freely Finally, we make several recommendations regarding configure, for example, virtual memory. Any software future hardware support for virtualization based on our running in the non-secure world therefore has access to experiences building and evaluating a complete ARM all non-secure memory, making it impossible to isolate hypervisor. We identify features that are important and multiple VMs running in the non-secure world. helpful to reduce the software complexity of hypervisor implementation, and discuss mechanisms useful to max- Non-Secure state Secure state imize hypervisor performance, especially in the context of multicore systems. PL0 PL0 This technical report describes our experiences de- User User signing, implementing, and evaluating KVM/ARM. Sec- PL1 PL1 tion 2 presents an overview of the ARM virtualization Kernel Kernel extensions and a comparison with x86. Section 3 de- PL2 scribes the design of the KVM/ARM hypervisor. Sec- Hyp tion 4 discusses the implementation of KVM/ARM and our experiences releasing it to the Linux community and Monitor Mode (Secure PL1) having it adopted into the mainline Linux kernel. Sec- tion 5 presents experimental results quantifying the per- formance and energy efficiency of KVM/ARM, as well Figure 1: ARMv7 CPU modes. as a quantitative comparison of real ARM and x86 vir- tualization hardware. Section 6 makes several recom- Hyp mode was introduced as a trap-and-emulate mendations about designing hardware support for virtu- mechanism to support virtualization in the non-secure alization. Section 7 discusses related work. Finally, we world. Hyp mode is a CPU mode that is strictly present some concluding remarks. more privileged than other CPU modes, user and ker- nel modes. Without Hyp mode, the OS kernel running in kernel mode directly manages the hardware and can 2 ARM Virtualization Extensions natively execute sensitive instructions. With Hyp mode enabled, the kernel continues running in kernel mode but Because the ARM architecture is not classically virtual- the hardware will instead trap into Hyp mode on various izable [20], ARM has introduced hardware virtualization sensitive instructions and hardware interrupts. To run support as an optional extension in the latest ARMv7 VMs, the hypervisor must at least partially reside in Hyp architecture [4] and a mandatory part of the upcoming mode. The VM will execute normally in user and ker- 2 nel mode until some condition is reached that requires TTBR VA intervention of the hypervisor. At this point, the hard- ware traps into Hyp mode giving control to the hyper- Stage-1 L1 visor, which can then manage the hardware and provide Table L2 the required isolation across VMs. Once the condition Table L3 is processed by the hypervisor, the CPU can be switched Page GPA back into user or kernel mode and the VM can continue VTTBR executing. To improve performance, ARM allows many traps to Stage-2 L1 be configured so they trap directly into a VM’s kernel Table L2 mode instead of going through Hyp mode. For exam- Table L3 Page L4 ple, traps caused by normal system calls or undefined Page PA exceptions from user mode can be configured to trap to a VM’s kernel mode so that they are handled by the guest OS without intervention of the hypervisor. This avoids Figure 2: Stage-1 and Stage-2 page table walk on going to Hyp mode on each system call or undefined ex- ARMv7 using the LPAE memory long format descrip- ception, reducing virtualization overhead. tors.