KVM/ARM: The Design and Implementation of the Linux ARM Hypervisor
Christoffer Dall and Jason Nieh ~ 1.2 billion ~ 300 million
Source: Gartner Market Report, Jan 2014 ARM Servers ARM Network Equipment Virtualization Extensions Key Challenges != Virtualization Extensions VT-x
• No PC-standard on ARM Key Contributions
• Split-Mode Virtualization
• Implemented and evaluated KVM/ARM
• Upstreamed implementation to mainline linux Outline
• ARM Virtualization Extensions
• Comparison to x86
• Split-Mode Virtualization
• Results
• Experiences ARM Virtualization Extensions
• Provides virtualization in 4 key areas:
• CPU Virtualization
• Memory Virtualization
• Interrupt Virtualization
• Timer Virtualization ARM Virtualization Extensions CPU Virtualization
User
Kernel ARM Virtualization Extensions CPU Virtualization
User
Kernel
Hyp ARM Virtualization Extensions CPU Virtualization
VM 0 VM 1
User User
Kernel Kernel
Hypervisor ARM Virtualization Extensions CPU Virtualization
• Separate stack
• Different control registers
• Different page table format
• Controls Memory, Interrupt, and Timer virtualization
• Can be completely bypassed ARM Virtualization Extensions Memory Virtualization
0 4 GB Virtual User space application Kernel Addresses Stage-1 Page Tables
MMU
0 2^40 Intermediate Physical RAM Devices Addresses (IPA) Stage-2 Page Tables
MMU
0 2^40 Physical RAM Devices Addresses ARM Virtualization Extensions Interrupt Virtualization
• Interrupts are an important part of any system
• The mechanism to notify CPU of events
• CPUs interact with an interrupt controller
• VMs accessing an interrupt controller traps
• Hurts performance ARM Virtualization Extensions Interrupt Virtualization
CPU IRQ Interface CPU 0
Dist. CPU ACK Interface CPU 1
GIC ARM Virtualization Extensions Interrupt Virtualization
CPU IRQ Interface CPU 0 Virtual CPU VIRQ Interface Dist. CPU ACK Interface CPU 1 Virtual CPU ACK Interface GIC x86
Root Non-Root (VM)
VM ENTRY
VM EXIT
VMCS ARM
User Space
Kernel
Trap
Hypervisor (HYP) ARM
User Space
Kernel Trap ? Hypervisor (HYP) Standalone hypervisor on ARM
• Reimplement basic OS functionality
• Hardware support is a pain! Hosted Hypervisor on ARM
Kernel
ARM Hardware Hosted Hypervisor on ARM
VM VM Guest Kernel Guest Kernel
Kernel Hypervisor
ARM Hardware Hosted Hypervisor on ARM
HYP mode? VM VM Guest Kernel Guest Kernel
Kernel Hypervisor
ARM Hardware Split-Mode Virtualization
OS Kernel Hypervisor
Kernel mode
HYP mode Split-Mode Virtualization
OS Kernel Hypervisor
Kernel mode Highvisor
HYP mode Lowvisor Split-Mode Virtualization
OS Kernel Hypervisor
Run VM Kernel mode Highvisor
HYP mode Lowvisor Split-Mode Virtualization
OS Kernel Hypervisor
Run VM Kernel mode Highvisor
Trap
HYP mode Lowvisor Split-Mode Virtualization
OS Kernel Hypervisor VM
Run VM Kernel mode Highvisor
Trap Trap
HYP mode Lowvisor Split-Mode Virtualization
OS Kernel Hypervisor VM
Kernel mode Highvisor
HYP mode Lowvisor Split-Mode Virtualization
OS Kernel Hypervisor VM
Kernel mode Highvisor
Trap
HYP mode Lowvisor Split-Mode Virtualization
OS Kernel Hypervisor VM
Kernel mode Highvisor
Trap Trap
HYP mode Lowvisor Split-Mode Virtualization
OS Kernel Hypervisor VM
Function call Kernel mode Highvisor
Trap Trap
HYP mode Lowvisor Split-Mode Virtualization
User Space VM User Space User mode
Kernel mode OS Kernel VM Kernel Hyper- visor HYP mode
ARM Hardware Split-Mode Virtualization
User Space VM User Space User mode
Kernel mode OS Kernel VM Kernel Hyper- visor HYP mode
ARM Hardware Split-Mode Virtualization
User Space VM User Space User mode
Kernel mode OS Kernel VM Kernel Hyper- visor HYP mode
ARM Hardware CPU Virtualization
OS Kernel Hypervisor VM
Highvisor
Kernel mode
HYP mode Lowvisor CPU Virtualization
OS Kernel Hypervisor VM
Run VM Highvisor
Kernel mode
HYP mode Lowvisor CPU Virtualization
OS Kernel Hypervisor VM
Run VM Highvisor
Kernel mode
HYP mode Lowvisor
Enables traps CPU Virtualization
OS Kernel Hypervisor VM
Highvisor
Kernel mode
HYP mode Lowvisor CPU Virtualization
OS Kernel Hypervisor VM
Highvisor
Kernel mode
HYP mode Lowvisor
Disables traps Memory Virtualization
OS Kernel Hypervisor VM
Highvisor
Kernel mode
HYP mode Lowvisor Memory Virtualization
OS Kernel Hypervisor VM
Highvisor
Kernel mode
HYP mode Lowvisor Configures Stage-2 Page Tables Memory Virtualization
OS Kernel Hypervisor VM
Highvisor
Kernel mode
HYP mode Lowvisor Configures Stage-2 Page Tables Memory Virtualization
OS Kernel Hypervisor VM
Highvisor
Kernel mode
HYP mode Lowvisor
Enables Stage-2 Translation Memory Virtualization
OS Kernel Hypervisor VM
Highvisor
Kernel mode
HYP mode Lowvisor Memory Virtualization
OS Kernel Hypervisor VM
Highvisor
Kernel mode
HYP mode Lowvisor
Disables Stage-2 Translation Interrupt Virtualization
OS Kernel Hypervisor VM
Highvisor
Kernel mode
HYP mode Lowvisor Interrupt Virtualization
OS Kernel Hypervisor VM
Highvisor
Kernel mode
HYP mode Lowvisor Prepare Virtual Interrupts Interrupt Virtualization
OS Kernel Hypervisor VM
Highvisor
Kernel mode
HYP mode Lowvisor Prepare Virtual Interrupts Interrupt Virtualization
OS Kernel Hypervisor VM
Highvisor
Kernel mode
HYP mode Lowvisor
Enable Virtual Interrupts Interrupt Virtualization
OS Kernel Hypervisor VM
Highvisor
Kernel mode
HYP mode Lowvisor Interrupt Virtualization
OS Kernel Hypervisor VM
Highvisor
Kernel mode
HYP mode Lowvisor
Disable Virtual Interrupts Split-Mode Virtualization
Pros Cons
• Existing OS functionality
• Existing hardware • Double-trap on support world-switch • HYP mode for virtualization Implementation
• Implemented KVM/ARM using split-mode virtualization
• Merged into mainline Linux v3.9 Experimental Setup
Arndale Board - Exynos 5250 MacBook Air Dual-Core Cortex A-15 @ 1.4 GHz Dual-Core Intel i7 @1.8 GHz eSATA SSD SATA SSD Experimental Setup
version 12.10 Kernel version 3.10 (Quantal Quetzal)
Same amount of memory Same kernel config across all runs Micro Benchmarks
Test ARM (cycles) x86 (cycles) Trap 27 632 Hypercall 5,326 1,336 Send virtual IPI 14,366 17,138 Ack virtual IPI 427 2,043 Macro Benchmarks
ARM x86 2.0
1.5
1.0
0.5
untar curl1k curl1g Apache MySQL hackbench memcached kernel compile Outline
• ARM Virtualization Extensions
• Comparison to x86
• Split-Mode Virtualization
• Results
• Experiences Maintainability is key Maintainability is key
Genuine fear of “Dump and Run” arch/arm
mach-aaec2000 mach-ixp4xx mach-omap2 mach-shark mach-at91 mach-keystone mach-orion5x mach-shmobile mach-bcm mach-kirkwood mach-picoxcell mach-socfpga mach-bcmring mach-ks8695 mach-pnx4008 mach-spear mach-clps711x mach-l7200 mach-prima2 mach-sti mach-davinci mach-lh7a40x mach-pxa mach-stmp378x mach-dove mach-loki mach-realview mach-stmp37xx mach-ebsa110 mach-mmp mach-rockchip mach-sunxi mach-ep93xx mach-msm mach-rpc mach-tegra mach-footbridge mach-mv78xx0 mach-s3c2400 mach-u300 mach-gemini mach-mvebu mach-s3c2410 mach-ux500 mach-h720x mach-mx1 mach-s3c2412 mach-versatile mach-highbank mach-mx2 mach-s3c2440 mach-vexpress mach-imx mach-mx25 mach-s3c2442 mach-virt mach-integrator mach-mx3 mach-s3c2443 mach-vt8500 mach-iop13xx mach-mxc91231 mach-s3c24a0 mach-w90x900 mach-iop32x mach-netx mach-s3c6400 mach-zynq mach-iop33x mach-nomadik mach-s3c6410 ! mach-ixp2000 mach-ns9xxx mach-s5pc100 ! mach-ixp23xx mach-omap1 mach-sa1100 Trust Trust
It’s about the people Linus Torvalds
Russell King Avi Kivity … arch/arm virt/kvm
…
Will Deacon Marc Zyngier (ARM Ltd.) (ARM Ltd.) Conclusion
• KVM/ARM was the first system to run unmodified guests on hardware
• Facilitated by split-mode virtualization
• Performance within 10% of native execution
• Implemented KVM/ARM and upstreamed to Linux
• Commercially supported and maintained Use and Contribute
• Web: http://systems.cs.columbia.edu/projects/kvm-arm
• Code: https://github.com/torvalds/linux
• Mailing list: [email protected]
• IRC: #kvm-arm on freenode