Intel Technology Journal, Volume 10, Issue 3, 2006 Extending Xen* with Intel® Virtualization Technology Yaozu Dong, Core Software Division, Intel Corporation Shaofan Li, Core Software Division, Intel Corporation Asit Mallick, Core Software Division, Intel Corporation Jun Nakajima, Core Software Division, Intel Corporation Kun Tian, Core Software Division, Intel Corporation Xuefei Xu, Core Software Division, Intel Corporation Fred Yang, Core Software Division, Intel Corporation Wilfred Yu, Core Software Division, Intel Corporation Index words: Xen, Virtualization, Hypervisor, Intel® VT, virtual machine monitor ABSTRACT “paravirtualization,” and it delivers near native performance for the guest OS, only if the guest OSs Xen* is an open source virtual machine monitor (VMM) source code can be modified. developed at the University of Cambridge to support operating systems (OSs) that have been modified to run Xen versions 1.0 and 2.0 use paravirtualization * on top of the monitor. Intel has extended the Xen VMM techniques to support 32-bit platforms and Linux guests. ∆ to use Intel® Virtualization Technology (VT) to support They use the standard IA-32 protection and unmodified guest OSs also. This was done for IA-32 segmentation architecture for system resource Intel® Architecture processors as well as Itanium® virtualization. The hypervisor runs in the highest architecture processors. privilege level ring 0 and has full access to all memory on the system. Guest OSs use privilege levels 1, 2, and 3 In this paper we describe the changes that have been as they see fit. Segmentation is used to prevent the guest made to Xen to enable this support. We also highlight OS from accessing the Xen address space. the optimizations that have been made to date to deliver good virtualized performance. Xen 3.0 is the first open-source VMM that uses Intel Virtualization Technology (VT) to support unmodified INTRODUCTION guest OSs as well as paravirtualized guest OSs. Xen 3.0 also added support for 64-bit platforms and 64-bit guests Xen is an open source virtual machine monitor (VMM) [9]. Page-level protection is used to protect the 64-bit that allows the hardware resources of a machine to be hypervisor from the guest. virtualized and dynamically shared between OSs running on top of it [1]. Each virtual machine (VM) is called a In this paper, we begin with a brief overview of Intel VT Domain, in Xen terminology. Xen provides isolated and then we explain how we extended Xen to take execution for each domain, preventing failures or advantage of VT. We highlight key virtualization issues ® Φ malicious activities in one domain from impacting for IA-32, Intel EM64T , and Itanium processors and another domain. The Xen hypervisor and Domain0 explain how they are addressed in Xen 3.0. Finally, we (Dom0) are a required part of any Xen-based server. highlight some of the changes that have been made to the Multiple user domains, called DomainU in Xen hypervisor and the device models to improve terminology, can be created to run guest OSs. performance. Unlike the full virtualization solutions offered by the INTEL® VIRTUALIZATION IBM VM/370*, or VMware’s ESX* and Microsoft’s Virtual PC product*, Xen began life as a VMM for guest TECHNOLOGY OSs that have been modified to run on the Xen Intel VT is a collection of processor technologies that hypervisor. User applications within these OSs run as is, enables robust execution of unmodified guest OSs on i.e., unmodified. This technique is called Intel VT-enhanced VMMs [2]. VT-x defines the Extending Xen* with Intel® Virtualization Technology 193 Intel Technology Journal, Volume 10, Issue 3, 2006 extensions to the IA-32 Intel Architecture [3]. VT-i address bits are available. When the guest OS is running defines the extensions to the Intel Itanium architecture with PSR.vm = 1, the uppermost implemented virtual- [4]. address bit is made unavailable to the guest. Instruction or data fetches with any of these address bits set will VT-x augments IA-32 with two new forms of CPU trigger unimplemented data/instruction address faults or operation: virtual machine extensions (VMX) root unimplemented instruction address traps. This provides operations and VMX non-root operations. The transition the VMM a dedicated address space that guest software from VMX root operation to VMX non-root operation is cannot access. called a VM entry. The transition from a VMX non-root operation to VMX root operation is called a VM exit. VT-i also defines the processor abstraction layer (PAL) interfaces that can be used by the VMM to create and A virtual-machine control structure (VMCS) is defined manage VMs. A Virtual Processor Descriptor (VPD) is to manage VM entries and exits, and it controls the defined to represent the resources of a virtual processor. behavior of instructions in a non-root operation. The PAL procedures are defined to allow the VMM to VMCS is logically divided into sections, two of which configure logical processors for virtualization operations are the guest-state area and the host-state area. These and to suspend or resume virtual processors. PAL run- areas contain fields corresponding to different time services are defined to support performance-critical components of processor state. VM entries load VMM operations. processor state from the guest-state area. VM exits save processor state to the guest-state area and then load processor state from the host-state area. EXTENDING XEN* WITH INTEL VT Xen 3.0 architecture (Figure 1) has a small hypervisor The VMM runs in root operation while the guests run in kernel that deals with virtualizing the CPU, memory, and VMX non-root operation. Both forms of operation critical I/O resources, such as the interrupt controller. support all four privilege levels (i.e., rings 0, 1, 2, and Dom0 is a paravirtualized Linux that has privileged 3). The VM-execution control fields in the VMCS allow access to all I/O devices in the platform and is an the VMM to control the behavior of some instructions in integral part of any Xen-based system. Xen 3.0 also VMX non-root operation and the events that will cause includes a control panel that controls the sharing of the VM exits. Instructions like CPUID, MOV from CR3, processor, memory, network, and block devices. Access RDMSR, and WRMSR will trigger VM exits to the control interface is limited to Dom0. Multiple user unconditionally to allow the VMM to control the domains, called DomainU (DomU) can be created to run behavior of the guest. paravirtualized guest OSs. Dom0 and DomU OSs use VT-i expands the Itanium processor family (IPF) to hypercalls to request services from the Xen hypervisor. enable robust execution of VMs. A new processor status When Intel VT is used, fully virtualized domains can be register bit (PSR.vm) has been added to define a new created to run unmodified guest OSs. These fully operating mode for the processor. The VMM runs with virtualized domains are given the special name of HVMs this bit cleared while the guest OS runs with it set. (hardware-based virtual machines). Xen presents to each Privileged instructions, including non-privileged HVM guest a virtualized platform that resembles a instructions like thash, ttag and mov cupid that may classic PC/server platform with a keyboard, mouse, reveal the true operating state of the processor, trigger graphics display, disk, floppy, CD-ROM, etc. This virtualization faults when operating in this mode. virtualized platform support is provided by the Virtual The PSR.vm bit also controls the number of virtual- I/O Devices module. address bits that are available to software. When a VMM In the following sections we describe the extensions to is running with PSR.vm = 0, all implemented virtual- each of these Xen components. Extending Xen* with Intel® Virtualization Technology 194 Intel Technology Journal, Volume 10, Issue 3, 2006 Domain0: Para-Virtualization DomainU: Para-Virtualization HVM (Hardware Virtual Machine) Domain Domain Domain Virtual I/O Virtual I/O Control Devices App App App App App Panel FE Virtual Drivers Unmodified OS F Backend Backend Drivers Drivers Drivers Drivers Virtual Virtual ront en ront Device Device Virtual Virtual Native Native driver driver d Guest BIOS Xenlinux Xenlinux Virtual Platform Hypercall/Event VM Exit/Entry Xen Hypervisor Virtual I/O Devices Virtual MMU Virtual CPU Local IO APIC, PIT Platform with Hardware-Based Virtualization (e.g. Intel® Virtualization Technology on IA-32, EM64T, IPF, aka IA-64) Figure 1: Xen 3.0 architecture Control Panel (MADT). The BIOS and the early OS loader expect to run in real mode. To create the environment needed by We have extended the control panel to support creating, these codes, we use VMXAssist to configure the VT-x controlling, and destroying HVM domains. The user can guest to execute in virtual-8086 mode. Instructions that specify configuration parameters such as the guest cannot be executed in this mode are intercepted and memory map and size, the virtualized disk location, emulated with a software emulator. network configuration, etc. For VT-i, we developed a guest firmware using the The control panel loads the guest firmware into the Intel® Platform Innovation Framework for Extensible HVM domain and creates the device model thread Firmware Interface (EFI). This guest firmware provides (explained later) that will run in Dom0 to service all EFI boot services required by IPF guest OSs. It is input/output (I/O) requests from the HVM guest. The compatible with the Developer’s Interface Guide for 64- control panel also configures the virtual devices seen by bit Intel® Architecture-based Servers (DIG64) and the HVM guest, such as the interrupt binding and the provides the System Abstraction Layer (SAL), ACPI 2.0, PCI configuration.