Operating System Extensions to Support Host Based Virtual Machines Samuel T. King and Peter M. Chen Department of Electrical Engineering and Computer Science University of Michigan [kingst,pmchen]@eecs.umich.edu Abstract: This paper presents two new operating sys- tem extensions that facilitate the fast execution of host based virtual machines: KTrace and MMA. KTrace provides a convenient and efficient mechanism for writing kernel modules that control the execution of user-level processes. MMA exposes more detail of the underlying memory management hardware, providing applications with access to high performance intra- address space protection and address space overloading functionality. These extensions are applied to UMLinux, an x86 virtual machine that implements Linux as a user-mode process. As a result, overhead for UMLinux is reduced from 33% for a CPU bound work- load and 819% - 1240% for system call and I/O inten- sive workloads, to 6% and 49% - 24% respectively. 1. Introduction Figure 1: Direct-on-hardware virtual machine. Originally developed by IBM in the 1960’s, vir- tual machines were used to provide software backward machine, secure services only need to trust the virtual compatibility on time-shared mainframe computers. machine monitor (VMM), not an entire operating sys- This amount of portability allowed users to run com- plete operating systems, meant for old hardware, on tem. This is beneficial because the VMM code size is new computers without modifications. Although this significantly smaller than a full-blown operating sys- level of portability is useful, virtual machines also tem, making verification much easier. exhibit behavior that makes them popular today. Vir- Although there are many different virtual tual machines provide software with the illusion that it machines available, running on various platforms, possesses exclusive access to the computer hardware. User-Mode Linux (UMLinux) running on x86 is the Therefore, multiple operating systems can run on the virtual machine CoVirt uses for security applications. same hardware in complete isolation. In addition to Researchers at the University of Erlangen-Nurnberg isolation, virtual machines can effectively multiplex developed UMLinux for fault injection experiments, hardware between several virtual machine instances in but the extremely small VMM implementation pro- a fair way. Because of these two properties, virtual vides a strong isolation barrier, making it an excellent machines have seen a resurgence of research activity in recent years. Among the current research projects in choice for security applications. However, the original the area are intrusion analysis [14] and prevention [22], UMLinux implementation suffers from poor perfor- fault injection [8], server farms [31], and secure mobile mance, making it unfeasible in a practical setting. The computing [23, 19]. main reason for inefficiencies is because UMLinux My research group, CoVirt, is interested in imple- sacrifices performance for VMM simplicity by using menting services below the virtual machine [11]. operating system abstractions, rather than accessing the Because they are implemented below a virtual hardware directly. Lack of efficient operating system 1 Figure 2: Host based virtual machine. Figure 3: The IBM VM/370 [3] is a classic virtual extensibility, and virtual memory abstractions that hide machine in that it implements a direct-on-hardware the power of the underling memory management virtual machine and exports a virtual instruction set haware are known to contribute to this performance that is identical to the underlying hardware. Like the degradation. In an effort to improve performance, I IBM VM/370, Denali [31] implements a direct-on- developed two new abstractions: KTrace and memory hardware virtual machine. However, Denali adds to management abstractions (MMA). KTrace is a mecha- and removes instructions from the virtual instruction nism through which kernel modules can register call- set, making the existing code base written for the back functions that are triggered by exceptions underlying architecture unusable. VMware [1] generated from a specified process. MMA unleashes employs a hybrid approach by using a host OS for the power of the x86 memory management hardware some services and manipulating hardware directly for others. SimOS [26] implements a complete guest- by providing access to page table manipulation func- VMM interface, without accessing hardware directly, tions, and hardware protection mechanisms. Together, running above the host OS. UMLinux [8] runs on top KTrace and MMA can be used to implement simple of the host OS, but deprecates certain instructions. VMMs, without sacrificing performance. These instructions are usually found in operating The remainder of this paper is organized as fol- systems, so operating systems require a UMLinux port, lows: Section 2 presents background information on but user-mode applications remain binary compatible. virtual machines and discusses the design tradeoffs of three modern virtual machines. Section 3 details the design of KTrace and MMA, and Section 4 present whereas virtual machines that rely on an operating sys- experimental results showing the improvements made tem for services are called host-based virtual machines by using KTrace and MMA on UMLinux. Section 5 (Figure 2). Although direct-on-hardware virtual discusses related work and section 6 concludes. machines have more direct control of the execution environment, host based virtual machines can be sim- 2. Virtual machines pler to implement by using well-known operating sys- As with any abstraction, the VMM is responsible tem abstractions. In the host based approach to for managing the hardware and other resources below virtualization, the underlying operating system is while providing an interface for virtual machine referred to as the host OS, whereas any operating sys- instances above. The interface below the VMM, called tem running on virtual hardware is called a guest OS. the platform-VMM interface, could include bare hard- Like the platform-VMM interface, the interface ware, an operating system, or any combination of the above the VMM can vary widely. This interface, called two. Virtual machines that run on bare hardware are the guest-VMM interface, is what virtual machines run called direct-on-hardware virtual machines (Figure 1), on. Some virtual machines implement a guest-VMM 2 interface that is identical to the underlying hardware operating systems already do well. This simplicity thus, allowing any code written for that particular hard- caused poor performance in old versions of VMware, ware to be run in the virtual machine. Other virtual but optimizations are used to mitigate the effects and machines add and subtract aspects of the underlying speed up I/O [28]. hardware. This modification of the instruction set In addition to VMM optimizations to improve I/O architecture (ISA) is done for various reasons ranging performance, several guest device drivers specific to from performance, to simplicity of VMM implementa- VMware are used. By using VMware device drivers tion. To show how different virtual machines define that know about running on virtual hardware, I/O per- these interfaces, several examples are given in Figure 3. formance approaches that of an operating system run- Virtual machine design is a tradeoff between gen- ning on bare hardware. VMware specific guest device erality, complexity, and performance. To illustrate this drivers can be viewed as a modification to the guest tradeoff, three modern virtual machines are discussed OS, and although VMware will work without it, the in detail: VMware, Denali, and UMLinux. performance is greatly enhanced using these modifica- tions. 2.1. VMware Memory management in VMware is carried out VMware is the most prevalent x86 virtual machine by the guest OS directly on the hardware. To insure today. By providing a complete implementation of the that there are no conflicts between host and guest mem- x86 ISA at the guest-VMM interface, VMware is a use- ory, VMware allocates and pins host RAM for use in ful tool with many different applications. Although the guest context. The guest then uses the allocated this complete guest-VMM interface is extremely gen- RAM. One aspect that complicates this is the VMM eral, it leads to a more complicated VMM. Because must reside in all guest contexts so it can gain control there are multiple operating systems running on the when needed, causing a small section of the guest lin- same hardware, the VMM must emulate certain ear address space to be emulated on each guest read instructions to correctly represent a virtual processor to and write. each virtual machine; instructions that must be emu- To gain control of the guest, VMware implements lated are called sensitive instructions. For performance interrupt service routines for all interrupts. Whenever reasons, virtual machines typically rely on the proces- guest code causes an exception, it is first examined by sor trap mechanism to run the VMM when a sensitive the VMM. Interrupts that correspond to I/O devices instruction is executed. However, x86 processors do are forwarded to the host so that they can be handled not trap on all sensitive instructions [25] so additional properly, while exceptions generated by guest applica- work must be done. To handle sensitive instructions tions (like system calls) are forwarded back to the that do not cause a trap, VMware uses a technique guest. called binary rewriting. In binary rewriting, all instruc- Overall, VMware has attained an excellent bal- tions are examined before being executed, and the ance between all virtual machine design factors. The VMM inserts a breakpoint in place of sensitive instruc- guest-VMM interface is true to x86 hardware, and tions. The breakpoint causes a trap to the VMM when using the host OS for I/O simplifies the VMM design. executed, where the instruction is emulated. Binary Optimizations to alleviate overhead have been imple- rewriting adds to VMM complexity in VMware, but mented, but the VMM does still carry significant com- provides a complete x86 virtual instruction set at the plexity because binary rewriting is used, and many guest-VMM interface.