Pre-virtualization: soft layering for virtual machines Joshua LeVasseur† Volkmar Uhlig§ Matthew Chapman‡¶ Peter Chubb‡¶ Ben Leslie‡¶ Gernot Heiser‡¶ †University of Karlsruhe, Germany §IBM T. J. Watson Research Center, NY ‡National ICT Australia ¶University of New South Wales, Australia Abstract 1 Introduction Para-virtualization ties a guest operating system and a Although many hypervisor implementers strive to build hypervisor together, which restricts the system architec- high-performance virtual machine (VM) environments, ture; e.g., when Linux uses the Xen API, Linux is un- the constraints for supporting commodity operating sys- able to run on alternative hypervisors such as VMware, tems are enormous and force costly optimizations (e.g., Linux itself, or a security kernel such as EROS. Further- VMware’s runtime binary translation). Many have pro- more, the lock-in obstructs evolution of its own para- posed to modify the operating system (OS) for co- virtualization interface — virtual machines provide the design with the hypervisor, i.e., para-virtualization [35], vital ability to run obsoleted operating systems along- to improve performance and correctness; the possibili- side new operating systems, but para-virtualization of- ties seem unlimited, but the cost has been the emergence ten lacks this feature, requiring all concurrent instances of many para-virtualization projects with incompatible to be the hypervisor’s supported version. Even general and unreconcilable architectures, yet overlapping main- purpose operating systems have weaker restrictions for tenance efforts for modifications to the guest OSes, and their applications. This lock-in discards the modularity customer lock-in. By using specialized interfaces rather of virtualization; modularity is an intrinsic feature of tra- than the neutral machine interface, para-virtualization ditional virtualization, helping to add layered enhance- discards the modularity of traditional virtual machines. ments to operating systems, especially when enhanced Modularity via the neutral machine interface is a key in- by people outside the operating system’s development gredient of virtualization’s benefits. It enables a guest OS community (e.g., Linux server consolidation provided by to run on hypervisors with substantially different archi- VMware). tectures, such as Xen [2] vs. Linux-as-hypervisor [10,19] Virtualization and its modularity solve many sys- vs. hardware-accelerated para-virtualization — most tems problems; combined with the performance of para- para-virtualization interfaces hard code assumptions that virtualization it becomes even more compelling. We prevent adaptation to these different scenarios. Modu- show how to achieve both together. We offer a set of de- larity permits runtime hypervisor upgrades (via check- sign principles, which we call soft layering, that govern pointing or migration [6, 31] of the guest OS); in con- the modifications made to the operating system. Addi- trast, XenoLinux 2.6.9 is tied to Xen 2.0.2, and is un- tionally, our approach is highly automated, thus reduc- able to even run on the newer Xen 3 series – high-level ing the implementation and maintenance burden of para- source code modifications lock-in a particular hypervi- virtualization, which is especially useful for enabling ob- sor, obstructing the vital capability of VMs to run ob- soleted operating systems. We demonstrate soft layer- soleted OSes alongside modern OSes. Modularity per- ing on x86 and Itanium: we can load a single Linux mits OS enhancements written outside the OS’s kernel binary on a variety of hypervisors (and thus substitute community to be added in layers [6], remaining inde- virtual machine environments and their enhancements), pendent of fast-paced changes of kernel internals, even while achieving essentially the same performance as if those features are unwelcome by the kernel developers para-virtualization with less effort. themselves (as most are); in contrast, the Linux kernel team can veto the ideas of the Xen team in the current merge of XenoLinux into the mainstream Linux source. Modularity supports proliferation of hypervisors; prolif- Technical report 2006-15, July 2006 1 Fakultat¨ fur¨ Informatik, Universitat¨ Karlsruhe (TH) eration is unavoidable if the proliferation of operating for example, the upper layer may have the impression systems is an indicator, whether for licensing reasons that a resource is infinite, when the opposite is the case; (proprietary [34] vs. open source [2]), or products from or the upper layer may have an algorithm that maps well competing vendors (e.g., VMware, Microsoft, and Paral- to a feature of the machine but which is hidden by the lels), or products that focus on different feature sets (e.g., abstractions of the lower layer. Virtual machines have a desktop [32] vs. server [2, 34, 35]). Modularity addition- problem with transparency, particularly since the guest ally supports stackable enhancements via recursive vir- OSes try to use the mechanisms of the hardware, but tual machine construction [8, 13, 17]. must instead use abstractions that result from the hyper- We show how to add modularity to para-virtualization, visor multiplexing the hardware mechanisms. Many vir- achieving high performance and many of the features tualization projects have addressed this issue by increas- of traditional virtualization. Our solution relies on con- ing the transparency between the guest OS and the virtual straining para-virtualization’s modifications according to machine, by modifying the guest OS to interact with the several principles, which we call soft layering. As orig- internals of the layers below. These modifications intro- inally proposed for layered network protocols [7], soft duce a dependency between the guest OS and its lower layering embraces co-design of neighboring software layer, often ruining the modularity of virtualization: the layers, but with some conditions: guest OS may no longer execute on raw hardware, within other virtual machine environments, or permit VM recur- 1. it must be possible to degrade to a neutral interface, sion (a layer to be inserted between the guest OS and the by ignoring the co-design enhancements (thus per- virtual machine below it). mitting execution on raw hardware and hypervisors that lack support for the soft layering); 1.2 Soft layering 2. the interface must flexibly adapt to the algo- rithms that competitors may provide (thus sup- It is the power of the platform’s primitive interface [12] porting arbitrary hypervisor interfaces without pre- that enables the rich solutions of traditional virtualiza- arrangement). tion, and its neutrality that enables uncoordinated devel- opment [6], and we focus on this interface in our solu- Additionally, we use tools to apply the soft layer to a tion. The solution is simple: mostly honor strict layering, guest kernel (with substantial automation) to easily sup- but move the hypervisor’s virtualization routines into the port obsoleted kernels. protection domain of the guest kernel, thus enabling low- cost virtualization (see Figure 1). This is similar to the 1.1 Strict layering technique used by VMware, but we additionally prepare the guest kernel to help achieve the performance of para- A traditional virtual machine enforces strict layering, in virtualization. the manner of Djikstra’s “THE” [9], with the hypervi- The soft describes the scope and type of modifications sor (lower) layer hiding resource multiplexing from the that we apply to the source code of the guest OS: the guest OS (upper) layer; the upper layer then focuses on modifications remain close to the original structure of its problems rather than those of the lower layers. This the guest OS (i.e., the guest kernel’s use of the privi- information hiding provides modularity: it permits the leged instructions and its internal design philosophies), upper layer to use simpler abstractions (in regards to the it uses the neutral platform interface as the default in- information hidden in the lower layer) via its pre-existing terface (i.e., the OS will execute directly on raw hard- use of the platform interface; and to remain independent ware while the enhancements require activation), and the of the lower-level’s implementation, which permits sub- OS includes annotations that help the hypervisor opti- stitution of a new lower layer without disturbing the up- mize performance. Soft layering forbids changes to the per layer. The virtual machine must provide the strict guest OS that would interfere with correct execution on layering because its guest operating systems are oblivi- the neutral platform interface, it discourages hypervisor- ous to any resource multiplexing handled by the lower specific changes, and it discourages changes that sub- layers; an OS was written to perform the resource mul- stantially penalize the performance of the neutral plat- tiplexing, but not to participate in collaborative resource form interface. multiplexing with other OSes. The decision to activate a specialized hypervisor inter- Design by layering involves choosing appropriate ab- face happens at runtime when the hypervisor and guest stractions and cut points for the layer interfaces. Layer- kernel are joined together. The guest OS adds to its bi- ing’s abstractions provide benefits, but as explained by nary a description of its soft layer, and the hypervisor Parnas, the abstractions easily introduce inefficiencies inspects the descriptor to determine whether they agree due to the lack of transparency into the