HVX: Virtualizing the Cloud Alex Fishman, Mike Rapoport, Evgeny Budilovsky, Izik Eidus Ravello Systems Abstract execution of unmodified operating systems. Guest de- privileging is performed using trap-and-simulate tech- Nowadays there is significant diversity in Infrastructure nique ([8]). Full virtualization hypervisors for x86 ar- as a Service (IaaS) clouds. The differences span from chitecture require hardware assistance available in recent virtualization technology and hypervisors, through stor- AMD and Intel processors ([2, 10]). By contrast, in the age and network configuration, to the cloud management para-virtualized environment, the guest operating system APIs. These differences make migration of a VM (or is aware that it is executed on a VM and it should contain a set of VMs) from a private cloud into a public cloud, modifications necessary to interface with the underlying or between different public clouds, complicated or even hypervisor instead of performing privileged operations impractical for many use-cases. ([1]). HVX is a virtualization platform that enables complete Virtual hardware devices. Different hypervisors ex- abstraction of underlying cloud infrastructure from the pose different sets of virtual hardware devices such as application virtual machines. HVX allows deployment CPU, chipset, network and storage controllers. Migrat- of existing VMs into the cloud without any modifica- ing a VM from one hypervisor to another resembles tions, mobility between the clouds and easy duplication transfer of a hard disk between different computers. of the entire deployment. Image formats. Each hypervisor supports its own HVX can be deployed on almost any existing IaaS proprietary image format, for example, VMware’s cloud. Each instance of the HVX deployment packs in VMDK, or KVM qcow and qcow2. a nested hypervisor, virtual hardware, network and stor- The situation in the IaaS world is even more com- age configuration. plex. Different cloud providers utilize different hyper- Combined with image store and management APIs, visors. For instance, the leading cloud operator Amazon the HVX can be used for the creation of a virtual cloud EC2 is based on Xen hypervisor, the HP cloud is built on that utilizes existing cloud provider infrastructure as the top of KVM, while the leading private cloud platform is hardware rather than using physical servers, switches and VMware ESX ([4, 9, 11, 20, 23]). In addition, IaaS cloud storage. providers offer incompatible APIs for VM provisioning, configuration and monitoring. Network and storage con- 1 Introduction figuration is different as well and varies from operator to operator. There are many virtualization solutions available today: Such levels of complexity pose significant challenges VMware ESX, KVM, Xen, just to name a few. These for the development and deployment of enterprise appli- solutions lack interoperability with each other, meaning cations in a public cloud. that a VM running on one hypervisor cannot be easily Migration of an existing application to the public migrated to another hypervisor. cloud requires provisioning and reconfiguration of VMs The existing virtualization solutions differ in several for virtualization technology used in cloud, adaptation aspects: for different storage infrastructure, and customization of Virtualization technique. There are two major ap- the networking stack. Development of new applications proaches to virtualization: full virtualization and para- requires analysis of existing cloud architecture, selection virtualization. The full virtualization hypervisors present of the appropriate public cloud, and creation of manage- their guest a complete virtual system and therefore allow ment software tailored for that particular cloud. There are efforts to resolve those issues. For instance, Cloud VM Xen-Blanket ([22]) uses nested virtualization to run ap- Unmodified plication VMs allowing thus to cope with virtualization application VM technology differences between IaaS providers - but it Guest VM hardware can only run PV guests. Another nested virtualization solution, the Turtles Project ([5]) provides an ability to Network Storage run unmodified guests, however it requires modifications HVX Hypervisor of the cloud provider infrastructure. Linux Most current commercial and open-source attempts to Cloud VM hardware ease the pains of cloud migration, interoperability and duplication are trying to meet these challenges by ap- Figure 1: HVX Architecture plying management only solutions. Typically, they re- build the VMs from scratch for each copy of the appli- cation, thus creating different VMs for each copy and agement and the networking stack significantly simpli- each cloud provider. These VMs are usually created ei- fies overall system design and implementation. ther using manually written scripts, or using a configura- HVX comprises three major components: nested hy- tion management tool ([6, 15]). The main disadvantages pervisor, virtual networking layer and cloud storage ab- of these solutions are that they require in-depth knowl- straction layer. HVX hypervisor provides its own set of edge regarding the application (meaning one cannot just virtual hardware, independent of the hardware supplied use unmodified VM images), and eventually the result- by the cloud operator. This enables complete isolation of ing VMs differ from the original ones, rendering these cloud specific virtualization technology and I/O configu- solutions problematic for many use-cases. ration details from the application VMs. The support for We propose a novel solution to these problems by in- the wide range of virtual storage and network devices al- troducing HVX - a thin virtualization layer running on lows reuse of existing VM images without the necessity top of the already virtualized hardware, providing unified of installing additional drivers and tools. The network- hardware, network and storage configuration to a guest ing layer provides an ability to define complex network VM and capable of running on top of most existing vir- topologies by creating an overlay virtual network on top tual machine monitors (VMM), including both full and of the cloud provider’s physical network. para-virtualized environments. The HVX layer supports Figure 1 illustrates HVX running inside a cloud virtual most of the existing image formats and contains imple- machine and executing an unmodified application VM. mentation of ESX, KVM and XEN VMMs virtual hard- ware thus allowing to run any existing VM without any modification on top of any cloud. For example, an un- 3 The Hypervisor modified guest VM with para-virtualized ESX devices can be run in EC2 cloud on top of a para-virtualized Xen 3.1 CPU Virtualization host. Moreover, the VM images can be exported back to The heart of HVX technology is the hypervisor. The hy- the original hypervisor format facilitating migration be- pervisor is responsible for execution of the application tween public and private clouds. HVX also contains a VM in a nested virtual environment. The HVX hyper- network abstraction layer that provides an overlay net- visor cannot rely on hardware virtualization extensions work to guest VMs making it possible, for example, to such as Intel VT ([19]) or AMD SVM ([3]) because it run several VMs in different public clouds that commu- already runs inside a virtual machine. On the other hand, nicate on the same network subnet as if they were physi- the nested virtualization support in hardware is not yet cally connected to the same L2 switch. mainstream and there is no guarantee that it will be avail- HVX management system provides convenient and able in public clouds in the near future. Hence, the HVX easy to use interface for creating, configuring and man- hypervisor implements full virtualization using binary aging the entire multi-VM deployment including appli- translation ([18]). cation virtual machines, storage and the logical network. The HVX hypervisor binary translation and simula- tion engine emulates the x86 processor ISA. It compiles 2 The HVX Architecture the original guest ring-0 machine code into another ma- chine code sequence that can be executed without alter- HVX is a thin virtualization layer running on Linux. ing privileged state of the physical CPU. HVX hypervi- Linux was chosen because of its support for para- sor fetches a basic block of the guest machine code and virtualization and the availability of device drivers for all replaces privileged instructions with a simulation code VMMs. Utilization of Linux scheduler, memory man- that operates on the emulated software CPU state rather than on the physical CPU. The original basic block exit Overlay network point is replaced with jump to HVX to return control to the hypervisor after execution of the translated basic block. The HVX repeats the algorithm again and again until the application VM is terminated. Guest 1 Guest 2 Guest 3 Guest 4 Guest 5 One of the most difficult challenges for nested virtu- 10.0.0.1 10.0.0.2 10.0.0.3 10.0.0.4 10.0.0.5 alization is achieving a high performance level. HVX utilizes many advanced techniques to accomplish this, HVX HVX HVX including basic block chaining, caching and reusing of 50.16.178.19 23.22.39.65 173.203.197.94 translated code, fast shadow MMU and APIC implemen- tation, direct execution of user space code (Ring 3), im- plementation of para-virtualized devices for network and Cloud provider Cloud provider network network IO, fast context switching between guest VM and the host kernel, and the use of Linux for guest VM schedul-