Embedded Hypervisor Xvisor: A comparative analysis Anup Patel Mai Daftedar Mohmad Shalan M. Watheq El-Kharashi Individual Researcher, Ain Shams University, American University of Cairo, Ain Shams University, Bangalore, India Cairo, Egypt Cairo, Egypt Cairo, Egypt [email protected] [email protected] [email protected] [email protected] Abstract—Virtualization technology has shown immense Similarly, embedded virtualization has effectively popularity within embedded systems due to its direct proven its efficiency in a number of use cases: relationship with cost reduction, better resource utilization, 1. Motorola Evoke, the first virtualized phone [4]. and higher performance measures. Efficient hypervisors are 2. Industrial automation in which virtualization required to achieve such high performance measures in allows the addition of extra software application virtualized environments, while taking into consideration the without the need to add more processing units [5]. low memory footprints as well as the stringent timing 3. Cars can consolidate multiple services on the same constraints of embedded systems. Although there are a hardware, by running Infotainment OS, number of open-source hypervisors available such as Xen, AUTOSAR OS, and RTOS on separate virtual Linux KVM and OKL4 Microvisor, this is the first paper to machines [6]. present the open-source embedded hypervisor eXtensible Versatile hypervISOR (Xvisor) and compare it against two 4. Other use cases include devices used in retail and of the commonly used hypervisors KVM and Xen in-terms of gaming. comparison factors that affect the whole system The presented analytical research compares the new performance. Experimental results on ARM architecture embedded hypervisor Xvisor to the two existing embedded prove Xvisor’s lower CPU overhead; higher memory open source hypervisors: KVM and Xen under the bandwidth; lower lock synchronization latency and lower following considerations: virtual timer interrupt overhead and thus overall enhanced 1. Research is based on ARM architecture since virtualized embedded system performance. latest ARM processors have hardware virtualization extensions and ARM processors are Keywords-component; Embedded Systems; Virtualization; widely used in embedded systems. In addition, the Xen, Linux KVM; Xvisor majority of the board support packages in common between the three hypervisors are for ARM I. INTRODUCTION architecture. 2. KVM and Xen are chosen for comparison due to: The increasing demands of multi core embedded their support for ARM architecture [7, 8], their systems, has recently lead researchers to make use of open source availability allows us to gather virtualization technology in embedded systems. Embedded performance numbers without any restrictions [9], systems usually feature resource limitations, real-time board support mapping with Xvisor's board constraints, high performance requirements, and increasing support packages and finally their extension to the application stack requirements that entail the use of limited embedded systems. peripherals [1]. 3. Experimental results use micro-benchmarks, Virtualization technology provides the means to run which test operations such as memory accesses, multiple operating systems (OSs) as virtual machines cache accesses, integer operations, and task (VMs or guests) on single or multi core processing units. handling on running guests. Performance Each guest runs on one or more virtual processing units enhancements in such operations, improves the (vCPUs). Furthermore, all vCPUs of a guest are isolated overall performance of the system. Unlike macro- from one another but share the same peripherals. benchmarks, which are dedicated to test a specific Accordingly, use of virtualization provides the workload such as web server, kernel compilation, following advantages [2]: graphics rendering, etc. a) consolidating services, which were once running 4. Experimental results run only General Purpose on different devices as separate VMs on the same Operating System (GPOS) Linux as guest, since device; for the current date this is the only guest OS b) combining the OS’s real-time and general purpose supported by all three hypervisors. There is no features on one device [3]; common real-time operating system (RTOS) c) better fault tolerance; available for all three hypervisors. Hence testing of d) providing isolation to highly secured applications; commitments to timing constraints will be carried out through measurements of these two factors: 1) 3) Micro-kernelized low memory and CPU overhead from the Micro-kernelized hypervisors are usually lightweight hypervisor; 2) efficiency of the guest OS micro-kernels providing basic host hardware access and scheduling with minimal overheard from the CPU virtualization in hypervisor micro-kernel. They hypervisor. depend on a management virtual machine for full host An explanation of Xen, KVM and Xvisor’s hardware access, guest IO emulation, and rest of the implementation and a discussion of their limitations, is services. Some of these micro-kernelized hypervisors run presented as follows: Section II explains the virtualization each host device driver under a separate driver virtual classification. Section III introduces Xvisor and provides machine instead of having it under common management an overview of the open source hypervisors Xen and virtual machine. Examples of micro-kernelized hypervisor KVM. Explaining the implementation details for the main are Xen, Microsoft Hyper-V [14], OKL4 Microvisor [15], components that affect the comparison factors are and INTEGRITY Multivisor [16]. highlighted in the proceeding sections. The sections that follow will include a comparative analysis between these B. Virtualization mode hypervisors in-context of ARM architecture [10]. We The virtualization mode determines the type of guests address the guest IO emulation in Section IV, while host that can run on top of the hypervisor [17, 18]. interrupt processing, lock synchronization, and memory 1) Full virtualization management and memory footprint are presented in Unmodified guest OSs are allowed to run as guest by Sections V, VI, VII, and VIII respectively. A brief providing system virtual machines (i.e. emulating whole description of the application benchmarks used for our system resembling real hardware for guest). analytical analysis is provided in section IX followed by 2) Para-virtualization experimental results in section X and a conclusion section. Modified guest OSs are allowed to run as guest by providing hypercalls. This mode requires guest to use II. VIRTUALIZATION CLASSFICATION hypercalls for various IO operations (i.e. network We classify hypervisors into five categories based on send/receive, block read/write, console read/write, etc.) two aspects [11]: (1) Hypervisor design and (2) and sometimes in place of critical instructions. These Virtualization mode as shown in Fig. 1. Accordingly, hypercalls trigger a trap interrupt in the hypervisor, which classification plays a big role in performance measures based on the hypercall parameters, causes the hypervisor to obtained by the guests. provide the desired service to the guest. III. OPEN SOURCE HYPERVISORS FOR EMBEDDED SYSTEMS Following is a brief introduction of the two open source hypervisors Xen and KVM compared against the hypervisor under study, Xvisor. Since our research is concerned with performance comparisons, we address the hypervisor implementation details for system components known to incur overheads Figure 1. Virtualization classification. for running guests, and hence affect the performance of the whole virtualized embedded system. This includes how A. Hypervisor Design each hypervisor handles CPU virtualization, guest IO The hypervisor serves as an interface between the emulation and host hardware accesses. Additionally, some hardware and virtual machines. The manner by which insights concerning the main advantage characterizing these hypervisors are implemented determines how each hypervisor are mentioned. effectively they operate as virtualization managers. Based A. XEN on their implementation, all hypervisors fall in one of these hypervisor design categories. The Xen hypervisor shown in Fig. 2 is a micro- kernelized hypervisor that supports both fully virtualized 1) Complete Monolithic and para-virtualized guests. The Xen hypervisor kernel is a Complete monolithic hypervisor is a single software lightweight micro-kernel which provides: CPU layer responsible for host hardware access, CPU virtualization, MMU virtualization, virtual IRQ handling, virtualization, and guest IO emulation. Examples of and inter-guest communication [19]. Domain is the term complete monolithic hypervisor are: Xvisor and VMware used in Xen kernel to refer to a virtual machine or guest. ESXi Server [12] Domain0 (Dom0) is a special type of domain that runs a 2) Partially Monolithic modified version of Linux kernel. It must be running a Partially monolithic hypervisors are usually an priori any other virtual machine or guest and has full extension of general purpose monolithic OS (e.g. Linux, access to underlying host hardware. FreeBSD, NetBSD, Windows, etc.). They support host Dom0’s main purpose is to provide IO virtualization hardware access and CPU virtualization in OS kernel and services and guest management services using Linux support guest IO emulation from user-space software. kernel. DomainU (DomU) refers to the guest virtual Examples of partially monolithic hypervisor are: Linux machine with