Performance Evaluation of Virtual Appliances Zhaoqian Chen and David Kaeli Electrical and Computer Engineering Department Northeastern University, Boston, MA {stchen,kaeli}@ece.neu.edu Kevin Murphy Network Engines Canton, MA [email protected] Abstract VMware ESX [2], along with the development of hardware support for virtualization from Intel [3] and AMD [4], are ac- Virtualization technology has become standard in many celerating the move to deploying virtual machines in produc- computing environments. Independent Software Vendors (ISVs) tion appliances (i.e., virtual appliances). are now using virtual machines to deliver multiple software ap- Virtualization technology has been used effectively for pliances. However, the performance of these appliances when server consolidation and optimization of hardware resources. run on virtualized platforms is not well understood. In this pa- When enterprises deploy virtualization, they also enjoy the per, we present a performance evaluation framework for virtu- benefits of a number of associated features that include: VM alized appliances. We are using the XenServer virtual machine check-pointing, physical to virtual (P2V) migration, data man- system and evaluate its performance on four different classes of agement, and others. They pay less attention to how their appli- workload: 1) voice-over-IP, 2) TCPIP networking, 3) IO disk cations are performing in this virtualized environment. When intensive, and 4) CPU intensive. Our primary interest is under- a virtualized appliance runs on the same platform with other standing the scalability of the VoIP application in the presence virtualized appliances, we need to pay careful attention to the of the otherthree classes of workload. We also considera range associated performance impact. For some near real-time appli- of different virtual machine configurations. cations (e.g., telephony server, game server, etc.), performance Our experiments show that the XenSource VMM can isolate and latency are crucial and must be managed carefully. hardware resource utilization (i.e., disk and network I/O) nicely There are two main approaches to virtualization. Full virtu- across multiple virtual machines. But when running multiple alization allows an unmodified operating system to run on top virtual appliances concurrently, sharing physical CPUs across of a hypervisor without any awareness of the underlying hyper- virtual machines introduces significant overhead. We present visor layer. The hypervisor is a customized operating system the results of this study and consider future work to address layer that provides for virtual machine management. When us- some of these performance issues. ing full virtualization, the execution overhead is typically larger than on systems that use paravirtualization [5]. Paravirtualiza- tion runs a modified operating system that is fully aware of the 1. Introduction hypervisor and cooperates in the virtualization process. The complexity and performance of each virtualization approach Today, virtualization technology plays a critical role in can vary greatly, though full virtualization seems to be gain- many computing enterprises. Virtualization can provide lower ing in popularity in server-class applications. cost of ownership, improved cost/performance, and higher While some commercial studies have compared their sys- performance/watt. Virtualization was first popularized on tems, little has been done by the performance evaluation com- mainframe-class systems, though it recent rise in popularity is munity to analyze the performance and scalability of these en- tied to its ability to share hardware infrastructure across multi- vironments. The main goal of this work is to gain a better un- ple virtual machines run on commodity hardware. derstanding of the overhead associated with virtualization by Server consolidation can help enterprises achieve higher utilizing workloads that place load on different subsystems on system utilization, reduce complexity and total cost. The lead- the virtualized system. We measure a number of performance ing virtualization technology vendors such as Xen [1] and metrics to identify the overhead involved with different virtu- alization configurations on a virtualized version of a software VAs installed and with the VA performance tester/monitor in- appliance. tegrated. The rest of this paper is organized as follows. In the next section we present our profiling framework. Section 3 intro- duces the XenSource virtualization architecture that is used in this paper. Section 4 describes our synthetic workload used. In Section 5, we present performance results. Finally, Section 6 concludes the paper and presents directions for future work. 2 Evaluation Framework Our Virtual Appliance (VA) performance evaluation frame- work is shown in Figure 1. The framework consists of two sep- arate systems: 1) the VA tester/monitor and 2) the target plat- form (including the Virtual Appliances and VMM) under test. The VA tester/monitor software contains a VMM level pro- filer, client-based VA workload drivers to drive each VA, and a management user interface (UI). The VMM profiler probes the VMM to provide VM-level information, such as VM sta- tus, virtual/physical CPU utilization, VMM scheduling statis- Figure 2. Integrated VA performance monitor tics, virtual I/O speed, etc. and evaluation. A number of different VA workloads are configured on the target system to act as typical virtual appliances. The VA tester/monitor drives the different VAs with load. We then obtain VMM statistics and compile these back on the VA 3 Implementation tester/monitor system. Using these workloads and the VMM profiler together, we are able to analyze each VA’s performance. We have implemented this framework on a XenServer vir- The management UI is designed to allow the tester/monitor tual machine. The main appliance we are targeting is Aster- system administrator to monitor VA performance performance iskNOW [6], which is a well-known telephony server appli- through a graphical presentation. ance running the SIP protocol. In this section, we describe the platform and appliance being tested. 3.1 The Xen Architecture Xen is a free and open-source virtual machine monitor which allows multiple guest operating system instances to run concurrently on a single physical machine. It supports both para-virtualization and full virtualization. The Xen virtual- ization platform has multiple layers. Figure 3 provides an overview of the Xen architecture. The lowest and most privileged layer is the Xen VMM, which presents a virtual hardware abstraction slightly differ- ent from the underlying hardware. Each guest operating sys- tem (OS) runs on a separate virtual machine (or guest domain in Xen terminology). The domains are scheduled by the Xen Figure 1. VA Performance Evaluation Platforms VMM to make effective use of the available physical CPUs. Each application, running in the guest OS, accesses hardware Our VA test/monitor is presently configured on a sepa- devices via a special privileged virtual machine called Domain- rate system in order to eliminate any overhead. The VA 0 (or IDD, isolated driver domain). This privileged VM owns tester/monitor could be integrated into the platform to reside specific hardware devices and talks to the I/O device drivers. on a privileged VM, as shown in Figure 2. This privileged VM All other guest domains use a simple device driver that com- has full access to the VMM so that our VA tester/monitor is municates with Domain-0’s driver to access real hardware de- still able to profile VMM stats and all other VMs. Thus, fu- vices. Domain-0 can directly access the hardware devices it ture platform vendors will be able to produce a single box with owns. However, a guest domain accesses the hardware de- vices indirectly through a virtual device connected to Domain- 0. Domain-0 maps through bridges and routes each physical interface through page-sized buffers which are transferred over the I/O channel. This helps to avoid copying during execution. Figure 3. Xen Architecture Figure 4. Basic stateful SIP scenario without au- thentication. 3.2 An Overview of the SIP Protocol party XML scripts that define new call flows. SIPp has many The Session Initiation Protocol (SIP) [7] is an application run-time options, such as multiple transport (UDP/TCP/TLS) layer control protocol for creating, maintaining, and tearing support and MD5-based hash digest authentication. down sessions for various types of media, including voice, video, and text. SIP is growing in importance as it is being used for many media-oriented applications such as Voice-over- 4 Experimental setup IP (VoIP), voicemail, instant messaging, telepresence, IPTV, network gaming, and more. It is also the core protocol for The hardware used in our experiments is an Intel Dual- the IP Multimedia Subsystem, the basis for the 3rd-Generation Core CPU Xeon 3060 that supports Intel VT [3], running at Partnership Program for both fixed and wireless telephone net- 2.4GHz with 4MB L2 Cache and 4GB main memory. We use works. SIP relies on an infrastructure of servers, which are re- XenServer 4.0.1 which is an enterprise version of Xen from sponsible for maintaining the location of users and forwarding Citrix, Inc [9]. XenServer consists of the Xen VMM, man- SIP messages across the application-layer SIP routing infras- agement software, and a number of service features (P2V con- tructure toward their eventual destinations. The performance version, migration, built-in guest templates