THE VIRTUALIZATION SPECTRUM FROM HYPERTHREADS TO GRIDS† Neil J. Gunther Performance Dynamics Company, Castro Valley, California, USA www.perfdynamics.com Modern virtual machines (VMs) can be opaque to conventional performance management tools because VM technology has surpassed standard measurement paradigms. We attempt to ameliorate that problem by (1) observing that disparate types of VMs lie on a discrete spectrum bounded by hyperthreading at one extreme and GRID-like services at the other, and (2) recognizing that poll-based scheduling is the common architectural element in many VM implementations. The associ- ated polling frequency (from GHz to µHz) positions each VM in its respective region of the VM-spectrum. Several case studies are analyzed to illustrate how this insight can make VMs more visible to performance management techniques. 1 INTRODUCTION The inclusion of GRIDs and P2P under the umbrella of virtualization is an unusual step, but our intent is to use the VM-spectrum both as a classification scheme to Virtualization remains a hot topic because of the opacity organize the discussion of an otherwise bewildering array of virtual systems from the standpoint of conventional of VM architectures, and as a quantitative framework for performance management. These difficulties are further explaining previously reported performance anomalies in exacerbated by the appearance of so many disparate im- a variety of controlled measurements on VM systems. plementations employing the adjective virtual [See e.g., Sin04]. Moreover, modern computer architectures that Most previous discussions have tended to organize VM create virtual resources and services out of physical re- capacity planning issues according to whether they are sources are quite distinct from the more familiar virtual- implemented in hardware or software [See e.g., Joh03, ization paradigms e.g., virtual memory or virtual storage. DBK03, Fri03, Bra05, Fer05]. Macro-VMs have not fea- tured in such discussions, as far as I am aware. Con- This paper proposes a more unified picture of mod- sequently, the inability to achieve full virtual capacity ern virtualization by recognizing that many of these ap- in hyperthreaded hardware and anomalous performance parently disparate forms of virtualized resources or vir- outcomes in software hypervisors, have been presented tual machines (VMs) can be considered to lie on a dis- as distinct performance effects. crete spectrum—the virtual machine spectrum or VM- spectrum—comprised of three principal regions: The VM-spectrum paradigm, however, views these ef- 1. Micro-VMs: represented by the hyperthreaded mul- fects as arising from an architectural feature that is com- ticore processors discussed in Section 3. mon to both hardware and software implementations; a form of scheduling which we define in Section 2.2 as 2. Meso-VMs: represented by virtual machine moni- proportional polling. In particular, it provides a simple tors and hypervisors discussed in Section 4. explanation for the observed Missing MIPS problem in 3. Macro-VMs: represented by the GRID services and Sect. 3.2.2. The VM-spectrum also leads to the no- peer-to-peer (P2P) architectures in Section 5. tion that performance management of modern VMs is a † Copyright c 2006 Performance Dynamics Company and function of the time and distance scales on which their CMG, Inc. Accepted for publication in the Proceedings of the respective polling mechanisms operate. Computer Measurement Group (CMG) 32nd International Confer- ence, Reno NV, December 3–8, 2006. Version of September 11, The paper is organized as follows. Section 2 defines the 2006 The Virtualization Spectrum rationale for the VM-spectrum and its three principal re- gions. Each of these VM-regions is examined in detail, Micro P2P starting in Section 3 with the micro-VM scale. The best waves known VM implementations in this region are virtual pro- 1. Micro-VM: represented by cessors or hyperthreaded CPUs. Virtual processors also GRIDs Macro Scale: constitute the first example of a polling-based scheduler Long wavelengths, operating in the GHz to kHz frequency range. The as- hyperthreaded multicore sociated performance case studies and recommendations IR PVM Low frequencies are presented in Section 3.2. Section 4 moves up the VM-spectrum (Fig. 1) to slower polling meso-VMs repre- sented by virtual machine monitors or hypervisors. These processors VMs constitute the second example of a polling-based Sysplex scheduler operating in the kHz (kilohertz) to mHz (mil- lihertz) frequency range. The associated performance analysis case studies and recommendations are presented VMWare in Section 4.2. Section 5 discusses the slowest polling macro-VM services represented by GRIDs and P2P net- works. At this scale, different types of state information Xen are collected via polling mechanisms which can have op- 2. Meso-VM: represented by erating periods in the range of days! Obviously, these Meso Scale: frequency ranges can have a significant impact on the Visible FSS Medium wavelengths, performance and scalability of virtual services. The as- virtual machine monitors and sociated performance analysis case studies are presented Visible frequencies in Section 5.2. Section 6 presents conclusions and rec- TSS ommendations. operating systems LPAR 2 VIRTUALIZATION SPECTRUM Virtualization is about creating illusions. In particular, z/VM modern computer systems are now sufficiently powerful to present users with the illusion that one physical ma- UV Mach 3. Macro-VM: represented by chine is really multiple virtual machines, each one run- Micro Scale: ning a separate instances of a different operating system Short wavelengths, (OS). This is one reason for the resurgence of interest High frequencies GRID services and P2P in virtualization technologies. In the interests of space, Micro we forego any review of the history or relative merits of code various VM projects and products, which are adequately discussed elsewhere [See e.g., Sin04, Fer05, and refer- architectures ences therein]. X-rays HTT As many authors have already noted, the idea of creat- ing virtual resources e.g., software emulators and virtual memory, is not new. For the purposes of this paper, how- Figure 1: The continuous EM-spectrum compared with 1 ever, we draw a distinction between modern VMs and the discrete VM-spectrum. In both kinds of spectra, older concepts of virtualized resources. Virtual mem- the relative location of each region is determined by ory, for example, is fundamentally different from VMs their respective frequency scales (Table 1). For the VM- in that it is intrinsically unstable and subject to thrash- spectrum, the frequency is set by the VM polling rate. ing [Gun95]. The isolation of modern VMs often helps Like the invisible regions of the EM-spectrum, the mi- to inhibit this kind of instability. (See Sect. 4) cro and macro regions of the VM-spectrum are also less visible to standard performance management tools. The distinctive architectural feature which modern VMs have in common is some form of polling mechanism by the underlying scheduling subsystem. To make this Table 1: VM-spectrum Scales statement more quantitative, we refer to a case where Spectral Distance Polling the polling periods are well documented [Gun99]: meso- Region Scale (m) Period Frequency VM scheduling (see Sect. 4.1). The polling period Tp, for Macro 102 to 106 min to day mHz to µHz the scheduler to associate physical resource consumption Meso 100 to 102 ms to min kHz to mHz with a each active OS instance (software VM), is once −6 −3 Micro 10 to 10 ns to µs GHz to MHz every 4000 ms or Tp = 4 s. The frequency is therefore f = 1/Tp = 0.25 cycles per second or 250 mHz. to accomplish resource sharing. Polling algorithms are We assume (because it is not documented) that the intrinsically stable and fair; even round-robin polling micro-VM polling period lies in the range of ns (the exhibits intrinsic fairness. The potential performance processor GHz clock frequency) to µs (MHz frequency). penalty arises from the sequential nature of polling be- Macro-VMs can take minutes or days to detect active ing worse than asynchronous communications. Based peer horizons. These frequencies are key VM perfor- on this distinction, we can classify the variety of VM mance determinants. For those who prefer to think in manifestations on a discrete spectrum, analogous to the terms of size, a distance scale d (in meters) can be loosely continuous electromagnetic or EM-spectrum (Fig. 1). related to the period Tp by d = vTp where v is the phase velocity of the communication signal. Typically, v = c; the speed of light. All these scales are summarized in 2.1 Principal VM Regions Table 1. Just as the EM-spectrum can be grouped into the ultra- violet (UV), visible and infra-red (IR) regions, the VM- 3 MICRO-SCALE: HYPERTHREADS spectrum can be similarly grouped into the micro-VM, meso-VM, and macro-VM spectral regions. The relative We begin a detailed analysis of VMs starting with the position of each VM is defined by their respective polling highest frequency (smallest size) scale on the bottom of rate or frequency scale in Table 1. the VM-spectrum in Fig. 1; virtualization of physical pro- cessing resources. Intel, for example, refers to this form The so-called visible region on the EM-spectrum is an of processor virtualization as hyper-threading technology anthropocentric term. Certain snakes can see in the IR (HTT)) or multithreading (MT) on its XeonTM and Pen- (detect heat) and bees can see in UV light. Similarly, tium 4TM product lines. The rationale is to maximize only meso-VMs are “visible” to us via conventional per- throughput performance by utilizing idle cycles. Part of formance management tools, in the sense of providing an the current confusion over hyperthreading performance immediate view of performance and thereby some level stems from two possible views of what HTT offers: of potential control.