By Bill Bitner & Brian Wade, Ph.D. he z/VM product has its roots in an IBM product named VM/370, released in August 1972. VM/370 grew from earlier unsold versions of virtual machine technology dating back to 1967. This article Texplores some of the current product’s constructs and concepts. Let’s start with the basics. VIRTUal Machines The IBM System z processors operate according to a data processing machine architecture defined in z/Architecture Principles of Operation. This book explains the machine’s instruction set, its I/O pro- cessing rules, and how it runs programs. Engineers implement z/Architecture in IBM System z machines such as those in the IBM z9 family. The role of many operating system kernels is to create well-defined program execution con- texts in which applications can run concurrently without interfering with one another. For z/VM, these execution contexts are called virtual machines. Where an ordinary operating system might create execution contexts for application programs, z/VM creates execution contexts for System z operating systems. In other words, z/VM’s task is to create virtual machines adhering to z/Architecture. z/VM can create nearly any virtual configuration that could legitimately exist in the real hardware. Further, z/VM lets many such virtual machines exist and operate simultaneously on one instance of real hardware; in fact, in the most interesting cases, z/VM overcommits the real hardware. A virtual machine also is often known as a VM user ID, a VM logon, a VM guest, or a virtual server. Just as a real System z machine consists of real processors, real memory, and real I/O devices, a virtual machine consists of virtual processors, virtual memory, and virtual I/O devices. For example, z/VM might equip a virtual machine with two virtual processors, 1GB of virtual memory, a virtual Ethernet card, and a virtual 3270 terminal device. Because of the histories of z/VM and its guests, some of the virtual devices found in a typical virtual machine’s configuration correspond to real devices IBM no longer manufactures. For example, each virtual machine will typically have a virtual card reader, > z/Journal • February/March 2 0 0 8 • 3 9 virtual card punch, and virtual printer. Virtualization of Processors called a SIE exit or a SIE break. CP Many guests still effectively use these Because z/Architecture defines that responds to the SIE break by handling antiquated virtual devices even today. a System z data processing system can the condition and eventually redispatch- Various System z operating systems house one to 64 processors, each virtual ing the virtual processor by issuing can run in virtual machines. One, machine z/VM creates can have one to another SIE instruction. Conversational Monitor System (CMS), 64 virtual processors. Usually, these are A customer obtains best utility from is sold as part of z/VM. CMS is an inter- all general-purpose instruction proces- a System z machine when the machine active, single-user operating system sors. However, z/VM does let the cus- spends its time in SIE running guests, meant to run in a virtual machine. tomer define some of a guest’s virtual instead of outside of SIE running CP. Though for many years CMS and its processors to be specialty processors z/VM performance engineers have applications were a mainstay of IBM’s such as System z Application Assist means to keep track of CPU time spent offerings in interactive general-purpose Processors (zAAPs), System z Integrated inside and outside of SIE and to keep computing, CMS recently has found its Information Processors (zIIPs), or track of the reasons why virtual proces- niche in supporting applications meant System z Integrated Facility for Linux sors leave SIE. An important aspect of to help manage the z/VM system. The (IFL) processors. z/VM dispatches z/VM performance management is z/VM TCP/IP stack, for example, runs guests’ virtual processors on the proces- building guest operating systems and in a CMS virtual machine. CMS isn’t the sors of the System z machine it’s con- their applications to be aware they’re only System z operating system that can trolling. running virtually and to accomplish run virtually. Other System z operating z/VM provides control over processor their work using techniques that tend to systems, such as z/OS, z/VSE, z/TPF, resources by letting a system administra- promote remaining in SIE. IBM has and Linux for System z, can all run in tor assign a share value to each virtual spent many years tuning its long-lived virtual machines on z/VM. machine. This share value typically sets System z operating systems, such as The heart of z/VM is a multi-pro- the minimum amount of processor CMS, z/OS, and z/VSE, to use SIE- gramming, multi-processing operating resource a virtual machine can expect CP friendly techniques. Linux for System z, system kernel known as the Control to allocate to it. z/VM also lets the system being comparatively young, is still Program (CP). (Be careful when using administrator define a maximum share evolving. the term “CP,” for in some communities, value to prevent a guest from excessively “CP” means “central processor.”) CP is consuming processor resource. The Virtualization of Memory the component of z/VM that creates z/VM system administrator or system Memory can be defined in the CP and dispatches virtual machines on the operator can adjust share settings while directory or can be changed with the real System z hardware. virtual machines are running. CP DEFINE command. The virtual CP supports hundreds of com- machine memory is usually defined as a mands. Some control the configura- The Heart of Virtualization contiguous range. However, one can tion of the virtual machine. For System z virtualization depends on a define gaps in the memory. This can be example, the CP DEFINE command hardware instruction called Start helpful in some situations. lets a customer create a virtual device Interpretive Execution (SIE, pronounced Of course, all guest memory is vir- and add it to a virtual machine’s con- “sigh”). This instruction gives CP a tual. To virtualize memory, z/VM har- figuration. Though CP lets customers means to tell System z hardware to run nesses the System z processor’s Dynamic change virtual machine configurations a virtual processor. CP and the System z Address Translation (DAT) facility. nearly at will, customers usually define hardware cooperate in their use of a With DAT, the System z processor trans- their virtual machines ahead of time memory-resident control structure— lates each instruction address or oper- in a user enrollment and definition file called the SIE State Descriptor or the and address from virtual to real, using called the CP directory. The CP direc- SIE Block—to track the state of the vir- translation tables CP maintains. CP tory describes virtual machines and tual processor. The SIE block contains overcommits physical memory by keep- sets attributes for each one, such as such things as the virtual processor’s ing resident only those guest pages that number of virtual processors, amount last register values, the pointers to the appear to have been needed in the of virtual memory, and complement of owning guest’s DAT tables, and other recent past, constantly adjusting the disk devices. state information that lets a virtual pro- DAT tables to preserve the memory Because of the nesting capabilities of cessor continue to run instructions. The illusion for the guests. When physical IBM’s System z virtualization technolo- address of the SIE block is an operand memory is scarce, CP moves stagnant gy, the System z computer on which of the SIE instruction. guest pages first to expanded storage (a z/VM perceives it’s running might itself Owing to the completeness of the high-speed page storage buffer) and be mythical. A commonly seen nesting virtual processor description stored in eventually to disk. CP brings these pages situation is the one where z/VM runs in the SIE block, the System z hardware back to memory if the guest ever needs a Logical Partition (LPAR) of a System z can handle many of the virtual proces- them again. computer. A System z hardware facility sor’s needs without requiring the help of CP manages the level of physical called the LPAR hypervisor creates and CP. However, some conditions, such as memory overcommitment by consider- manages logical partitions in much the occurrence of a page fault or the guest’s ing a guest’s apparent memory need as same way that z/VM creates and man- use of an I/O instruction, are too com- part of deciding whether to run (dis- ages virtual machines. System z plex for the hardware to handle on its patch) the guest. If CP determines that machines contain special hardware that own. When the System z hardware letting a virtual machine run could lets this commonly found two-level encounters such a condition, it ends the cause the system to enter a thrashing nesting arrangement run with almost virtual processor’s time in SIE and gives condition because of an insurmountable no performance penalty. control back to CP. This phenomenon is lack of physical memory, CP prohibits 4 0 • z/Journal • February/March 2 0 0 8 the virtual machine from being dis- to a virtual machine. This gives the vir- idisks, it can use memory to improve patched for a period of time. Instead of tual machine exclusive use of the entire their performance. Central to z/VM’s moving the virtual machine to the dis- real device.