IIP (System Z9 Integrated Information Processor) Computer CEC (Central Electronics Complex) Server

IBM System z

z/VM Basics

Arwed Tschoeke Systems Architect tschoeke@de.ibm.com

Introduction

We'll explain basic concepts of System z: – Terminology – Processors – Memory – I/O – Networking We'll see that z/VM virtualizes a System z machine: – Virtual processors – Virtual memory – … and so on Where appropriate, we'll compare or contrast: – PR/SM or LPAR – z/OS – Linux

2 z/VM: The Very Basics

z/VM: The Very Basics 1 IBM System z

System z Parts Nomenclature

x86, UNIX, etc. System z Memory Storage (though we are moving toward "memory") Disk, Storage DASD – Direct Access Storage Device Processor Processor, Engine, PU (processing unit) IOP (I/O processor) CPU (central processing unit) CP (central processor) SAP (system assist processor) Specialty engines –IFL (Integrated Facility for Linux) –zAAP (System z Application Assist Processor) –zIIP (System z9 Integrated Information Processor) Computer CEC (central electronics complex) Server

3 z/VM: The Very Basics

IBM System z Virtualization Genetics Over 40 years of continuous innovation in virtualization – Refined to support modern business requirements System z10 . – Exploit hardware technology for economical growth , .. ity System z9 z/VM V5 bil – LPAR, Integrated Facility for Linux, HiperSockets xi 64-Bit Fle – System z Application Assist Processors s, zSeries es VM/ESA Virtual Switch – System z Information Integration stn 9672 bu ESA Guest LANs Set Observer Ro Processors y, 9x21 ilit VM/XA Virtual Machine Resource Manager ab eli 3090 31-Bit Virtual Disks in Storage Performance Toolkit , R ity VM/HPO CMS Pipelines QDIO Enhanced Buffer State Mgmt bil 308x ala 64 MB Real Accounting Facility Minidisk Cache HiperSockets Sc 303x e : Absolute | Relative SHARE SIE on SIE Automated Shutdown alu 4381 VM/SP V Discontiguous Saved Segments Named Saved Systems I/O Priority Queuing ss SMP ine Instruction TRACE Start Interpretive Execution (SIE) Host Page-Management Assist us VM/370 B Programmable Operator (PROP) LPAR Hypervisor Integrated Facility for Linux HyperSwap S/370 Inter-User Communication Vehicle (IUCV) VM Assist Microcode Adapter Interruption Pass-Through CP-67 Conversational Monitor System (CMS) Dedicated I/O Processors Multiple Logical Channel Subsystems (LCSS) S/360 Diagnose Hypervisor Interface Program Event Recording (PER) Open Systems Adapter (OSA) Network Switching Control Program Hypervisor Translation Look-Aside Buffer (TLB) REXX Interpreter Large SMP Dynamic Virtual Machine Timeout Dynamic Address Translation (DAT) Zone Relocation Expanded Storage Multiple Image Facility (MIF) N_Port ID Virtualization (NPIV) 1960s 1972 1980 1981 1988 1995 2007... IBM System z – a comprehensive and sophisticated suite of virtualization function 4 z/VM: The Very Basics

z/VM: The Very Basics 2 IBM System z

Virtual Machines

Virtual Machine Virtual Machine … Virtual Machine

Hypervisor (z/VM Control Program) A virtual machine is an execution context that obeys the architecture The purpose of z/VM is to virtualize the real hardware: – Faithfully replicate the z/Architecture Principles of Operation – Permit any virtual configuration that could legitimately exist in real hardware – Let many virtual machines operate simultaneously – Allow over commitment of the real hardware (processors, for example) – Your limits will depend on the size of your physical zSeries computer Virtual machine aka VM user ID, VM logon, VM Guest, Virtual Server

6 z/VM: The Very Basics

z/VM: The Very Basics 3 IBM System z

Virtual Machines in Practice

Linux Linux z/OS CMS z/VSE z/TPF Others 31-bit 64-bit Hypervisor (z/VM Control Program) Control Program Component – manages virtual machines that adhere to the S/390 architecture and the z/Architecture

Extensions available through CP system services and features CMS is special single user system and part of z/VM Control Program interaction via console device

7 z/VM: The Very Basics

Phrases associated with Virtual Machines

In VM – Guest: a system that is operating in a virtual machine, also known as user or userid – Running under VM or Running on VM: running a system as a guest of VM – Running second level: running a system as a guest of VM which is itself a guest of another VM – A virtual machine may have multiple virtual processors – Sharing is very important In relationship to LPAR (partitioning) – Logical Partition: LPAR equivalent of a virtual machine – Logical Processor: LPAR equivalent of a virtual processor – Running native or Running in BASIC mode : running without LPAR • Note: Basic mode is not available on z890, z990 or z9 – Isolation is very important

8 z/VM: The Very Basics

z/VM: The Very Basics 4 IBM System z

Phrases Associated with Virtual Machines

Linux

Virtual Linux z/OS or z/VSE processor

z/VM Control Program

Virtual Virtual Virtual Virtual processor processor processor processor

z/VM Control Program

Logical Logical Logical Logical Logical Logical Logical processor processor processor processor processor processor processor Logical Partition Logical Partition PR/SM LPAR

9 z/VM: The Very Basics

A Virtual Machine

z/Architecture We permit any 512 MB of memory configuration that a 2 processors real zSeries machine Basic I/O devices: could have – A console In other words, we – A card reader completely Virtual – A card punch implement the Machine – A printer Some read-only z/Architecture disks Principles of Some read-write Operation disks There is no “standard Some networking virtual machine devices configuration”

10 z/VM: The Very Basics

z/VM: The Very Basics 5 IBM System z

VM User Directory

Definitions of: USER LINUX01 MYPASS 512M 1024M G – memory MACHINE ESA 2 IPL 190 PARM AUTOCR – architecture CONSOLE 01F 3270 A – processors SPOOL 00C 2540 READER * – spool devices SPOOL 00D 2540 PUNCH A – network device SPOOL 00E 1403 A – disk devices SPECIAL 500 QDIO 3 SYSTEM MYLAN LINK MAINT 190 190 RR – other attributes LINK MAINT 19D 19D RR LINK MAINT 19E 19E RR MDISK 191 3390 012 001 ONEBIT MW MDISK 200 3390 050 100 TWOBIT MR

11 z/VM: The Very Basics

CP Commands

CP DEFINE – Adds to the virtual configuration somehow – CP DEFINE STORAGE – CP DEFINE PROC – CP DEFINE {device} {device_specific_attributes} CP ATTACH – Gives an entire real device to a virtual machine CP DETACH – Removes a device from the virtual configuration CP LINK – Lets one machine's disk device also belong to another's configuration CP SET – Change various characteristics of virtual machine Changing the virtual configuration after logon is considered normal Usually the guest operating system detects and responds to the change

12 z/VM: The Very Basics

z/VM: The Very Basics 6 IBM System z

Getting Started

IML – Initial Machine Load or Initial Microcode Load – Power on and configure processor complex – VM equivalents are: • LOGON uses the MACHINE statement in the CP directory entry • The CP SET MACHINE command – Analogous to LPAR image activation IPL – Initial Program Load – Like booting a Linux system – zSeries hardware allows you to IPL a system – z/VM allows you to IPL a system in a virtual machine via the CP IPL command – Linux kernel is like VM nucleus – Analogous to the LPAR LOAD function

13 z/VM: The Very Basics

Processors

z/VM: The Very Basics 7 IBM System z

Processors

Configuration – Virtual 1- to 64-way • Defined in user directory, or • Defined by CP command – A real processor can be dedicated to a virtual machine Control and Limits – Scheduler selects virtual processors according to apparent CPU need – “Share” setting - prioritizes real CPU consumption • Absolute or relative • Target minimum and maximum values • Maximum values (limit shares) either hard or soft – “Share” for virtual machine is divided among its virtual processors

15 z/VM: The Very Basics

Start Interpretive Execution (SIE)

SIE = “Start Interpretive Execution”, an instruction z/VM (like the LPAR hypervisor) uses the SIE instruction to “run” virtual processors for a given virtual machine. SIE has access to: – A control block that describes the virtual processor state (registers, etc.) – The Dynamic Address Translation (DAT) tables for the virtual machine z/VM gets control back from SIE for various reasons: – Page faults – I/O channel program translation – Privileged instructions (including CP system service calls) – CPU timer expiration (dispatch slice) – Other, including CP asking to get control for special cases CP can also shoulder tap SIE from another processor to remove virtual processor from SIE (perhaps to reflect an interrupt)

16 z/VM: The Very Basics

z/VM: The Very Basics 8 IBM System z

Scheduling and Dispatching

VM – Scheduler determines priorities based on share setting and other factors – Dispatcher runs a virtual processor on a real processor – Virtual processor runs for (up to) a minor time slice – Virtual processor keeps competing for (up to) an elapsed time slice LPAR hypervisor – Uses weight settings for partitions, similar to share settings for virtual machines – Dispatches logical processors on real engines Linux – Scheduler handles prioritization and dispatching processes for a time slice or quantum

17 z/VM: The Very Basics

Anomalies of Time

VM virtualizes various timers or clocks – CPU timer – runs as processor time consumed – Time of day (TOD) clock – Clock comparator Anomaly – TOD always moves at wall clock speed – Virtual CPU timer “moves” slower as the sharing of the real processor increases – Problem when calculations assume CPU timer is moving at TOD clock speed LPAR – Same potential, but seldom shares processors to high enough degree to create drastic anomalies

18 z/VM: The Very Basics

z/VM: The Very Basics 9 IBM System z

Anomalies of Time Stop running virtual server A, and dispatch CPU virtual server B Timer Running Waiting Server A 20 5 Seconds Seconds

CPU Timer Running 30 Waiting 5 Server B Seconds Seconds

TOD 60 Seconds Wall Clock time

Virtual Server Total CPU CPU Timer Incorrect Correct Timer ‘busy’ Utilization Utilization A 25 20 80% 33% B 35 30 86% 50%

19 z/VM: The Very Basics

Allows z/VM guests to expand or contract the number of virtual processors it uses without affecting the overall CPU capacity it is allowed to consume – Guests can dynamically optimize their multiprogramming capacity based on workload demand – Starting and stopping virtual CPUs does not affect the total amount of CPU capacity the guest is authorized to use – Linux CPU hotplug (cpuplugd) daemon starts and stops virtual CPUs based on Linux Load Average value. • The cpuplugd daemon is available with SLES10 SP2 and IBM is working with it Linux distributor partners to provide this function in other Linux on System z distributions. Helps enhance the overall efficiency of a Linux-on-z/VM environment

CPU 0 CPU 1 CPU 2 CPU 3 Reduced Need for CPU 0 CPU 1 CPU 2 CPU 3 SHARE=25 SHARE=25 SHARE=25 SHARE=25 Multiprogramming SHARE= 50 SHARE= 50 Stopped Stopped

Guest SHARE = 100 Stop 2 CPUs Guest SHARE = 100

CPU 0 CPU 1 CPU 2 CPU 3 Increased Need for CPU 0 CPU 1 CPU 2 CPU 3 SHARE=50 SHARE=50 Stopped Stopped Multiprogramming SHARE= 25 SHARE= 25 SHARE= 25 SHARE= 25

Guest SHARE = 100 Start 2 CPUs Guest SHARE = 100

Note: Overall CPU capacity for a guest system can be dynamically adjusted using the SHARE setting 20 z/VM: The Very Basics

z/VM: The Very Basics 10 IBM System z

Memory

Virtual Memory

Configuration – Defined in CP directory entry or via CP command – Can define storage with gaps (useful for testing) – Can attach expanded storage to virtual machine Control and Limits – Scheduler selects virtual machines according to apparent need for storage and paging capacity – Virtual machines that do not fit criteria are placed in the eligible list – Can reserve an amount of real storage for a guest’s pages – Can lock certain specific guest pages into real storage

22 z/VM: The Very Basics

z/VM: The Very Basics 11 IBM System z

Shared Memory

1GB Key Points: – Sharing: • Read-only Shared address range (one copy) 800MB • Read-write • Security knobs – Uses: 768MB Virtual • Common Machine Virtual kernel Machine 512MB Virtual • Shared Machine programs

Control Program (hypervisor)

23 z/VM: The Very Basics

Layout of Real Memory

z/VM 5.1 z/VM 5.2 Expanded Storage Expanded Storage CP paging CP paging Minidisk caching Minidisk caching Real Storage Real Storage Virtual pages Above Virtual pages Above Minidisk caching or Minidisk caching or below CP free storage below 2GB Frame table 2GB 2GB 2GB System execution space table CP free storage Page tables Frame table Trace tables Page tables Prefix pages Trace tables Prefix pages 2000 CP Nucleus 2000 CP Nucleus 0 0

24 z/VM: The Very Basics

z/VM: The Very Basics 12 IBM System z

Memory Management

VM – Demand paging between central and expanded – Block paging with DASD (disk) – Steal from central based on LRU with reference bits – Steal from expanded based on LRU with timestamps – Paging activity is traditionally considered normal LPAR – Dedicated storage, no paging Linux – Paging on per-page basis to swap disks – No longer swaps entire processes – Traditionally considered bad

25 z/VM: The Very Basics

VM Memory Virtualization

Guest VM Paging 1 Swapping 2 4 4 3

3 3

2 1 2 1

Guest Virtual Guest Real Host Real

26 z/VM: The Very Basics

z/VM: The Very Basics 13 IBM System z

Collaborative Memory Management Assist(CMMA) Extends coordination of memory and paging between Linux and z/VM to the level of individual pages z/VM reclaims “unused”pages at higher priority

Bypass host page writes for unused and “volatile”pages (clean disk cache pages) Signal exception if guest references discarded volatile page Use Host Page-Management Assist to re-instantiate pages for next use z/VM support included in V5.3

27 z/VM: The Very Basics

Saved Segment and NSS Support

DCSS (Discontiguous Saved Segments) – Defines an address range (MB boundary) to the system – A single copy is shared among all guests – Guest "loads" the DCSS (maps DCSS into its address space) • Can be located outside guest's defined storage – DAT lets this work with minimal CP involvement – Contains: • Data (e.g. file system control blocks) • Code (e.g. CMS code libraries) NSS (Named Saved Systems) – An IPL-able saved segment – Great for CMS or for Linux • 1 shared copy on system for N guests, instead of N copies. • Faster boot Special Cases – Writable by guest, or by CP – Restricted (sensitive data) – Can have both exclusive and shared ranges

28 z/VM: The Very Basics

z/VM: The Very Basics 14 IBM System z

Linux Exploitation of z/VM DiscontiguousSaved Segments (DCSS) DCSS support is data-in-memory technology – Share a single, real memory location among multiple virtual machines – High-performance data access – Can reduce real memory utilization Linux exploitation support available today – Execute-in-place (xip2) file system – DCSS memory locations can reside outside the defined virtual machine configuration – Access to file system is at memory speeds; executables are invoked directly out of the file system (no data movement required) – Avoids duplication of virtual memory and data stored on disks – Enables throughput benefits for Linux guest images and helps enhance overall system performance and scalability

29 z/VM: The Very Basics

z/VM V5.4 exploits dynamic memory reconfiguration Users can nondisruptively add memory to a z/VM LPAR – Additional memory can come from: a) unused available memory, b) concurrent memory upgrade, or c) an LPAR that can release memory – Systems can now be configured to reduce the need to re-IPL z/VM – Memory cannot be nondisruptively removed from a z/VM LPAR z/VM virtualizes this hardware support for guest machines – Currently, only z/OS and z/VM support this capability in a virtual machine environment

Linux Linux Linux z/VSE z/VM z/OS Linux Linux

z/VM I/O and Network LPAR Dynamically add Resources Memory New with V5.4 resources to z/VM LPAR CPU Smart economics: Nondisruptively scale your z/VM environment by adding hardware assets that can be shared with every virtual server

VMV54_29030 z/VM: The Very Basics

z/VM: The Very Basics 15 IBM System z

I/O Resources

Device Management Concepts

Dedicated or attached – The guest has exclusive use of the entire real device. Virtualized – Present a slice of a real device to multiple virtual machines – Slice in time or slice in space – e.g. DASD, crypto devices Simulated – Provide a device to a virtual machine without the help of real hardware – Virtual CTCAs, virtual disks, guest LANs, spool devices Emulated – Provide a device of one type on top of a device of a different type – FBA emulated on FCP SCSI

32 z/VM: The Very Basics

z/VM: The Very Basics 16 IBM System z

Device Management Concepts

Terminology – RDEV is Real Device • Can refer to the device address or the control block – VDEV is Virtual Device • Can refer to the device address or the control block – UCB is Unit Control Block • Used in hardware definitions – RDEV=UCB=subchannel=device=adapter Control and Limits – Indirect control through “share” setting – Real devices can be “throttled” at device level – Channel priority can be set for virtual machine – MDC fair share limits (can be overridden)

33 z/VM: The Very Basics

Virtualization of Disks Excellent swap device if not storage-constrained Minidisk Cache (High-speed, Virtual Disk Virtual Disk 101 in-memory disk in Storage in Storage cache) (memory) (memory) R/W D R/O R/O R/W R/W B01 B01 100 E R/W 200 A 100 100 Linux C Linux R/W Linux R/W 1 2 3 B

z/VM

TDISK : on-the-fly disk 2B00 2B01 2B02 Minidisk : z/VM disk allocation pool TDISK 1 Minidisk 1 allocation technology Dedicated Minidisk 2 TDISK space Minidisk 3 Notes: R/W = Read/Write Enterprise Storage Server ™ (Shark) R/O = Read Only

34 z/VM: The Very Basics

z/VM: The Very Basics 17 IBM System z

z/VM Disk Technology – SCSI

Minidisk Cache (High-speed, FBA in-memory disk cache) R/W T1 R/O R/O

FBA FBA FBA Linux B Linux R/W Linux R/W 1 2 A 3 C

z/VM

TDISK : on-the-fly disk allocation pool SCSI Disks SCSI SCSI SCSI 00 attached to z/VM. TDISK 1 Minidisk A 100 Appear to guests Paging Minidisk B 200 and rest of VM as TDISK space Minidisk C emulated FBA. 399 Enterprise Storage Server ™ (Shark)

35 z/VM: The Very Basics

Data-in-Memory

Minidisk Cache – Write-through cache for non-dedicated disks – Cached in central or expanded storage – Psuedo-track cache – Great performance - exploits access registers – Lots of tuning knobs Virtual Disk in Storage – Like a RAM disk that is pageable – Volatile – Appears like an FBA disk – Can be shared with other virtual machines – Plenty of knobs here too

36 z/VM: The Very Basics

z/VM: The Very Basics 18 IBM System z

Networking

Virtual Networks

Connecting virtual machines to one another – Guest LAN • QDIO or HiperSockets – Virtual Switch Guest LAN • Layer 2 or Layer 3 Connecting virtual machines to another LPAR – HiperSockets – Shared OSA Connecting virtual machines to the physical network – Dedicated OSA device – Virtual Switch • Layer 2 or Layer 3

38 z/VM: The Very Basics

z/VM: The Very Basics 19 IBM System z

Virtual Switch Guest LAN

Virtual Machine Virtual Machine … Virtual Machine

z/VM Control Program Virtual Switch

Network

Linux Linux Linux Guest Guest Guest One Two “N”

Switch Network

39 z/VM: The Very Basics

VSWITCH

40 z/VM: The Very Basics

z/VM: The Very Basics 20 IBM System z

Time

Anomalies of Time

42 z/VM: The Very Basics

z/VM: The Very Basics 21 IBM System z

Anomalies of Time Stop running virtual server A, and dispatch CPU virtual server B Timer Running Waiting Server A 20 5 Seconds Seconds

CPU Timer Running 30 Waiting 5 Server B Seconds Seconds

TOD 60 Seconds Wall Clock time

Virtual Server Total CPU CPU Timer Incorrect Correct Timer ‘busy’ Utilization Utilization A 25 20 80% 33% B 35 30 86% 50%

43 z/VM: The Very Basics

Questions?

Linux SSL XML Web Servers SET

Browsers HTTP Open Source Java HTML TCP/IP GUIs Open Standards

44 z/VM: The Very Basics

z/VM: The Very Basics 22 IBM System z

English Hindi

Thai Gracias

Russian Spanish

Traditional Chinese Obrigado Brazilian Portuguese Arabic Danke German Grazie Simplified Chinese Italian Merci French

Japanese

Tamil Korean

45 z/VM: The Very Basics

Bonus Material

z/VM: The Very Basics 23 IBM System z

Virtual Machine Modes (Architectures)

An architecture is a formal set of rules for how a computer operates VM has kept pace with the evolution of IBM mainframe architecture ESA – ESA/390 or z/Architecture if running on System z processor – SIGP Set Architecture order must be issued for z/Architecture – ESA/390 when running on ESA/390 processor XC – ESA/XC is unique to z/VM virtual machines (DAT-off use of AR mode) XA – Processes the same as ESA mode (compatibility with older VM releases) 370 – No longer supported as a virtual machine mode – Processes according to ESA/370 architecture – CP and CMS provide 370 Accommodation features to help run 370 applications in ESA, XA, and XC modes (DAT off)

47 z/VM: The Very Basics

Other Processor Resources

Registers – General purpose, control, access, and floating point • CP saves and restores between invocations of SIE • Manipulation of control registers sometimes requires CP’s involvement (SIE exit) Timers – CPU timer – Clock comparator – Virtualized TOD clock • SET VTOD command to set virtual machine TOD clock to a specific value or to that of another virtual machine Storage Keys PSW, interrupts, prefixing, and other architected structures

48 z/VM: The Very Basics

z/VM: The Very Basics 24 IBM System z

Virtual Machine Address Translation

V=R V=F V=V (Virtual=Real) (Virtual=Fixed) (Virtual=Virtual) Fixed contiguous area of host Fixed contiguous area of host Does not map permanently to real storage real storage host real storage Absolute page zero High end of V=R area – never Storage allocated from DPA (low end of V=R area) – no absolute page zero address translation Not paged by CP Not paged by CP Guest real storage paged in and out of host real storage by CP Automatic recovery Automatic recovery No automatic recovery Preferred guest – CP provides Preferred guest – CP provides Not preferred performance benefits performance benefits Only 1 may be logged on Up to 6 may be logged on (or 5 Limited only by resources, plus 1 V=R) design point of roughly 100,000 Not supported in z/VM V5 Not supported in z/VM V5 Available in z/VM V5 Not supported on z890/z990/z9

49 z/VM: The Very Basics

Classic Scheduler / Dispatcher Picture

Dispatch List Eligible List Resources available VMDBK D1 VMDBK E1 (time slice and priority basis) . . . . Minor time slice expires . . other interrupt or intercept . . Elapsed time slice expires . . VMDBK Dn ...ready VMDBK En

User becomes User goes idle ...wait state runnable Dormant list

User logs off User logs on

50 z/VM: The Very Basics

z/VM: The Very Basics 25 IBM System z

Multiple Virtualization Layers

Multiple Levels of SIE – Both z/VM and LPAR use SIE – z/VM running on LPAR = 2 levels of SIE • No V=F support, and V=R loses I/O Assist • Rest of SIE features can be shared without performance loss – z/VM running on z/VM on LPAR = 3 levels of SIE • A layer of SIE now has to be virtualized • Fairly expensive 2nd level (and 3rd level …) Systems – Often used for testing purposes or disaster recovery – Most levels I ever saw was 9 Performance Data between Levels – LPAR and VM support Diagnose 204 to provide processor utilization to virtual servers supported – VM provides a Diagnose that a guest can use to pass data to the Control Program – VM provides Diagnoses for guest to gather some information – Anomalies in data when guest systems make poor assumptions (i.e. wall clock time = total processor time)

51 z/VM: The Very Basics

z/VM: The Very Basics 26