Introduction to Virtualization

System z

SHARE Anaheim, Session 14561

Romney White, IBM romneyw@us.ibm.com System z Architecture and Technology

° Introduction to Virtualization – Concept – Server Virtualization Approaches – Hypervisor Implementation Methods – Why Virtualization Matters ° Virtualization on System z – Logical Partitions – Virtual Machines

Virtual Resources ó Proxies for real resources: same interfaces/functions, different attributes ó May be part of a physical resource or multiple physical resources

Virtualization óCreates virtual resources and "maps" them to real resources óPrimarily accomplished with software or firmware

Resources óComponents with architecturally-defined interfaces/functions óMay be centralized or distributed - usually physical óExamples: memory, disk drives, networks, servers

ó Separates presentation of resources to users from actual resources ó Aggregates pools of resources for allocation to users as virtual resources

Hardware Partitioning Bare-metal Hypervisor Hosted Hypervisor Apps ... Apps Apps ... Apps Apps ... Apps OS OS OS OS OS OS

Adjustable partitions Hypervisor Hypervisor Partition Controller Host OS SMP Server SMP Server SMP Server

Server is subdivided into fractions Hypervisor provides fine-grained Hypervisor uses OS services to each of which can run an OS timesharing of all resources do timesharing of all resources

Physical partitioning Hypervisor software/firmware Hypervisor software runs on S/370™ SI-to-PP and PP-to-SI runs directly on server a host operating system Sun Domains, HP nPartitions System z LPAR and z/VM ® VMware GSX ® Logical partitioning POWER™ Hypervisor Microsoft Virtual Server IBM eServer™ pSeries ® LPAR VMware ESX Server HP Integrity VM HP vPartitions Xen Hypervisor KVM

Characteristics: • Bare-metal hypervisors offer high efficiency and availability • Hosted hypervisors are useful for clients where host OS integration is important • Hardware partitioning is less flexible than hypervisor-based solutions

4 System z Virtualization Technology © 2013 IBM Corporation Hypervisor Implementation Methods Trap and Emulate Translate, Trap, and Emulate Virt Mach Virt Mach • VM runs in user mode • VM runs in user mode L • All privileged instructions L • Some IA-32 instructions must A cause traps A be replaced with trap ops ST ST PrivOp TrapOp L Trap Hypervisor PrivOp L Trap Hypervisor PrivOp ... emulation code ... emulation code

Examples : CP-67, VM/370 Examples : VMware, Microsoft VS Benefits : Runs unmodified OS Benefits : Runs unmodified, translated OS Issues : Substantial overhead Issues : Substantial overhead

Hypervisor Calls (“Paravirtualization”) Direct Hardware Virtualization Virt Mach Virt Mach • VM runs in normal modes • VM runs in normal modes L • OS in VM calls hypervisor L • Hardware does most of the A to access real resources A virtualization (SIE architecture) ST ST • Hypervisor provides control Hcall PrivOp L Call Hypervisor L Exit Hypervisor service ...... service

Examples : POWER Hypervisor, Xen Examples : System z LPAR, z/VM, Xen Benefits : High efficiency Benefits : High efficiency, runs unmodified OS Issues : OS must be modified to issue Hcalls Issues : Requires underlying hardware support

Roles: Possible Benefits: Virtual Servers Consolidations High resource utilization

Dynamic provisioning / hosting Great usage flexibility Workload management Enhanced workload QoS Workload isolation Virtualization High availability / security Software release migration Low cost of availability Mixed production and test Low management costs Physical Mixed OS types/releases Server Enhanced interoperability Reconfigurable clusters Legacy compatibility Low-cost backup servers Investment protection

Virtualization can fill many roles and provide many benefits In the final analysis, its potential benefits take three forms: • Help reduce hardware costs – Help increase physical resource utilization – Small footprints • Can improve flexibility and responsiveness – Virtual resources can be adjusted dynamically to meet new or changing needs and to optimize service level achievement – Virtualization is a key enabler of on demand operating environments such as cloud • Can reduce management costs – Fewer physical servers to manage – Many common management tasks become much easier

HiperSockets & Virtual Networking and Switching ° Multi-dimensional virtualization Web technology Sphere WebSphere Traditional SysAdmin – System z provides Test Apache OLTP and FTP Test Tools Batch logical (LPAR) and software (z/VM) Linux Linux CMS Linux WLM WLM Linux z/OS partitioning z/OS z/OS – PR/SM enables highly scalable virtual server z/VM Control Program z/VM hosting for LPAR and z/VM virtual machine I/O & Network environments Memory Memory Memory Memory – IRD coordinates allocation of CPU IFL Processors Processors and I/O resources among z/OS and L P A R L P A R L P A R L P A R non-z/OS ® LPARs* Intelligent Resource Director (IRD)

Processor Resource / System Manager (PR/SM) * Excluding non-shared resources like Integrated Facility for Linux processors

Logical CPU SIE: Start Interpretive Program “instruction” Execution Instruction stream Instructions

Hardware Load Load, Store, Add, ... Store Problem state Instructions Instruction Add Execution Controls Hardware or Firmware High-Frequency Start Subchannel, SSCH Control Instructions that Test Subchannel, ... TSCH require virtualization Instruction SIE Interception Interpretation to hypervisor Low-Frequency LPAR Handling Control hypervisor Virtualization Instructions that SIE LPAR require hypervisor E.g., Modify CPU Assists virtualization Subchannel STATE Descriptor

Physical CPU Instruction Execution Unit

z/VM Logical z/OS Logical Partition z/VM Logical Partition Partition General General zAAP zIIP General General General zAAP zIIP IFL IFL Purpose Purpose Purpose Purpose Purpose Logical Logical Logical Logical Logical Logical Logical Logical Logical Logical Logical Processor Processor Processor Processo r Processor Processor Processor Processor Processo r Processo r Processo r

JAVA General General General IFL IFL zAAP zIIP z/OS & z/OS & & DB2 Purpose Purpose Purpose Virtual Virtual Virtual Virtual Virtual Virtual Virtual Processor Processor DB2 DB2 Processor Processor XML Processor Processor Processor

LPAR hypervisor dynamically dispatches: 1. General-purpose logical processors on general-purpose physical processors 2. zAAP logical processors on zAAP physical processors 3. zIIP logical processors on zIIP physical processors 4. IFL logical processors to IFL physical processors

Shared Shared Shared Shared Shared Shared Shared Shared Shared Shared General General General General zAAP zAAP zIIP zIIP IFL IFL Purpose Purpose Purpose Purpose Physical Physical Physical Physical Physical Physical Physical Physical Physical Physical Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor

Shared IFL Shared General-Purpose Shared zAAP Shared zIIP Physical Processor Physical Processor Pool Physical Processor Physical Processor Pool Pool Pool

A collection of up to LPAR 1 LPAR 2 LPAR N 231 - or A 264 -bytes collection virtual of up to address 64 spaces 2 -bytes z/VM Virtual Machine z/OS or Linux High- virtual Each Physical Memory Space Performance Logical address virtual spaces machine Partition Physical Memory may have Spaces z/OS and its own Linux virtual exploit address multiple spaces such mapped virtual to this address common z/VM Hypervisor spaces logical partition Logical Partition Hypervisor address space

= the real partition memory pages associated with a virtual address space; that is, the sets of dynamically-allocated physical memory pages necessary to run a z/OS task or a Linux process

= the real partition memory pages associated with a virtual machine; that is, the sets of dynamically-allocated physical memory pages necessary to run a guest operating system in a virtual machine

P P P P P P P P P P P A A A A A A A A A A A R R R R R R R R R R R T T T T T T T T T T T I I I I I I I I I I I T T T T T T T T T T T I I I I I I I I I I I O O O O O O O O O O O N N N N N N N N N N N

CPU

Logical Channel Logical Channel Logical Channel Logical Channel Subsystem- Subsystem Subsystem Subsystem Image 1 Image 2 Image 3 Image 4

System z Physical Channel Subsystem

FICON Channel Path OSA Ethernet Adapter Transparently Shared by Transparently Shared By All Logical Partitions Logical Partitions Configured FICON Switch, to Channel Subsystem Switches, CU - Devices, etc Image N Network Links, etc.

Shared I/O Adapter Storage Controller Subchannel Physical Logical Image For Channel Adapter Logical Partition 1 Path Interface Device Partition 1 Image 1 (e.g., a FICON Subchannel channel path) Image For Logical Physical Partition 1 Device Device – Subchannel Channel Volume Image for Path Logical Logical Image 2 Partition 2 Device e.g., a Partition 2 Parallel Subchannel Channel Access Logical Image for Path Logical Volume Partition 3 Image 3 Device Partition 3 The I/O (PAV) operations for each logical Logical Subchannel Channel partition are Device Image for Path multiplexed Logical Partition N Image N within the Partition N adapter/channel path and on the associated I/O interface

° The I/O infrastructure (adapters/channels, their transmission links, and attached I/O resources are shared by LPARs at native speeds (without hypervisor involvement) ° I/O requests, their associated data transfers , and I/O interruptions flow between each OS instance and the shared I/O components, just as if the I/O components were physically dedicated to a single OS instance

Virtual CPU SIE: Start Interpretive Program “instruction” Execution Instruction stream Instructions

Hardware Load Load, Store, Add, ... Store Problem state Instructions Instruction Add Execution Controls Hardware or Firmware High-Frequency Set Storage Key, SSKE Control Instructions that Signal Processor, ... SIGP require virtualization Instruction SIE Interception Interpretation to hypervisor Control z/VM Handling Instructions that hypervisor Virtualization require hypervisor SIE z/VM virtualization E.g., Start CPU Assists Subchannel STATE Descriptor Logical CPU => Physical CPU Instruction Execution Unit

z/OS Guest z/VM Guest Linux Guest General General zIIP General General General zIIP Purpose zAAP Purpose Purpose Purpose zAAP IFL Virtual IFL Virtual Purpose Virtual Virtual Virtual Virtual Virtual Virtual Virtual Virtual Virtual Processo r Processo r Processor Processor Processor Processo r Processor Processor Processor Processor Processo r

JAVA General General General Linux Linux zAAP zIIP z/OS & z/OS & & DB2 Purpose Purpose Purpose Virtual Process Process Virtual Virtual Virtual Virtual DB2 DB2 Processor Processor XML Processor Processor Processor

z/VM hypervisor dynamically dispatches: 1. General-purpose Virtual processors on general-purpose Logical processors 2. zAAP Virtual processors on zAAP or general-purpose Logical processors 3. zIIP Virtual processors on zIIP or general-purpose Logical processors 4. IFL Virtual processors to IFL or general-purpose Logical processors

Shared Shared Shared Shared Shared Shared Shared Shared Shared Shared General General General General zAAP zAAP zIIP zIIP IFL IFL Purpose Purpose Purpose Purpose Physical Logical Logical Logical Physical Logical Logical Logical Logical Logical Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor

General-Purpose Logical Processor zAAP Logical zIIP Logical IFL Logical Pool Processor Pool Processor Pool Processor Pool

LPAR 1 Virtual Machine 1 Virtual Virtual Machine N Machine 2

z/OS or Linux Virtual z/VM Virtual Machine Machine Virtual Memory Virtual Memory Space Spaces

z/VM Hypervisor z/VM Hypervisor Logical Partition Hypervisor

= the guest real memory pages associated with a virtual address space; that is, the sets of dynamically-allocated host virtual memory pages necessary to run a z/OS task or a Linux process

= the guest real memory pages associated with a virtual machine; that is, the sets of dynamically-allocated host virtual memory pages necessary to run a guest operating system in a virtual machine

Minidisk Cache (High-speed, Virtual Disk Virtual Disk in-memory disk in Memory in Memory R/W cache) T1 R/W R/W

R/W R/O R/O N

Linux B Linux R/W Linux R/W Shared 1 Application 2 A 3 C Code z/VM

TDISK : dynamic Minidisk : real disk temporary disk partitioning technology TDISK 1 Minidisk A allocation pool Fullpack N Minidisk B TDISK space Minidisk C Legend: R/W = Read/Write ESS 750, ESS 800, DS8000, DS6000 R/O = Read-Only

Linux Linux Linux Linux Linux Linux VM Guest Guest Guest Guest Guest Guest Switch NIC NIC NIC NIC NIC NIC Controller

Port 65 Port 66 Port 67 Port 68 Port 69 Port 70

Load Balancer Aggregator / Multiplexer

LACP z/VM Virtual Port 1 Port 2 Port 3 Port 4 Switch z/VM System z LPAR

OSA OSA OSA OSA

Port 1 Port 2 Port 3 Port 4 LACP (Link Aggregation Control Protocol) Real Switch

1. Send all Linux console output to a single CMS Virtual virtual machine server farm

Realtime 2. Use PROP and PROP Performance Graphs REXX to interrogate Toolkit console messages REXX LinuxLinux LinuxLinux for VM

ConsoleConsole CMS VirtualConsole CMS Reports, Console Historical Data

3. Issue hypervisor commands on behalf of Linux servers

Hypervisor CP CP Operations Monitor

1. Use the CP Monitor to automatically 2. Use Performance Toolkit for VM to capture performance and resource Monitor process Monitor data consumption data for each Linux server Data

Over 45 years of continuous innovation in virtualization zEC12 – Refined to support modern business requirements .. z196 – Exploiting hardware technology for economical growth , . lit y ibi System z10 lex , F ss System z9 z/VM tne us 64-Bit Live Guest ob zSeries Relocation R VM/ESA ty, Virtual Switch SSI ili 9672 ESA ab Guest LANs Set Observer eli 9x21 , R VM/XA Virtual Machine Resource Manager SMAPI lity bi 3090 31-Bit Virtual Disks in Storage Performance Toolkit ala Sc CMS Pipelines QDIO Enhanced Buffer State Mgmt e: 308x VM/HPO alu 64 MB Real Accounting Facility Minidisk Cache HiperSockets xCAT s V 303x es Absolute | Relative SHARE SIE on SIE Automated Shutdown sin 4381 VM/SP Bu SMP Discontiguous Saved Segments Named Saved Systems I/O Priority Queuing VM/370 Instruction TRACE Start Interpretive Execution (SIE) Host Page-Management Assist S/370 Programmable Operator (PROP) LPAR Hypervisor Integrated Facility for Linux HyperSwap Inter-User Communication Vehicle (IUCV) VM Assist Microcode Adapter Interruption Pass-Through CP-67 Conversational Monitor System (CMS) Dedicated I/O Processors Multiple Logical Channel Subsystems (LCSS) S/360 Diagnose Hypervisor Interface Program Event Recording (PER) Open Systems Adapter (OSA) Network Switching OpenStack Control Program Hypervisor Translation Look-Aside Buffer (TLB) REXX Interpreter Large SMP Dynamic Virtual Machine Timeout Dynamic Address Translation (DAT) Zone Relocation Expanded Storage Multiple Image Facility (MIF) N_Port ID Virtualization (NPIV) 1960s 1972 1980 1981 1988 1995 2004 2013 … IBM System z – a comprehensive and sophisticated suite of virtualization function