Introduction to Virtualization

z Systems

SHARE Orlando Linux and VM Program

Romney White, IBM romneyw@us.ibm.com z Systems Architecture and Technology

° Introduction to Virtualization – Concept – Server Virtualization Approaches – Hypervisor Implementation Methods – Why Virtualization Matters ° Virtualization on z Systems – Logical Partitions – Virtual Machines

Virtual Resources ó Proxies for real resources: same interfaces/functions, different attributes ó May be part of a physical resource or multiple physical resources

Virtualization óCreates virtual resources and "maps" them to real resources óPrimarily accomplished with software or firmware

Resources óComponents with architecturally-defined interfaces/functions óMay be centralized or distributed - usually physical óExamples: memory, disk drives, networks, servers

ó Separates presentation of resources to users from actual resources ó Aggregates pools of resources for allocation to users as virtual resources

Hardware Partitioning Bare-metal Hypervisor Hosted Hypervisor Apps ... Apps Apps ... Apps Apps ... Apps OS OS OS OS OS OS

Adjustable partitions Hypervisor Hypervisor Partition Controller Host OS SMP Server SMP Server SMP Server

Server is subdivided into fractions Hypervisor provides fine-grained Hypervisor uses OS services to each of which can run an OS timesharing of all resources do timesharing of all resources

Physical partitioning Hypervisor software/firmware Hypervisor software runs on S/370™ SI-to-PP and PP-to-SI runs directly on server a host operating system Sun Domains, HP nPartitions z Systems LPAR and z/VM ® VMware GSX ® Logical partitioning POWER™ Hypervisor Microsoft Virtual Server IBM eServer™ pSeries ® LPAR VMware ESX Server HP Integrity VM HP vPartitions Xen Hypervisor KVM

Characteristics: • Bare-metal hypervisors offer high efficiency and availability • Hosted hypervisors are useful for clients where host OS integration is important • Hardware partitioning is less flexible than hypervisor-based solutions

Virt Mach • VM runs in user mode Virt Mach • VM runs in user mode L • All privileged instructions L • Some IA-32 instructions must A cause traps A be replaced with trap ops ST ST PrivOp TrapOp L Trap Hypervisor PrivOp L Trap Hypervisor PrivOp emulation code emulation code ......

Examples : CP-67, VM/370 Examples : VMware, Microsoft VS Benefits : Runs unmodified OS Benefits : Runs unmodified, translated OS Issues : Substantial overhead Issues : Substantial overhead

Hypervisor Calls (“Paravirtualization”) Direct Hardware Virtualization

Virt Mach • VM runs in normal modes Virt Mach • VM runs in normal modes L • OS in VM calls hypervisor L • Hardware does most of the A to access real resources A virtualization (SIE architecture) ST ST • Hypervisor provides control Hcall PrivOp Hypervisor L Call L Exit Hypervisor service service ......

Examples : POWER Hypervisor, Xen Examples : z Systems LPAR, z/VM, Xen Benefits : High efficiency Benefits : High efficiency, runs unmodified OS Issues : OS must be modified to issue Hcalls Issues : Requires underlying hardware support

Roles: Possible Benefits: Virtual Servers Consolidations High resource utilization

Dynamic provisioning / hosting Great usage flexibility Workload management Enhanced workload QoS Workload isolation Virtualization High availability / security Software release migration Low cost of availability Mixed production and test Low management costs Physical Mixed OS types/releases Server Enhanced interoperability Reconfigurable clusters Legacy compatibility Low-cost backup servers Investment protection

Virtualization can fill many roles and provide many benefits In the final analysis, its potential benefits take three forms: • Help reduce hardware costs – Help increase physical resource utilization – Small footprints • Can improve flexibility and responsiveness – Virtual resources can be adjusted dynamically to meet new or changing needs and to optimize service level achievement – Virtualization is a key enabler of on demand operating environments such as cloud • Can reduce management costs – Fewer physical servers to manage – Many common management tasks become much easier

HiperSockets & Virtual Networking and Switching ° Multi-dimensional virtualization Web technology Sphere WebSphere Traditional SysAdmin – z Systems provides Test Apache OLTP and FTP Test Tools Batch logical (LPAR) and software (z/VM) Linux Linux CMS Linux WLM WLM Linux z/OS partitioning z/OS z/OS – PR/SM enables highly scalable virtual server z/VM Control Program z/VM hosting for LPAR and z/VM virtual machine I/O & Network environments Memory Memory Memory Memory – IRD coordinates allocation of CPU IFL Processors Processors and I/O resources among z/OS and L P A R L P A R L P A R L P A R non-z/OS ® LPARs* Intelligent Resource Director (IRD)

Processor Resource / System Manager (PR/SM) * Excluding non-shared resources like Integrated Facility for Linux processors

Logical CPU SIE: Start Interpretive Program “instruction” Execution Instruction stream Instructions

Hardware Load Load, Store, Add, ... Store Problem state Instructions Instruction Add Execution Controls Hardware or Firmware High-Frequency Start Subchannel, SSCH Control Instructions that Test Subchannel, ... TSCH require virtualization Instruction SIE Interception Interpretation to hypervisor Low-Frequency LPAR Handling Control hypervisor LPAR CPU Virtualization Instructions that SIE require hypervisor E.g., Modify STATE Assists virtualization Subchannel Descriptor

Physical CPU Instruction Execution Unit

z/VM Logical z/OS Logical Partition z/VM Logical Partition Partition General General zIIP General General General zIIP Purpose Purpose zAAP Purpose Purpose Purpose zAAP IFL Logical IFL Logical Logical Logical Logical Logical Logical Logical Logical Logical Logical Processo r Processo r Processor Processor Processor Processo r Processor Processor Processor Processor Processo r

JAVA General General General IFL IFL zAAP zIIP Virtual z/OS & z/OS & & DB2 Purpose Purpose Purpose Processor Virtual Virtual Virtual Processor Processor DB2 DB2 XML Virtual Virtual Virtual Processor Processor Processor Processor

LPAR hypervisor dynamically dispatches: 1. General-purpose logical processors on general-purpose physical processors 2. zAAP logical processors on zAAP physical processors 3. zIIP logical processors on zIIP physical processors 4. IFL logical processors to IFL physical processors

Shared Shared Shared Shared Shared Shared Shared Shared Shared Shared General General General General zAAP zAAP zIIP zIIP IFL IFL Purpose Purpose Purpose Purpose Physical Physical Physical Physical Physical Physical Physical Physical Physical Physical Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor

Shared IFL Shared General-Purpose Shared zAAP Shared zIIP Physical Processor Physical Processor Pool Physical Processor Physical Processor Pool Pool Pool

A collection of up to LPAR 1 LPAR 2 LPAR N 231 - or A 264 -bytes collection virtual of up to address 64 spaces 2 -bytes z/VM Virtual Machine z/OS or Linux High- virtual Each Physical Memory Space Performance Logical address virtual spaces machine Partition Physical Memory may have Spaces z/OS and its own Linux virtual exploit address multiple spaces such mapped virtual to this address common z/VM Hypervisor spaces logical partition Logical Partition Hypervisor address space

= the real partition memory pages associated with a virtual address space; that is, the sets of dynamically-allocated physical memory pages necessary to run a z/OS task or a Linux process

= the real partition memory pages associated with a virtual machine; that is, the sets of dynamically-allocated physical memory pages necessary to run a guest operating system in a virtual machine

P P P P P P P P P P P A A A A A A A A A A A R R R R R R R R R R R T T T T T T T T T T T I I I I I I I I I I I T T T T T T T T T T T I I I I I I I I I I I O O O O O O O O O O O N N N N N N N N N N N

CPU

Logical Channel Logical Channel Logical Channel Logical Channel Subsystem- Subsystem Subsystem Subsystem Image 1 Image 2 Image 3 Image 4

z Systems Physical Channel Subsystem

FICON Channel Path OSA Ethernet Adapter Transparently Shared by Transparently Shared By All Logical Partitions Logical Partitions Configured FICON Switch, to Channel Subsystem Switches, CU - Devices, etc Image N Network Links, etc.

Shared I/O Adapter Storage Controller Physical Adapter Subchannel Image For Interface (e.g., a Logical Logical Channel FICON channel path) Partition 1 Path Device Partition 1 Image 1 Subchannel Image For Logical Physical Partition 1 Device Device – Subchannel Channel Volume Image for Path Logical Logical Image 2 Partition 2 Device e.g., a Partition 2 Parallel Subchannel Channel Access Image for Path Logical Volume Logical Partition 3 Image 3 TTThe I/O operations for Device (PAV) Partition 3 each logical partition are multiplexed within the Logical adapter/channel path and Subchannel Channel Device Image for Path on the associated I/O Logical Partition N Image N interface Partition N

° The I/O infrastructure (adapters/channels, their transmission links, and attached I/O resources are shared by LPARs at native speeds (without hypervisor involvement) ° I/O requests, their associated data transfers , and I/O interruptions flow between each OS instance and the shared I/O components, just as if the I/O components were physically dedicated to a single OS instance

Virtual CPU SIE: Start Interpretive Program “instruction” Execution Instruction stream Instructions

Hardware Load Load, Store, Add, ... Store Problem state Instructions Instruction Add Execution Controls Hardware or Firmware High-Frequency Set Storage Key, SSKE Control Instructions that Signal Processor, ... SIGP require virtualization Instruction SIE Interception Interpretation to hypervisor Control z/VM Handling Instructions that hypervisor z/VM CPU Virtualization require hypervisor SIE virtualization E.g., Start STATE Assists Subchannel Descriptor

Logical CPU => Physical CPU Instruction Execution Unit

z/OS Guest z/VM Guest Linux Guest General General zAAP General General General zAAP Purpose Purpose Virtual zIIP Virtual Purpose Purpose Purpose Virtual zIIP Virtual IFL Virtual IFL Virtual Virtual Virtual Processo r Virtual Virtual Virtual Processo r Processo r Processo r Processor Processor Processor Processor Processor Processor Processor

JAVA General General General zIIP Virtual Linux Linux z/OS & z/OS & DB2 Purpose Purpose Purpose zAAP Process Process & Virtual Processor DB2 DB2 XML Virtual Virtual Virtual Processor Processor Processor Processor

z/VM hypervisor dynamically dispatches: 1. General-purpose Virtual processors on general-purpose Logical processors 2. zAAP Virtual processors on zAAP or general-purpose Logical processors 3. zIIP Virtual processors on zIIP or general-purpose Logical processors 4. IFL Virtual processors to IFL or general-purpose Logical processors

Shared Shared Shared Shared Shared Shared Shared Shared Shared Shared General General General General zAAP zAAP zIIP zIIP IFL IFL Purpose Purpose Purpose Purpose Physical Logical Logical Logical Physical Logical Logical Logical Logical Logical Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor

General-Purpose Logical Processor zAAP Logical zIIP Logical IFL Logical Pool Processor Pool Processor Pool Processor Pool

LPAR 1 Virtual Machine 1 Virtual Virtual Machine N Machine 2

z/OS or Linux Virtual z/VM Virtual Machine Machine Virtual Memory Virtual Memory Space Spaces

z/VM Hypervisor z/VM Hypervisor Logical Partition Hypervisor

= the guest real memory pages associated with a virtual address space; that is, the sets of dynamically-allocated host virtual memory pages necessary to run a z/OS task or a Linux process

= the guest real memory pages associated with a virtual machine; that is, the sets of dynamically-allocated host virtual memory pages necessary to run a guest operating system in a virtual machine

Minidisk Cache (High-speed, Virtual Disk Virtual Disk in-memory disk in Memory in Memory R/W cache) T1 R/W R/W

R/W R/O R/O N

Linux B Linux R/W Linux R/W Shared 1 Application 2 A 3 C Code z/VM

TDISK : dynamic Minidisk : real disk temporary disk partitioning technology TDISK 1 Minidisk A allocation pool Fullpack N Minidisk B TDISK space Minidisk C Legend: R/W = Read/Write ESS 750, ESS 800, DS8000, DS6000 R/O = Read-Only

Linux Linux Linux Linux Linux Linux VM Guest Guest Guest Guest Guest Guest Switch NIC NIC NIC NIC NIC NIC Controller

Port 65 Port 66 Port 67 Port 68 Port 69 Port 70

Load Balancer Aggregator / Multiplexer

LACP z/VM Virtual Port 1 Port 2 Port 3 Port 4 Switch z/VM z Systems LPAR

OSA OSA OSA OSA

Port 1 Port 2 Port 3 Port 4 LACP (Link Aggregation Control Protocol) Real Switch

1. Send all Linux console output to a single CMS Virtual virtual machine server farm

Realtime 2. Use PROP and PROP Performance Graphs REXX to interrogate Toolkit console messages REXX LinuxLinux LinuxLinux for VM

ConsoleConsole CMS VirtualConsole CMS Reports, Console Historical Data

3. Issue hypervisor commands on behalf of Linux servers

Hypervisor CP CP Operations Monitor

1. Use the CP Monitor to automatically 2. Use Performance Toolkit for VM to capture performance and resource Monitor process Monitor data consumption data for each Linux server Data

Over 45 years of continual innovation in z13 virtualization zEC12 – Refined to support modern business requirements z196 – Exploiting hardware technology for economical growth z Systems10 z z/VM Systems9 64-Bit Live Guest zSeries Relocation VM/ESA Virtual Switch SSI 9672 ESA 9x21 Guest LANs Set Observer VM/XA Virtual Machine Resource Manager SMAPI 3090 31-Bit Virtual Disks in Storage Performance Toolkit 308x VM/HPO CMS Pipelines QDIO Enhanced Buffer State Mgmt 303x 64 MB Real Accounting Facility Minidisk Cache HiperSockets xCAT 4381 VM/SP Absolute | Relative SHARE SIE on SIE Automated Shutdown SMP Discontiguous Saved Segments Named Saved Systems I/O Priority Queuing VM/370 Instruction TRACE Start Interpretive Execution (SIE) Host Page-Management Assist S/370 Programmable Operator (PROP) LPAR Hypervisor Integrated Facility for Linux HyperSwap Inter-User Communication Vehicle (IUCV) VM Assist Microcode Adapter Interruption Pass-Through CP-67 Conversational Monitor System (CMS) Dedicated I/O Processors Multiple Logical Channel Subsystems (LCSS) S/360 Diagnose Hypervisor Interface Program Event Recording (PER) Open Systems Adapter (OSA) Network Switching OpenStack Control Program Hypervisor Translation Look-Aside Buffer (TLB) REXX Interpreter Large SMP Dynamic Virtual Machine Timeout Dynamic Address Translation (DAT) Zone Relocation Expanded Storage Multiple Image Facility (MIF) N_Port ID Virtualization (NPIV) 1960s 1972 1980 1981 1988 1995 2004 2013 … IBM z Systems – a comprehensive and sophisticated suite of virtualization function