Virtualization Options for Hypervisors on Linux on IBM Z15 and IBM Linuxone III

Virtualization options for hypervisors on Linux on IBM z15 and IBM LinuxONE III

Richard Young IBM Executive IT Specialist IBM Z and LinuxONE IBM Systems Lab Services Architectural options for installations with Linux on IBM Z & LinuxONE Main Layers for Linux on Z Components - In Flight Security - At Rest - Key Management

- SUSE Linux Distribution - Red Hat - Ubuntu

- LPAR only - z/VM Virtualization - KVM - Containers, SSC, K8S, RH OCP

- OSA options Network attachments - RoCE / ISM ~ TCP, SMC-R, SMC-D - Hipersockets - Virtual (MacvTap, Vswitch, bond)

LPAR Management - DPM - PR / SM

- FCP / SCSI Storage - FICON ECKD & Disk attachments - Internal NVMe - Spectrum Scale Agenda

Ø Benefits of virtualization • Available options • Considerations for virtualization decisions • Virtualization options • Firmware hypervisors • Software hypervisors • Software Containers • Firmware hypervisor decision guide • Virtualization decision guide • Summary Why do we virtualize? What are the benefits of virtualization?

§ Simplification – use of standardized images, virtualized hardware, and automated configuration of virtual infrastructure § Migration – one of the first uses of virtualization, enable coexistence, phased upgrades and migrations. It can also simplify hardware upgrades by make changes transparent. § Efficiency – reduced hardware footprints, better utilization of available hardware resources, and reduced time to delivery. Reuse of deprovisioned or relinquished resources. § Resilience – run new versions and old versions in parallel, avoiding service downtime § Cost savings – having fewer machines translates to lower costs in server hardware, networking, floor space, electricity, administration (perceived) § To accommodate growth – virtualization allows the IT to be more responsive to business growth, hopefully avoiding interruption

4 © Copyright IBM Corporation 2020 Agenda

• Benefits of virtualization Ø Available options • Considerations for virtualization decisions • Virtualization options • Firmware hypervisors • Software hypervisors • Software Containers • Firmware hypervisor decision guide • Virtualization decision guide • Summary What hypervisors and virtualization options on Linux on IBM Z & LinuxONE

q IBM traditional PR/SM or via DPM (Dynamic Partition Manager) – Firmware based virtualization to securely share and partition hardware resources. DPM providing graphical interface & REST interfaces with simplified management, automation, and dynamic capability for LinuxONE. q IBM z/VM – IBM developed, software-based mainframe virtualization that can be traced back to the beginning of Virtualization in computing q Linux KVM – Open source software-based virtualization. Supports multiple hardware architectures. Kernel based virtual machines started in mid 2000’s. Available via Linux Distro’s. q Containers – System Containers and Application containers. Via Linux cgroups and namespaces, provide an isolated environment for applications to run. Containers share a single host kernel. qOCI based Containers - Standard for container with a toolset ( Docker, Podman, ..) image build process, an API & CLI, a registry. Clustering added with additional tools like Dock swarm, Kubernetes. q IBM Secure Service Container (SSC) – Special partition for fully encrypted workloads. Traditional system administrator access removed. Limited and encrypted network access.

6 © Copyright IBM Corporation 2020 Agenda

• Benefits of virtualization • Available options Ø Considerations for virtualization decisions • Virtualization options • Firmware hypervisors • Software hypervisors • Software Containers • Firmware hypervisor decision guide • Virtualization decision guide • Summary Considerations for virtualization decisions q Current in house standards – Distros q Open vs proprietary q Software supported in combination with it q Outage avoidance – Live migration/relocation q Hardware support – i.e. NVMe, CTC, ISM q Feature/Function and requirements q Colocation requirements ( z/OS, x86) ØLive relocation requirements x, y ,z q Available skill set in house to manage q Dynamic by design – No outages to change q Ability to hire talent with needed skills q Performance / Scalability rd q Learning curve / duration to become q Ecosystem – Documentation, training, 3 fluent/expert – Simplicity vs complexity party solutions and support q Level of Isolation / security q Cost – Direct / Indirect for additional features qMonitoring , Security, Automation, Auditing, q Certifications & Multitenancy requirements Time to train rd q Automation capability – Rest APIs or 3 party q Integrity and Isolation tooling – i.e. Kickstart deployment, qSecure boot OpenStack, or Ansible qSecure Execution qAutomated / Manual encryption 8 © Copyright IBM Corporation 2020 Agenda

• Benefits of virtualization • Available options • Considerations for virtualization decisions Ø Virtualization options • Firmware hypervisors • Software hypervisors • Software Containers • Firmware hypervisor decision guide • Virtualization decision guide • Summary IBM LinuxONE virtualization options overview

Server virtualization. There are typically Application isolation. There are typically dozens or hundreds of Linux servers in a thousands of Containers in Linux on KVM or z/VM LPAR. IBM Z.

IBM LinuxONE 2nd level virtualization Linux Linux for Test & QA only Linux SSC Linux 2

Linux Virtual Linux

Secure Linux CPUs Linux ServiceLinux (cores) Linux Container KVM Linux z/VM Linux (SSC) Virtual CPUs Server (cores) virtualization KVM z/VM

LPAR LPAR1 LPAR2 LPAR3 LPAR4 virtualization Logical (PR/SM or DPM) CPUs (cores)

Real P1 P2 P3 P4 P5 P6 P7 P8 CPUs* (cores)

P1 – P8 are Central Processor Units (CPU -> core) or Integrated Facility for Linux (IFL) Processors (IFL -> core) * - One shared Pool of cores per System only 10 2020 IBM Corporation Note: - LPARs can be managed by DPM or traditional PR/SM IBM Z virtualization options

Server virtualization. Typically dozens - hundreds of Linux Application isolation. There are typically servers in a KVM or z/VM LPAR collocated with z/OS or thousands of Containers in Linux on others IBM Z.

IBM Z 2nd level virtualization Linux z/OS for Test & QA only Linux SSC Linux 2

Linux Virtual Linux

Secure Linux CPUs Linux z/OS or ServiceLinux (cores) Linux Container KVM Linux z/VM z/VSE or Linux (SSC) Virtual z/TPF CPUs Server (cores) virtualization KVM z/VM

LPAR LPAR1 LPAR2 LPAR3 LPAR4 LPAR5 virtualization Logical with PR/SM or CPUs DPM (With (cores) supported operating Real systems) P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 CPUs* (cores)

P1-P2 CPUs as general-purpose CPs, P3 – P10 are Central Processor Units (CPU -> core) or Integrated Facility for Linux (IFL) Processors (IFL -> core) * - One shared Pool of cores by type per System only 11 Note: - LPARs can be managed by DPM or traditional PR/SM 2020 IBM Corporation Architectural Options 1) Firmware hypervisor management • Traditional PR/SM • IBM Dynamic Partition Manager

12 © Copyright IBM Corporation 2020 2020 IBM Corporation Traditional PR/SM Management • Does not have to be implemented in a dynamic manner, but can be under the right conditions and process. You must take care to be dynamic capable. • You could build an assembler macro deck in a text file, but this is error prone and very labor intensive. • Typically you would build an IODF and populate an IOCDS from the IODF. • An IODF can be created and managed with HCD or HCM. • Both of these programs are available in z/OS and z/VM environments • In a z/VM environment, HCD manages an IODF and IOCDS, but does not have the panel system to build the IODF, that is where HCM is used. • z/VM can also manage the IO configuration via CP commands • While HCM graphically builds an IODF, it does not write an IOCDS or activate a new configuration.

13 © Copyright IBM Corporation 2020 What is IBM Dynamic Partition Manager?

• Built on existing PR/SM technology capabilities

• Simplified, consumable, enhanced, partition life-cycle and integrated dynamic I/O management capabilities LINUX LINUX LINUX SSC • Provides the technology foundation that enables APIs for IaaS and secure, private KVM Clouds

PR/SM DPM IBM DPM Powerful and easy HMC

14 © Copyright IBM Corporation 2020 Architectural decisions for LPAR level virtualization management PR/SM - Processor Resource/Systems Manager • For Mixed workload (i.e. z/OS & Linux) with all features supported • For LinuxONE with all HW features supported • Needs specialized skill for new IBM Z & LinuxONE Admins • Requires use of HCD and optionally HCM to manage the IO configuration, which comes with z/VM or z/OS LINUX LINUX LINUX SSC DPM – Dynamic Partition Manager KVM • For Linux, z/VM, KVM, and SSC only, no z/OS, VSE, or others • Intuitive Graphical interface , all configuration from HMC PR/SM DPM

• REST APIs for integration in SDD - Software Defined Datacenter HMC • Python and Ansible libraries for REST APIs • No support for z/VM SSI & LGR (CTC support – required) • No support yet for ISM and GDPS Appliance (NVMe is supported) • Requires FC0016 and two 1000BaseT adapters

15 © Copyright IBM Corporation 2020 Architectural Options 1) Firmware hypervisor management • Traditional PR/SM • IBM Dynamic Partition Manager

2) Optionally, one or more software hypervisor • IBM z/VM • KVM

16 © Copyright IBM Corporation 2020 2020 IBM Corporation Optionally a software Hypervisor, what if you choose none?

Ø Limited to the number of partitions the Ø Could still utilize containers for enhanced isolation machine supports. Inhibits scale. vs a single Linux instance Ø Eliminates any hypervisor imposed limits Ø Could still be virtualized later under KVM or z/VM (CPUs, memory, devices), only machine limits. Ø No chance for live migration/relocation or Ø Can use Secure Boot, but not Secure snapshots Execution. Ø Manual cloning may still be possible, but more Ø One less layer to administer, secure, patch, complex for ECKD environments. However backup, restore, test, and monitor scripted installs may be adequate. Ø Eliminates any overhead, but also any added Ø Can still utilize FlashCopy, Hyperswap, and value a hypervisor might bring. mirroring capabilities of storage servers. Ø Virtualization is not a consideration for any Ø Reduced skills need, scope of learning for a new debugging. solution more limited Ø Storage virtualization at the Hypervisor level is Ø Limited prebuilt automated solutions to create and lost – For example sparse qcow2, disk deploy Linux in to new partitions. (AutoYast, partitioning, or minidisks KickStart, Preseed don’t create a partition with needed resources)

17 © Copyright IBM Corporation 2020 The z/VM Hypervisor - Overview

Ø Proprietary commercial hypervisor optimized for IBM Z and LinuxONE Ø Virtualizes CPU, memory, disks, network, switches, and cryptographic hardware Ø Supports not only Linux, but CMS, z/OS, z/VSE, z/TPF, and z/VM guests Ø Resource sharing with priorities and CPU pooling for guests within and across CPCs. Ø Supports dynamically changing resource allocations Ø Cluster for up to four z/VM images or physical systems as members of a Single System Image (SSI) cluster Ø Live Linux Guest Relocation (LGR) between the nodes of a SSI cluster Ø Contains LDAP and RACF Security capabilities, now with Multi-Factor authentication Ø Cloud integration via IBM Cloud Infrastructure Center Ø Designed for FIPs 140-2 certification Ø Pervasively encrypt hypervisor paging volume

18 © Copyright IBM Corporation 2020 Combine LPARs with z/VM CPU Pooling

§LPAR with 5 Linux CPU / IFLs §Create 2 Pools – one with 4-CPU / cores and one with 1-CPU / core §Place the four WAS guests in the 4-cores pool and the two DB2 guests in the 1-core pool • Requires 4-core WAS entitlement • Requires 1-core DB2 entitlement

WAS WAS WAS WAS DB2 DB2 PVU Entitlements Guest Guest Guest Guest Guest Guest

2 vores 2 cores 2 cores 2 cores 1 cores 1 cores 700

600

cores Pool cores Pool 500 Capacity 4 cores Capacity 1 core 400 WAS LPAR with 5 cores 300 DB 2 200

100

§Avoids increase in software license requirements (and costs) 0 §Reduces z/VM system management and maintenance workload 5-cores LPAR With cores Pooling §Consolidates resources (memory, paging, network) for greater efficiency

Note: All PVU Entitlement examples based on a per core base (120PVU per core)

19 © Copyright IBM Corporation 2020 z/VM virtualization scalability

§ Proven support for up to 80 logical process or cores in host and 64 virtual cores in guest § Supports up to 2TB of central storage / 4TB in plan ( see continuous delivery news) § Virtual machine support for up to 1 TB of memory § 65635 IO devices (subchannels) for host, 24K devices per virtual machine § Concurrent IO to ECKD disk device = 1 (Up to 8 with HyperPAV) § Largest device: ECKD CMS mindisk (up to 22.5GB), EAV ~45GB. Non-CMS ECKD: 45GB / ~180GB EAV (CP only the first (64K cylinders). § Dynamically add processors, memory, disk devices, fiber channel, network adapters, cryptographic adapters … no disruption § Support for Linux and z/OS, z/VSE, z/TPF, and z/VM on same hypervisor

§ Continuous Delivery News: http://www.vm.ibm.com/newfunction/

20 © Copyright IBM Corporation 2020 z/VM Cluster functionality Single System Image Clustering and Live Guest Relocation

§ Previously optional priced feature –Now included in z/VM 7.1 base § Connect up to four z/VM systems as members of a Single System Image cluster § Cluster members can be run on the same or different physical System § Simplifies management of a multi-z/VM environment

- Single user directory - Cluster management from any member Cross- system communications for “ single system image ” management § Apply maintenance to all members in the cluster z/VM 1 z/VM 3 from one location § Issue commands from one member to operate on another

- Built-in cross-member capabilities Shared disks - Resource coordination and protection of network and disks z/VM 2 z/VM 4 Cross- system external network Private disks connectivity for guest systems

21 © Copyright IBM Corporation 2020 IBM Cloud Infrastructure Center 1.1.2 and z/VM V7

Ø Support for Linux Infrastructure as a Service Ø Rest API can be used to integrate with other Cloud Orchestration tools such as Terraform Ø Solution based in OpenStack Ussuri Ø Guest support for: RHEL 7.7, 7.8, 8.1, 8.2, Ø Very easy to install, vs build your own RHCOS 4.3, 4.4, 4.5, Ubuntu 20.04, SLES 15 Ø Management/Compute nodes hosted only on SP1, SLES 15 SP2 RHEL 8.2 Ø Self Service UI is tailorable with optional Ø HTTPS APIs request approvals, optional and tailorable expiration dates, optional and tailorable email Ø OpenStack CLI notifications and more Ø Web UI (Admin and User Self Service) Ø Utilizes DIRMAINT Extent Control pool for Ø Supports Ephemeral and persistent disk ephemeral storage Ø Persistent disk now on Storwise, DS8K Ø Persistent disk is FCP attached storage only. rd storage platforms. 3 party with San Volume Ø Optional LDAP integration Controller Ø Onboard existing virtual servers

22 © Copyright IBM Corporation 2020 IBM Wave for z/VM – Systems Management IBM Wave for z/VM provides the graphical interface that simplifies and helps to automate the management of z/VM and Linux virtual servers for Linux on Z and IBM LinuxONE systems

§ Monitors and manages virtual servers and resources from a single graphical interface § Simplifies and Automates tasks § Provisions virtual resources (Linux Guests, Network, Storage) § Supports advanced z/VM capabilities such as Single System Image and Live Guest Relocation § Allows delegation of administrative capabilities to the appropriate teams

A simple, intuitive graphical tool providing management, provisioning, and automation for a z/VM environment, supporting Linux virtual servers.

23 © Copyright IBM Corporation 2020 • KVM = “Kernel Virtual Machine” The KVM Hypervisor - The KVM Hypervisor - Overview •OverviewKVM is type 1 open source hypervisor in Linux implemented via optional packages

• The “KVM” module is added to Virtual Machine Virtual Machine Virtual Machine the Linux kernel to allow a user space programs to utilize the Linux Linux Linux hardware virtualization Applications Applications Applications

• KVM typically receives hypervisor Linux Linux Linux management via Libvirt which Guest OS Guest OS Guest OS

abstracts over different Other libvirt “hypervisors”: KVM, Xen.. exploiters Linux Host KVM Libvirt QEMU • Qemu (user space) typically used OpenStack to emulate devices with KVM IBM Z or LinuxONE • Para-virtualization provides a fast virsh CLI means of communication for guests Virtualization Manager to use devices on the host machine. aka Virt-Manager • All Virtio devices have two parts: the host device and the guest driver. © Copyright IBM Corporation 2020 Features Overview of KVM for IBM Z and LinuxONE

Features of KVM for Z Benefits

KVM Hypervisor § Supports running multiple Linux instances in a single partition

Processor virtualization § Supports sharing, dedicating prioritizing, limiting use of CPU resources by virtual servers Disk, Network, Cryptographic device virtualization § Sharing physical resources across virtual servers improves utilization § Boot from ECKD, LUN, NVMe, ISO, Image file (GPFS, NFS, local) Memory and CPU overcommit § Support over-commitment of memory & encrypted paging of memory

Live & Cold virtual server migration § Enables workload migration, maintenance of host Dynamic addition and deletion of virtual devices § Helps eliminate downtime to modify device configurations for virtual servers Thin provisioned virtual servers § Supports copy-on-write virtual disks which saves on storage by not needing full disks until used Installation/Configuration tools § Includes tools to automatically install and configure KVM Pervasively encrypt host boot disk (except /boot), § Enhanced Hypervisor Security swap, and all data volumes with protected keys Secure Boot (host only) and § Ensures you are only booting the Linux distributor provided code Secure Execution § Hardware enforced guest data isolation from host, encrypted initrd

25 © Copyright IBM Corporation 2020 KVM Live Migration and Clustering

Live Migration Inherits Linux clustering capabilities q Same as other KVM hardware platforms q Clustered filesystems q Does not require FICON CTCs • GPFS/Spectrum Scale q Requires only some sort of network • GFS2 connectivity • NFS v4 • OSA 1Gb, 10Gb, 25Gb • GlusterFS • RoCE Express 10Gb, 25Gb q Open Source - Linux HA q No cluster pre-definition required, other • PaceMaker than guest needs to target the same or • CoroSync newer machine model with access to the q Commercially - IBM System Automation same compute resources q Precopy and Postcopy live migration options q Can monitor status and convert the type of copy

26 © Copyright IBM Corporation 2020 KVM virtualization scalability KVM Guest Support Linux Host support • Guest domains - up to 4096 with • CPUs/core 256 (190 on z15) supporting memory and CPU • Memory Up to 16 TB (LPAR limit) • Up to 248 virtual cores • IO devices 64K per sub channel set • Virtio-ccw block devices – up to 1024 • Virtual NICs 8192 • Virtual IDE - up to 4 virtual • Fast internal network IO (Open Virtual Switch) • Virtio-ccw NICs – up to 1024 • Dynamic add/remove of CPU, Memory, and IO • Memory Overcommitment devices. • Virtual DVDs • Channel subsystems and Subchannel sets • Text and graphical consoles • RoCE Express, and ISM SMC-D, Hipersockets • RocE Express virtual functions • ECKD, SCSI, Internal NVMe Storage • CryptoExpress and AP Domains • Huge/Large Pages • Concurrent live guest migrations – 64 • Huge/Large pages

27 © Copyright IBM Corporation 2020 IBM Secure Execution for Linux

• Secure execution is a technical method to § Designed to guarantee a running guest in secure protect guest confidentiality and guest execution model (on-prem and cloud) provide data integrity from privileged administrator confidentiality against anybody including the host admin and data integrity. • Requires opt-in from Ø hardware, § Workload Protection against hypervisor administrator Ø hypervisor (host support), and − Protection from infrastructure admin attacks from Ø Linux os (guest support) outside § Protects against memory access of guest memory Guest support available with − Data on a running system cannot be seen outside § Red Hat RHEL 8.1, 7.8 guests § SUSE SLES 12 SP5 § Ubuntu 19.10 − Secure execution guest is close to the security profile Host support: of a distinct LPAR • SuSE SLES 15 SP2 • Ubuntu 20.04 LTS • Not available with z/VM and LPAR

Copyright IBM 2020 28 Technology Comparison IBM Secure Execution IBM Secure Service Container Price • No-cost offering (Enable via feature code) • Priced offering (via Hyper Protect Services) Features • Hardware-based security technology • Software-based secure hosting appliance • Straight forward deployment without requiring • Straight forward deployment without requiring code code changes to exploit security capabilities. changes to exploit security capabilities. • Unlimited scaling for secure environments • Scale up to 85 highly secure environments within the machine’s capacity • Provides management tooling and interface for • Applicable existing Linux and KVM tooling greater user experience • Supported operating environment: RHEL 8.>2, • Supported operating environment Ubuntu 16.04, SLES 15 SP2, Ubuntu 20.04 LTS 18.04 Security • Provides data confidentiality and integrity • Provides data confidentiality and integrity • Restricted Host administrator access to help • Restricted Administrator access to help prevent prevent misuse of privilege user credentials misuse of privilege user credentials • Encrypt boot disk, data, and initrd, in flight • Automatic pervasive encryption of data and code and at rest in-flight and at rest • Configurable allowance for disaster recovery • Tamper protection during installation or other designated machines • Additional protection from side channel attacks • Protected Memory for data and code

Automation Monitoring • IBM Cloud Infrastructure Center • IBM Tivoli Monitoring • OpenStack • Nagios • Chef • ELK • Ansible • Grafana • Puppet • Prometheus • Nagios • Zabbix • Kickstart, AutoYast, Preseed • OpenNMS • Linux HA (PaceMaker, CoroSync) • Cacti • IBM System Automation • Splunk

30 © Copyright IBM Corporation 2020 KVM Virtual Machine Manager

Guest Management • Lifecycle (stop,start,create,destroy) • Suspend/resume • Snapshotting • Cloning • Guest resource dynamic add/remove • Console (text & graphical) • Live resize CPU & Memory • Live Migration • Resource monitoring Host Management • Network resources • Storage resources

31 © Copyright IBM Corporation 2020 IBM Cloud Infrastructure Center 1.1.2 and RHEL KVM

Ø Support for Linux Infrastructure as a Service Ø Rest API can be used to integrate with other Cloud Orchestration tools such as Terraform, Ø Solution based in OpenStack Ussuri Red Hat CloudForms, or Ansible Ø Very easy to install, vs build your own Ø Guest support for: RHEL 7.8, 8.1, 8.2, Ø Management/Compute nodes hosted only on RHCOS 4.5 RHEL 8.2 Ø Self Service UI is tailorable with optional Ø HTTPS APIs request approvals, optional and tailorable expiration dates, optional and tailorable email Ø OpenStack CLI notifications and more Ø Web UI (Admin and User Self Service) Ø Support for Cold Migration Ø Supports Open Virtual Switch Ø Optional LDAP integration Ø Ephemeral / Local storage – Support any disk Ø Persistent disk support not yet available usable by the KVM host (Such FCP, FICON, internal NVMe)

32 © Copyright IBM Corporation 2020 Summary of KVM and z/VM for IBM LinuxONE

LPAR virtualization LPAR virtualization can be done via the new IBM DPM or traditional PR/SM

IBM z/VM KVM • World class quality, security, • Standardizes configuration and reliability - powerful and versatile IBM LinuxONE System operation of server virtualization • Scalability creates cost savings • Can leverage existing Linux opportunities skills/tools Linux • Exploitation of advanced Linux • Flexibility and agility leveraging the Linux Linux Linux on z on Linux z on Linux z on Linux technologies, such as: open-source community • Shared memory (Linux kernel, z/VM KVM • Shared memory (KSM) executables, communications) LPAR LPAR • Granular resource controls based in • Granular control over resource pool cgroups Processors, Memory and IO • Provides virtualization for all Z • Provides an open-source Support Element operating systems virtualization choice • Can integrate with external tools via • Utilizes standard libvirt API exposed z/VM REST APIs • Full OpenStack deployment possible

33 © Copyright IBM Corporation 2020 Architectural decisions for Hypervisors

Target environments for KVM Target environments for z/VM

(New) Linux environments that … Linux installs that … • Committed to open technologies, • Already use z/VM for Linux workloads open-source oriented • Are skilled in z/VM and prefer a mature model • Are x86 centric, and are affine with KVM or • Are invested in tooling for z/VM environment containers • Require technical capabilities in z/VM (e.g., • Have Linux admin skills with KVM GDPS managed HyperSwap...) • Need to integrate into a distributed • Require a software certification (e.g., Oracle Linux/KVM environment, using standard DB) interfaces (ie OpenStack deployment) • Virtualization of non-Linux operating systems • Storage server managed HyperSwap

Both can coexist on the same server in different partitions should you require capabilities/features of both.

34 © Copyright IBM Corporation 2020 Architectural Options 1) Firmware hypervisor management • Traditional PR/SM • IBM Dynamic Partition Manager

2) Optionally, one or more software hypervisor • IBM z/VM • KVM

3) Optionally, one or more container technologies • Basic container, Docker, Podman, and CRIO • IBM SSC and HPVS • Red Hat OCP

35 © Copyright IBM Corporation 2020 2020 IBM Corporation Linux control groups and namespaces are used for isolation

To simplify: Container 1 Container 2 - “cgroups” will control resources in your container Kernel Kernel l CPU Namespaces Namespaces l Memory

l Disk I/O throughput App App

- “namespaces” will isolate cgroups cgroups App App App

l process IDs

l Hostnames Kernel

l User IDs Linux Guest l network access

l interprocess communication

l filesystems Systemd service unit file l Today cgroups are integrated in to systemd and can control and … account for resource usage [Service] l For more information on systemd resource control information see: CPUQuota=40% https://www.freedesktop.org/software/systemd/man/systemd.resourc MemoryLimit=1200000 e-control.html …

Build Ship Run Describes steps to build a container image Operator Dockerfile for Deploys Application Containers containerization Container A Container N Image N Image Program Container Build M Container Container B Engine Get N Run N Code Image … Repository (Build) Repository

Push new Developer actions: Image to Container Engine Repository Ø Develops App, Ø Builds Container, Host OS Ø commits image Ø pushes Container Server image to Registry

Red Hat provides a new container tooling • RHEL 8 can still run Docker images ecosystem • Podman - implements almost all the Docker CLI Ø Container build & deployment: Podman commands, without a daemon as required in a Ø Container building: Buildah legacy Docker environment

Ø Manage container images and registry: Skopeo • For the most basic deployment, containers could be launched using the podman command from a Ø Cri-o: container runtime used in Red Hat Systemd unit file. Openshift v4 • However one would typically use a Kubernetes From RHEL 8 and Fedora 29, no Docker based environment to manage containers engine is included.

© Copyright IBM Corporation 2017. Technical University/Symposia materials may not be reproduced in whole or in 38 part without the prior written permission of IBM. Container engine and tooling availability

RHEL docker podman RHEL 7.5 1.13 0.9.2 Docker is available and supported in RHEL 7.6 1.13 1.4.4 • Ubuntu 16.04 and later RHEL 7.7 1.13 1.4.4 • RHEL 7.5-7.8 with extras repository RHEL 8 - 1.05 • SLES 15 and later RHEL 8.1 - 1.6.4 RHEL 8.2 - 1.6.4 Podman is able and supported in RHEL 8.3 - 2.0.5.5 • RHEL 7.5 and later • SLES 15 SP1 and later SLES docker podman SLES 15 17.09 -

SLES 15 SP1 18.09.1 1.0.1 Docker as a community edition • Ubuntu 16.04 and later Ubuntu docker podman • Fedora 28 and later 16.04 LTS 18.09.7 -

18.04 LTS 18.09.7 -

20.04 LTS 19.03.8 -

Open-source platform for managing containerized workloads and services

What it provides: • Service discovery and load balancing • Storage orchestration What it is not: • Automated rollouts and rollbacks • Does not limit the types of applications supported • Does not deploy source code and does not build your • Automatic back pinning (ie how much application compute resource) • Does not provide application level middleware (ie J2EE, • Self-healing (ie Restarts containers that Messaging, Database) failed) • Does not dictate logging, monitoring, or alerting • Does not provide or mandate a configuration language or • Secret and configuration management system • Does not provide or adopt any comprehensive machine configuration, maintenance, or management

• Deployed with juju • Juju gui • Switched from Docker to contianerd • Using snaps, point releases of Kubernetes are automatically installed. • Whenever there is a new major version, use Juju charm. • LXD based CDK environment on top of a s390x LPAR: http://ubuntu-on-big- iron.blogspot.com/2019/08/deploy-cdk- on-ubuntu-s390x.html

• Image registry • OpenShift IS Kubernetes, 100% Certified by the CNCF • Certified Databases • pre-built dashboards, metrics, alerts that come from Red Hat’s experience in monitoring clusters • Certified Middleware • Authentication with a built-in oauth provider, integration to • Automated Image builds your identity providers such as LDAP, ActiveDirectory, • CI/CD and devops workflow OpenID Connect • Monitoring • OpenShift 4 also adds a graphical UI (the developer console) dedicated to developers, allowing them to easily • Log aggregation deploy applications to their namespaces from different sources (git code, external registries, Dockerfile…) • Zero downtime and patching • OpenShift provides Codeready sets of tools for developers, • Validated storage plugins such as Codeready Workspaces, a fully containerized web- • Metering and chargeback IDE that runs on top of OpenShift itself, providing an IDE- as-a-service experience.

See: https://www.openshift.com/learn/topics/kubernetes/#what-openshift-adds and https://www.openshift.com/blog/enterprise-kubernetes-with-openshift-part-one

IBM LinuxONE IBM Z

OpenShift OpenShift z/OS zCX Wazi z/OSMF Ansible automation IMS

Connect transactional z/OS z/OS services CICS z/VM 7.x z/VM 7.x

VSWITCH VSWITCH VB DB2

LPAR LPAR LPAR OSA/ RoCE OSA/ RoCE HS OSA

• Available February 28, 2020 for on premises deployment • Deployed in a Secure Service Container appliance • You deploy container images • Solution supplies Ubuntu base images with and without ssh. • Managed via non-SSC host thru CLI / Rest interface • Store source in Git • Build with Secure build server and Push the built image to remote Docker repository such as DockerHub or IBM Container Registry • Available Grep11 container communicates with Hardware Security Module (HSM) and generates asymmetric (public and private) key pairs. • collectd-host/monitoring-host can be used to collect metrics from Secure Service Container framework.

• Due to the simplified, dynamic, and agile nature of DPM, one would plan to use DPM by default unless you have an overriding requirement that prevents you from utilizing it. • Situations that would require you to use traditional PR/SM management:

Requirements driving traditional PR/SM

z/OS, VSE, or TPF operating systems These would not run on a LinuxONE GDPS Virtual appliance HA/DR can be managed in other ways z/VM SSI Requires FICON CTC today, DPM does not support this requirement Internal NVMe SSD Can not configure internal NVMe with DPM FICON attached TAPE SCSI tape possible. Would not expect FICON tape in a LinuxONE only environment Internal Shared Memory (ISM) and SMC-D Only RoCE Express cards and HiperSockets. Consider shared RoCE Express port instead of ISM

Virtualization Description Advantages Considerations Option Divide one physical LinuxONE system into up to 85 • Comes with the system • Configuration for a non-Z Traditional logical partitions (LPARs) running isolated and • Powerful / Efficient administrator. PR/SM secured in parallel. Share resources across LPARs • Supports z/OS, z/VSE, z/TPF, z/VM, Linux • Need HCD and possibly HCM. or dedicate to a particular LPAR. • Web API • Care to be dynamic

Dynamic Easy-to-use version of PR/SM, designed to make it • Comes with the system • No z/OS, z/VSE, z/TPF support easy for distributed systems administrators to set up • Powerful / Efficient • No CTC support, required by Partition and manage the system. • Easy to use, Dynamic by default z/VM Single System Image Manager • Supports z/VM, Linux, KVM, SSC • NVMe not supported yet (IBM DPM) • Web API, Python, and Ansible • Requires two 1000BaseT OSA IBM proprietary server virtualization. Supported on • Very mature, efficient and resilient • Requires unique skills IBM LinuxONE Systems. z/VM will continue to be • CPU Pooling across LPARs • Steep learning curve for z/VM enhanced to support Linux workloads. • Only hypervisor Oracle certifies on Z & LinuxONE distributed administrators • Optional IBM Wave or IBM Cloud Infrastructure • Costs more than “free” KVM Center • Many add on products

KVM provides an open-source choice for IBM • Included with the Linux distros, Open-source • Each distro has their own version LinuxONE systems virtualization for Linux. • Community-based development of KVM –some inconsistencies Best for when the staff is not familiar with z/VM and • Easy for Linux admins, greatly reduced learning • Not supported by Oracle on any KVM have Linux experience • Includes OpenStack support platform • Secure Execution – Hardware enforced isolation • NVMe host and guest including boot support Containers Crio runtime is built with Kubernetes in mind • Doesn’t need a hypervisor for isolating apps • Varied levels of isolation Kubernetes is the industry standard container • More efficient for applications and microservices • Existing applications may need Kubernetes manager/orchestrator • Highly portable – ubiquitous across platforms, rework OCP Changes development and operational models! clouds, non-cloud • Images need to support s390x SSC SSC – Encrypted by default • Docker/Podman included with Linux distros • SSC limited by partition scale

48 © Copyright IBM Corporation 2020 Virtualization decision guide from software perspective Linux on Linux under Linux under Linux Use Case LPAR KVM z/VM Containers Greater than 40/85 instances ● ● ● ● Product Support - Oracle DB Certification ●<86/● ●/● ● ●/● Product Support - Db2 LUW ●<86/● ● ● ● MongoDB, PostgreSQL, MariaDB, etc. ●<86/● ● ● ● No IBM Z skills, require simplified / easy ● ● ●/● ● Cloud-native microservices, / Kubernetes ● ●value?/● ●value?/● ● Product Support - Red Hat OCP ● ● ● ● Java, Node.js, J2EE ●<86/● ● ● ● Secure Boot Integrity Check ● ● ● ● Secure Execution Isolation ● ● ● ● Non-Docker apps (monolithic apps) ●<86/● ● ● ● 49 © Copyright IBM Corporation 2020 Agenda

• Benefits of virtualization • Available options • Considerations for virtualization decisions • Virtualization options • Firmware hypervisors • Software hypervisors • Software Containers • Firmware hypervisor decision guide • Virtualization decision guide Ø Summary Architectural decision for Virtualization summary q The virtualization decision is a per LPAR decision on IBM Z and IBM LinuxONE! • Up to 40 LPARs on Rockhopper II, up to 85 on LinuxONE III q There are 4 virtualization options for LinuxONE & Linux on Z • Linux on “bare metal” (Linux on LPAR, no software hypervisor) • Linux under KVM • Linux under z/VM • Linux Containers – in a Linux LPAR, under z/VM, or under KVM q All options have some advantages and all have considerations beyond what we covered here We’re happy to help you in your virtualization decision q Software containers can be used in Linux on LPAR, Linux on z/VM, or Linux on KVM q Oracle is certified under z/VM but not under KVM (or VMWare) q SLES, Red Hat & Ubuntu all include their own KVM hypervisor q All 3 distributions include Container engines today q Send us your observations!