How to Virtualize with KVM Christian Bornträger <[email protected]> © 2018,2019 IBM Corporation Trademarks IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml The following are trademarks or registered trademarks of other companies. ● Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. © 2018,2019 IBM Corporation 2 Agenda ● Overview ● Components and management infrastructure ● Devices ● Performance measurement ● Problem determination ● Usability © 2018,2019 IBM Corporation 3 What is KVM ● KVM is the abbreviation for Kernel-based virtual machine ● It is part of the Linux ecosystem ● KVM is used as a building block in the industry – KVM components are used to build the public clouds ● KVM Forum conference had many participants from Tencent, AliBaba, Yandex, AWS, Google and many more – IBM HyperProtect Services also use KVM components – KVM is used to isolate containers (e.g. Kata containers) – KVM is used as a normal hypervisor instead of VMWare, z/VM, XEN and others © 2018,2019 IBM Corporation 4 How to get KVM ● Red Hat – RHEL 7.6-alt: Kernel 4.14, QEMU 2.12, Libvirt 4.5 – RHEL 8: Kernel 4.18, QEMU 2.12, Libvirt 4.5 ● SLES – SLES 12 SP4: Kernel 4.12, QEMU 2.11, Libvirt 4.0 SLES 15 SP1: Kernel 4.12; QEMU 3.1, Libvirt 5.0 ● Ubuntu – 16.04 LTS : Kernel 4.4, QEMU 2.5, Libvirt 1.3.1 – 18.04 LTS : Kernel 4.15, QEMU 2.11, Libvirt 4.0 © 2018,2019 IBM Corporation 5 How to get KVM ● Red Hat – yum install qemu-kvm libvirt virt-install – do a modprobe kvm (once) ● SLES – zypper install qemu-kvm libvirt virt-manager virt-install ● Ubuntu – apt install qemu-kvm libvirt-daemon libvirt-clients virt-manager © 2018,2019 IBM Corporation 6 ©2018,2019 Corporation IBM components ● ● ● BASE Others Management – – – – – – – – – – – … Openvswitch virt-install Kubevirt libvirt provided lineVirsh command tool tool small management Virtmanager: Openstack: infrastructurebig management Libvirt: base layer management device emulation, QEMU: exploitation,...kvm Linux kernel: kvm module Linux base system .. LINUX kernel LINUX process LINUX process LINUX process HW guest QEMU KVM guest QEMU sie guest QEMU libvirt virtmanager openstack 7 Being a Linux process ● As KVM guests are just normal processes, KVM inherits a huge amount of features from Linux – SMT – Paging – CPU scheduling – ECKD, FCP, NVMe disks – OSA,ROCE,Hipersocket – Multiple subchannel sets – Parallel Access Volumes (PAVs) – Other storage protocols also available (ie: iSCSI, NFS, GFS2, OCFS2, GPFS) – Encryption of guest disks via dmcrypt – ….. © 2018,2019 IBM Corporation 8 Being related to x86 KVM ● As KVM on z shares a lot of code with the x86 variant , KVM inherits a huge amount of features from that – Live guest migration via TCP/IP – Installs can be automated via scripts (AutoYast, KickStart, PreSeed) – Support for both character and graphical consoles – Network IPL – Can emulate modern hardware such as DVDs via ISO files. – Supports live resize of CPUs, networking, and storage devices in both the KVM host and guest virtual servers – Can take snapshots of running virtual machines (not just the disk storage) – With “Copy On Write” storage, snapshots and cloning of virtual servers are near instantaneous – Support for nested virtualization (KVM under KVM) – ... © 2018,2019 IBM Corporation 9 Being on Z provides unique features ● When running on Z the mature hardware support for virtualization makes it easy to provide support – CPACF crypto hardware exploitable by Host and Guests – Guarded Storage for Java Pause-less Garbage Collection – CryptoExpress cards are exploitable by hosts and guests – support for large pages in guest and host © 2018,2019 IBM Corporation 10 Management by libvirt © 2018,2019 IBM Corporation 11 Domain XML <domain type='kvm'> <name>vs1</name> <memory>524288</memory> <vcpu>2</vcpu> <os> <type arch='s390x' machine='s390-ccw-virtio'>hvm</type> </os> <iothreads>1</iothreads> <devices> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' iothread='1' Host io='native'/> <source dev='/dev/disk/by-path/ccw-0.0.1000'/> vs1 <target dev='vda' bus='virtio'/> 512MB </disk> <interface type='direct'> <source dev='bond0' mode='bridge'/> eth0 <model type='virtio'/> vda </interface> <console type='pty'> <target type='sclp'/> </console> </devices> bond0 </domain> Dont worry! Most things can be handled by tools! 0.0.1000 0.0.2000 0.0.3000 © 2018,2019 IBM Corporation 12 Disk storage © 2018,2019 IBM Corporation 13 Disk Storage Options – Guest View ● KVM on Z provides no storage emulation ● Paravirtualized storage provided to the guest using virtio interfaces – Virtual hard disks and CD/DVD drives ● Passthrough of host devices in progress © 2018,2019 IBM Corporation 14 Disk Storage Options – Host Backing ● Image files (raw, QCOW) – Residing in host filesystem – Flexible and space efficient ● Full disks (ECKD, FCP, other SAN) – Requires planning – Best performance ● Network storage (NAS) – Image files stored on NFS, CIFS, ... © 2018,2019 IBM Corporation 15 Disk Storage (virtio) Host <disk type='file' device='disk'> vs1 vs2 vs3 <source file='/var/libvirt/images/img01'/> <driver name='qemu' type='raw' io='native' vda vda vda cache='none' iothread='1'/> <target dev='vda' bus='virtio'/> </disk> <disk type='block' device='disk'> img01 <source dev='/dev/dasda'/> dm-0 img00 <driver name='qemu' type='raw' io='native' cache='none' iothread='1'/> <target dev='vda' bus='virtio'/> </disk> multipath dasda dasdb <disk type='block' device='disk'> <source dev='/dev/dm-0'/> <driver name='qemu' type='raw' io='native' fcp0 fcp1 ficon0 ficon1 cache='none' iothread='1'/> <target dev='vda' bus='virtio'/> </disk> SCSI ECKD ECKD LUN vola volb © 2018,2019 IBM Corporation 16 Networking © 2018,2019 IBM Corporation 17 Networking Options ● KVM guest OS only sees virtio network interfaces ● Virtio interfaces are backed most commonly by – Host interfaces directly using macvtap – Linux bridges – OpenVSwitches ● Host interfaces can be – OSA – Hipersocket – PCI (with limitations) © 2018,2019 IBM Corporation 18 Think Switches Logical View ● Conceptually all network Virtual Server 1 Virtual Server 2 attachments can be viewed as switches VNICs eth0 eth1 eth0 ● Differences in capabilities and characteristics exist though p0 p1 p2 p3 … Switch Ports VLAN 42 VLAN 17 VLAN 42 Virtual Switch up0 …Uplink Bonding I/F bond0 (for HA) Host NICs eth0 eth1 © 2018,2019 IBM Corporation 19 Macvtap as a Switch Implementation View Logical View Virtual Server 1 Virtual Server 2 Virtual Server 1 Virtual Server 2 eth0 eth1 eth0 VNICs eth0 eth1 eth0 p0 p1 p2 p3 … Switch VLAN 42 VLAN 17 VLAN 42 Ports macvtap0@ macvtap1@ macvtap2@ Virtual Switch bond0.17 Bond0.42 Bond0.42 up0 …Uplink bond0.17 bond0.42 bond0 Bonding I/F bond0 (for HA) eth0 eth1 Host NICs eth0 eth1 © 2018,2019 IBM Corporation 20 Macvtap: a More Common Setup Implementation View Logical View Virtual Server 1 Virtual Server 2 Virtual Server 1 Virtual Server 2 eth0 eth0 VNICs eth0 eth0 p0 p1 p2 p3 … Switch Ports Virtual Switch macvtap0@ macvtap1@ bond0 bond0 up0 …Uplink bond0 Bonding I/F bond0 (for HA) eth0 eth1 Host NICs eth0 eth1 © 2018,2019 IBM Corporation 21 Macvtap Characteristics ● No extra setup required ● Fastest connectivity option ● Layer 2 Only ● Can use shared OSAs, hipersockets – But no sharing between migration hosts ● VLAN supported in access mode © 2018,2019 IBM Corporation 22 Linux Bridge as a Switch Implementation View Logical View Virtual Server 1 Virtual Server 2 Virtual Server 1 Virtual Server 2 VNICs eth0 eth1 eth0 eth0 eth1 eth0 tap0 tap1 tap2 p0 p1 p2 p3 … Switch br_vlan17 br_vlan42 VLAN 42 VLAN 17 VLAN 42 Ports Virtual Switch bond0.17 bond0.42 up0 …Uplink bond0 Bonding I/F bond0 (for HA) eth0 eth1 Host NICs eth0 eth1 © 2018,2019 IBM Corporation 23 Linux Bridge: a More Common Setup Implementation View Logical View Virtual Server 1 Virtual Server 2 Virtual Server 1 Virtual Server 2 VNICs eth0 eth0 eth0 eth0 tap0 tap1 p0 p1 p2 p3 … Switch Ports virbr0 Virtual Switch up0 …Uplink bond0 Bonding I/F bond0 (for HA) eth0 eth1 Host NICs eth0 eth1 © 2018,2019 IBM Corporation 24 Linux Bridge Characteristics ● Moderate setup requirements ● Layer 2 Only ● OSA must operate in bridgeport mode – Not shareable between KVM hosts ● Hipersockets must enable VNICC learning and flooding – Shareable between KVM hosts ● VLAN supported in access mode © 2018,2019 IBM Corporation 25 OpenVSwitch IS a Switch Logical View Logical View Virtual Server 1 Virtual Server 2 Virtual Server 1 Virtual Server 2 eth0 eth1 eth0 VNICs eth0 eth1 eth0 tap0 tap1 tap2 p0 p1 p2 p3 p0 p1 p2 p3 … Switch VLAN 42 VLAN 17 VLAN 42 VLAN 42 VLAN 17 VLAN 42 Ports ovsbr0 Virtual Switch up0 up0 …Uplink bond0 Bonding I/F bond0 (for HA) eth0 eth1 Host NICs eth0 eth1 © 2018,2019 IBM Corporation 26 OpenVSwitch Characteristics ● Requires some understanding of switches in general and OVS specifically ● Most flexible ● Layer 2 Only ● OSA must operate in bridgeport mode – Not shareable between KVM hosts ● Hipersockets must enable VNICC learning and flooding – Shareable between KVM hosts ● VLAN supported in access and trunk mode © 2018,2019 IBM Corporation 27 Other devices © 2018 IBM Corporation 28 What else ● virtio-balloon: lightweight memory hotplug ● virtio-random: share hardware random numbers ● Virtio-scsi: use a virtual SCSI HBA to talk to disks and others ● Virtio-vsock: network-less communication between guests<→host ● virtio-9p: file system passthrough ● Virtio-gpu: provide a frame buffer device (useful for VNC to the guest) ● Sclp line mode console ● ….