Dynamically Provisioning Resources to KVM Hosted Linux Virtual Servers

Dynamically provisioning resources to KVM hosted Linux virtual servers

Richard Young Executive I.T. Specialist IBM Systems Lab Services Virtualization and Linux

2019 IBM Systems Technical University October 7 - 11 | Las Vegas Session: l109937 Tue 3:15-4:15 Veronese 2506-Level 2 LinuxONE Fast Start for Linux admins

Lectures here

Labs here

IBM Systems Technical University (c) IBM Corporation 2019 2 LinuxONE Fast Start for Linux admins – Badge requirements

To earn a badge for this track, you must do l109257 L4E: Open ecosystem exploits scalability and encryption on IBM z14 the following: l109420 L4E: IBM LinuxONE 101 1. Enroll in the LinuxONE Fast Start l109928 L4E: Options to consider when architecting a LinuxONE deployment (send email to tpearson@us.ibm.com) l109930 L4E: Pervasive encryption in a Linux on Z environment 2. Attend at least 9 of 12 lectures listed l109932 L4E: Dynamically provisioning resources with LPAR or z/VM here (name and signature on sign-in sheet l109933 L4E: Hardware Management Console tour for newbies or badge scan) l109934 L4E: What is unique about Linux on Z - A Linux admin perspective 3. Complete all required lab exercises. You l109935 L4E: Simplify deployment with IBM Dynamic Partition will be assigned to one of 3 lab time slots. Manager Lab instructors will record your name after l109936 L4E: Networking with KVM -- So many choices! you show them you have completed those l109937 L4E: Dynamically provisioning resources with KVM exercises. l110571 L4E: Lab: Working with Dynamic Partition Manager on LinuxONE z109804 L4E: IBM z/VM virtualization basics z110037 L4E: Monitoring and Managing z/VM for Linux Virtual Servers

IBM Systems Technical University (c) IBM Corporation 2019 3 Agenda

1 The Value of Dynamically Provisioning and Deprovisioning Resources

2 Channels and ignored devices

3 Dynamically Adjusting Disk Storage Resources

4 Dynamically Adjusting Networking Resources

5 Dynamically Adjusting Memory Resources

6 Dynamically Adjusting CPU Resources

7 Live migrating the workload to the available resource

2019 IBM Systems Technical University Agenda

1 The Value of Dynamically Provisioning and Deprovisioning Resources

2 Channels and ignored devices

3 Dynamically Adjusting Disk Storage Resources

4 Dynamically Adjusting Networking Resources

5 Dynamically Adjusting Memory Resources

6 Dynamically Adjusting CPU Resources

7 Live migrating the workload to the available resource

2019 IBM Systems Technical University Dynamic Resource Configuration

• Helps to avoid Linux guest restarts and potential outage/downtime resource allocation changes • Accommodate unplanned increases in application workload demands or application that consume more resources than expected resource • It can allow for more efficient overall hypervisor operation (reduced operational overhead) • Automated policy based reconfiguration is more responsive than manual adjustments. • May provide assistance with upgrades by provisioning lower levels of resources both before a virtual server is in production and after it is removed from production.

2019 IBM Systems Technical University Agenda

1 The Value of Dynamically Provisioning and Deprovisioning Resources

2 Channels and ignored devices

3 Dynamically Adjusting Disk Storage Resources

4 Dynamically Adjusting Networking Resources

5 Dynamically Adjusting Memory Resources

6 Dynamically Adjusting CPU Resources

7 Live migrating the workload to the available resource

2019 IBM Systems Technical University Dynamically Provisioning Resources - Channels

• All (non-PCI) IO devices are attached via a defined channel • In a native LPAR implementation you may need to change the channel (CHPID) state from Linux • Be aware that lscss does not display the CHPID state • Use chchp and lschp

2019 IBM Systems Technical University Ignored Devices

• This example is from a 1st level Linux in a LPAR • Ignore <> offline • A cio_ignore list was used on boot to restrict the available devices • This list can be dynamically modified to make new devices available • While a disk example is shown, cio_ignore applies to all IO devices • There is a cio_ignore command and you do not have to use cat and echo

2019 IBM Systems Technical University Ignore Devices

• There is a cio_ignore command and you do not have to use cat and echo • Most important for a first level Linux • -l lists • -u builds blacklist based on offline devices • -p purges from linux access until removed from blacklist • -r removes from blacklist

• Don’t leave other systems devices exposed

2019 IBM Systems Technical University Ignoring devices

• The cio_ignore list is shown on the kernel parameters line of the zipl.conf • Be sure to update it with newly (de)provisioned devices as you change the configuration of your system • Ignore list very important for first level Linux systems • Much less important for virtual servers running under a hypervisor

2019 IBM Systems Technical University Agenda

1 The Value of Dynamically Provisioning and Deprovisioning Resources

2 Channels and ignored devices

3 Dynamically Adjusting Disk Storage Resources

4 Dynamically Adjusting Networking Resources

5 Dynamically Adjusting Memory Resources

6 Dynamically Adjusting CPU Resources

7 Live migrating the workload to the available resource

2019 IBM Systems Technical University Dynamically Adding Disk Resources in a KVM environment

— Disk Storage Resource Types • ECKD o Full Volume o Partition o Images files • SCSI Luns o Full LUN device via FCP o Partial LUN device via FCP o Images files • Virtual CD/DVD • Network attached storage — All types can be dynamically added — General Process • Add resource from hypervisor • Make new resource available to guest domain • Bring virtual device online in virtual server • Provision as usual

2019 IBM Systems Technical University Virtual CDROM in Virtual Machine Manager (aka Virt Manager)

2019 IBM Systems Technical University 14 © Copyright IBM Corporation 2019 Dynamic addition thru Virt Manager

Attaching an ISO image to a guest with an existing virtual CDROM device works

Trying to dynamically add a CDROM device via Virtual Machine Manager fails

However there is another way

2019 IBM Systems Technical University 15 © Copyright IBM Corporation 2019 Before mounting ISO on guest device

2019 IBM Systems Technical University 16 © Copyright IBM Corporation 2019 Dynamically attach ISO to existing CDROM device – Virt Manager

If the device already exists in the guest domain it is a simple matter of browsing for the desired ISO and clicking apply

Whether the cdrom/iso is automatically mounted on a filesystem location is a function of guest

2019 IBM Systems Technical University 17 © Copyright IBM Corporation 2019 After adding ISO image and mounted on filesystem on guest

Before adding ISO image

After adding ISO image

2019 IBM Systems Technical University 18 © Copyright IBM Corporation 2019 Sample domain XML for virtual cdrom and iso image

• Example of resulting XML generated by Virtual Machine Manager

2019 IBM Systems Technical University 19 © Copyright IBM Corporation 2019 Adding device and attaching ISO directly via virsh

• Does not dynamically attach as an IDE, only SCSI • Defines device and attach the ISO

[root@lbskvm2 /]# virsh attach-disk guest1 /var/lib/libvirt/images/RHEL-ALT-7.5-20180308.0- Server-s390x-dvd1.iso sda --type cdrom Disk attached successfully

[root@lbskvm2 /]# ssh ryoung1@guest1 ryoung1@guest1's password: Last login: Mon Apr 15 12:06:48 2019 from gateway [ryoung1@guest1 ~]$ sudo su [sudo] password for ryoung1: [root@guest1 ryoung1]# blkid /dev/vda1: UUID="8e5020b1-1be0-4f04-a1cf-ff8a6f284ff8" TYPE="xfs" /dev/vda2: UUID="7446f77f-2bb8-48b2-b4a7-520d44ebce66" TYPE="swap" /dev/vda3: UUID="5169d08e-bbab-4875-9df3-ee236acc0a82" TYPE="xfs" /dev/vdb1: UUID="BoFb06-qEM6-PjAT-9RkQ-slY2-77Pk-3o8Rbw" TYPE="LVM2_member" /dev/sr0: UUID="2018-03-08-07-42-03-00" LABEL="RHEL-ALT-7.5 Server.s390x" TYPE="iso9660" [root@guest1 ryoung1]#

2019 IBM Systems Technical University 20 © Copyright IBM Corporation 2019 Fiber attached Virtual Disk Storage Resources

d0 d1 d2 d3 Guests

.img .img lvm lvm dm-0

mpathb dasda dasdb multipathd multipathd FCP FCP Ficon Ficon Ficon Ficon FCP FCP

ECKD ECKD SCSI

LUN ECKD LUN

ECKD Hypervisor Hypervisor Layer

# virsh domblklist sles12s1b • Storage can be dynamically attached and removed Target Source from running virtual servers ------vda /var/lib/libvirt/images/sles12sp1b.img

• Subdriver qcow2 must be specified # virsh attach-disk sles12s1b --source # qemu-img create -f qcow2 /var/lib/libvirt/images/sles12sp1b-disk2.qcow2 3G var/lib/libvirt/images/sles12sp1b-disk2.qcow2 --target vdd --driver Formatting '/var/lib/libvirt/images/sles12sp1b-disk2.qcow2', fmt=qcow2 qemu --subdriver qcow2 --targetbus virtio --persistent size=3221225472 encryption=off cluster_size=65536 lazy_refcounts=off Disk attached successfully refcount_bits=16 # virsh domblklist sles12s1b # qemu-img info /var/lib/libvirt/images/sles12sp1b-disk2.qcow2 Target Source image: /var/lib/libvirt/images/sles12sp1b-disk2.qcow2 ------file format: qcow2 vda /var/lib/libvirt/images/sles12sp1b.img virtual size: 3.0G (3221225472 bytes) vdd var/lib/libvirt/images/sles12sp1b-disk2.qcow2 disk size: 196K cluster_size: 65536 linux:~ # ls -la /dev/vd* Format specific information: brw-rw---- 1 root disk 253, 0 May 17 20:38 /dev/vda compat: 1.1 brw-rw---- 1 root disk 253, 1 May 17 20:02 /dev/vda1 lazy refcounts: false brw-rw---- 1 root disk 253, 2 May 17 20:02 /dev/vda2 refcount bits: 16 brw-rw---- 1 root disk 253, 16 May 17 20:55 /dev/vdb corrupt: false linux:~ # fdisk -l /dev/vdb

# ls -la /var/lib/libvirt/images/sles12sp1b-disk2.qcow2 Disk /dev/vdb: 3 GiB, 3221225472 bytes, 6291456 sectors -rw-r--r--. 1 root root 196656 May 17 20:53 /var/lib/libvirt/images/sles12sp1b- Units: sectors of 1 * 512 = 512 bytes disk2.qcow2 Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

2019 IBM Systems Technical University 22 Dynamically adding/remove disk to a guest – image file

• Removed just as easily as it was added • Sparsely populated 3GB qcow2 only used 196k initially

# virsh domblklist sles12s1b Target Source ------vda /var/lib/libvirt/images/sles12sp1b.img vdd var/lib/libvirt/images/sles12sp1b-disk2.qcow2

# virsh detach-disk sles12s1b --target vdd --persistent Disk detached successfully

# virsh domblklist sles12s1b Target Source ------vda /var/lib/libvirt/images/sles12sp1b.img

2019 IBM Systems Technical University 23 Adding a disk image to a running guest via Virt-Manager.

2019 IBM Systems Technical University 24 © Copyright IBM Corporation 2019 Adding a disk image to a running guest via Virt-Manager.

2019 IBM Systems Technical University 25 © Copyright IBM Corporation 2019 Dynamically adding/remove disk to a guest – block device

• Partitioned device can be used as easily as image files linux:~ # ls -la /dev/vd* brw-rw---- 1 root disk 253, 0 May 17 20:38 /dev/vda brw-rw---- 1 root disk 253, 1 May 17 20:02 /dev/vda1 • Other device types could be added/removed as well brw-rw---- 1 root disk 253, 2 May 17 20:02 /dev/vda2 brw-rw---- 1 root disk 253, 16 May 17 20:44 /dev/vdb

linux:~ # fdisk -l /dev/vdb

linux:~ # ls -la /dev/vd* Disk /dev/vdb: 6.9 GiB, 7385235456 bytes, 1803036 sectors brw-rw---- 1 root disk 253, 0 May 17 20:38 /dev/vda Units: sectors of 1 * 4096 = 4096 bytes brw-rw---- 1 root disk 253, 1 May 17 20:02 /dev/vda1 Sector size (logical/physical): 4096 bytes / 4096 bytes brw-rw---- 1 root disk 253, 2 May 17 20:02 /dev/vda2 I/O size (minimum/optimal): 4096 bytes / 4096 bytes linux:~ #

# virsh attach-disk sles12s1b --source /dev/dasdd1 --target vdc --persistent # virsh detach-disk sles12s1b --target vdc --persistent Disk attached successfully Disk detached successfully

# virsh domblklist sles12s1b # virsh domblklist sles12s1b Target Source Target Source ------vda /var/lib/libvirt/images/sles12sp1b.img vda /var/lib/libvirt/images/sles12sp1b.img vdc /dev/dasdd1

2019 IBM Systems Technical University 26 Dynamically adding a SCSI LUN in the KVM host Usually Required • chzdev can dynamically define a LUN and persist the activation of root@lbskvm3:/# chzdev -e zfcp-host 0.0.8b00 its zfcp device, the path to the FCP device 0.0.8b00 configured LUN, and the LUN itself Note: Automatic LUN scan disabled - LUNs must be configured manually • If lun autoscan is enabled, you root@lbskvm3:/# chzdev -e zfcp-host 0.0.8d00 only need to ensure the FCP FCP device 0.0.8d00 configured device is brought online. Note: Automatic LUN scan disabled - LUNs must be configured manually • All LUNs available down the path would be made available if lun autoscan is enabled Typically NOT Required – Unless you disable autoscan root@lbskvm3:/# lsluns | grep -E 'port|adapter|24007' Scanning for LUNs on adapter 0.0.8b00 at port 0x500507630718d122: 0x4042400700000000 Scanning for LUNs on adapter 0.0.8d00 at port 0x500507630713d122: 0x4042400700000000 root@lbskvm3:/# chzdev --enable zfcp-lun 0.0.8b00:0x500507630718d122:0x4042400700000000 zFCP LUN 0.0.8b00:0x500507630718d122:0x4042400700000000 configured root@lbskvm3:/# chzdev --enable zfcp-lun 0.0.8d00:0x500507630713d122:0x4042400700000000 zFCP LUN 0.0.8d00:0x500507630713d122:0x4042400700000000 configured

2019 IBM Systems Technical University Dynamically adding a SCSI LUN in the KVM host

• LUNs enabled to the KVM should be visible in the multipather • rescan-scsi-bus.sh

root@lbskvm3:/# multipath -l 36005076307ffd1220000000000004207 dm-4 IBM,2107900 size=20G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=0 status=active |- 0:0:0:1074217026 sda 8:0 active undef unknown `- 1:0:0:1074217026 sdb 8:16 active undef unknown root@lbskvm3:/#

• Provision as you normally would (Partition, pvcreate, vgextend/vgcreate, lvextend/lvcreate, mkfs)

2019 IBM Systems Technical University 28 © Copyright IBM Corporation 2019 Adding a block device to a running guest via Virt-Manager.

2019 IBM Systems Technical University 29 © Copyright IBM Corporation 2019 Added a block device to a running guest via Virt-Manager.

2019 IBM Systems Technical University 30 © Copyright IBM Corporation 2019 Agenda

1 The Value of Dynamically Provisioning and Deprovisioning Resources

2 Channels and ignored devices

3 Dynamically Adjusting Disk Storage Resources

4 Dynamically Adjusting Networking Resources

5 Dynamically Adjusting Memory Resources

6 Dynamically Adjusting CPU Resources

7 Live migrating the workload to the available resource

2019 IBM Systems Technical University Dynamically enable a NIC in DPM

• Select “New NIC” from partition details screen

2019 IBM Systems Technical University 32 Dynamically enable NIC in DPM

• Name the NIC • Select the adapter • Optionally provide a device number

2019 IBM Systems Technical University 33 Dynamically enable a NIC in DPM

• Before the “New NIC” is defined/saved we have devices 1-7

2019 IBM Systems Technical University 34 Dynamically enable a NIC in DPM

• The new “DATAOSA” is device number 8

2019 IBM Systems Technical University 35 Dynamically enable a NIC in DPM

• Save the change

2019 IBM Systems Technical University 36 Dynamically enable a NIC in KVM host

• Utilize znetconf to define the new “NIC”/OSA device • Devices 0.0.0008 thru 0.0.000a

2019 IBM Systems Technical University 37 Dynamically enable a NIC in KVM host

• Successfully defined another NIC/OSA dynamically

• It can now be defined to the Linux TCPIP stack or passed to Open Virtual Switch as you normally would.

2019 IBM Systems Technical University 38 Enable OSA NIC on KVM host

• chzdev can also set the QDIO OSA device online and set its attributes • It can create the udev entries so the definition is persistent • Alternative to znetconf method • chzdev support more than just network devices

# chzdev -e qeth 0.0.1000:0.0.1001:0.0.1002 layer2=1 QETH device 0.0.1000:0.0.1001:0.0.1002 configured # chzdev -e qeth 0.0.1100:0.0.1101:0.0.1102 layer2=1 QETH device 0.0.1100:0.0.1101:0.0.1102 configured # chzdev -e qeth 0.0.1200:0.0.1201:0.0.1202 layer2=1 QETH device 0.0.1200:0.0.1201:0.0.1202 configured # chzdev -e qeth 0.0.1300:0.0.1301:0.0.1302 layer2=1 QETH device 0.0.1300:0.0.1301:0.0.1302 configured #

2019 IBM Systems Technical University 39 © Copyright IBM Corporation 2019 Enable OSA NIC on KVM host # lszdev 1a0 -i DEVICE qeth 0.0.01a0:0.0.01a1:0.0.01a2 • Attributes can be set via chzdev Names : enc1a0 Network interfaces : enc1a0 • Many attributes are available to be set Resources provided : IPv6 address fe80::783f:9ff:febf:3667/64 IPv4 address 9.12.22.232/24 Modules : qeth Online : yes # chzdev --by-interface enc1a0 buffer_count=128 Exists : yes QETH device 0.0.01a0:0.0.01a1:0.0.01a2 configure failed Persistent : yes Error: Cannot set buffer_count='128' while online='1' (*) Note: You can use --force to override safety checks (*) ATTRIBUTE ACTIVE PERSISTENT # bridge_hostnotify "0" - # bridge_reflect_promisc "none" - # chzdev -p --by-interface enc1a0 buffer_count=128 bridge_role "none" - Configuring devices in the persistent configuration only buffer_count "64" "64" QETH device 0.0.01a0:0.0.01a1:0.0.01a2 configured hw_trap "disarm" - # isolation "none" - layer2 "1" "1" online "1" "1" performance_stats "0" - • Some attributes must be set while the OSA is offline portname "" - • chzdev –p, will set the option boot/activation and not portno "0" - priority_queueing "always queue 2" - attempt a live change vnicc/bridge_invisible "n/a" - vnicc/flooding "n/a" - vnicc/learning "n/a" - vnicc/learning_timeout "n/a" - vnicc/mcast_flooding "n/a" - vnicc/rx_bcast "n/a" - vnicc/takeover_learning "n/a" - vnicc/takeover_setvmac "n/a" -

2019 IBM Systems Technical University 40 © Copyright IBM Corporation 2019 Dynamically add Open Virtual Switch to KVM host root@lbskvm3:~# virsh net-list root@lbskvm3:~# ovs-vsctl add-br ovsbr0 Name State Autostart Persistent root@lbskvm3:~# ovs-vsctl set bridge ovsbr0 stp_enable=true ------root@lbskvm3:~# ovs-vsctl show default active yes yes dd702bc3-707d-4e3c-8743-05f8425a96f9 Bridge "ovsbr0" root@lbskvm3:~# virsh net-define vlans.xml Port "ovsbr0" Network vlans defined from vlans.xml Interface "ovsbr0" root@lbskvm3:~# virsh net-list type: internal Name State Autostart Persistent ovs_version: "2.11.0" root@lbskvm3:~# cat vlans.xml ------ default active yes yes vlans root@lbskvm3:~# virsh net-start vlans Network vlans started root@lbskvm3:~# virsh net-autostart vlans Network vlans marked as autostarted root@lbskvm3:~# virsh net-list Name State Autostart Persistent ------ default active yes yes vlans active yes yes root@lbskvm3:~#

2019 IBM Systems Technical University 41 © Copyright IBM Corporation 2019 Open vSwitch based “vlans” network available in Virt-Manager

• “vlans” network available to be managed from Virt-Manager • Stop • Start • Delete • Manage autostart setting • Assign to guests

2019 IBM Systems Technical University 42 © Copyright IBM Corporation 2019 Dynamically adding an Open vSwitch network to a running guest

• Initially the eth0 is the only external network

• lxcbr0, docker0, and lo are all internally only networks

• We will add an Open vSwitch connection as eth1

2019 IBM Systems Technical University 43 © Copyright IBM Corporation 2019 Dynamically adding an Open vSwitch network to a running guest

• Initially only one NIC show • Click “Add Hardware” • Select our recently added “vlans” network as the network source

2019 IBM Systems Technical University 44 © Copyright IBM Corporation 2019 Dynamically adding an Open vSwitch network to a running guest

• Optionally select one of the VLANs we defined previously to use for this guest • Click Finish!

2019 IBM Systems Technical University 45 © Copyright IBM Corporation 2019 Dynamically adding an Open vSwitch network to a running guest

• Now has two NICs • The new network interface is shown • Link state is active

• Note assigned MAC

2019 IBM Systems Technical University 46 © Copyright IBM Corporation 2019 Dynamically adding an Open vSwitch network to a running guest

• Links eth0 and eth1 now available

• Device is ready to be configured within the guest Linux server

2019 IBM Systems Technical University 47 © Copyright IBM Corporation 2019 KVM Guest/Domain interface information # virsh domifstat sles12s1b vnet3 vnet3 rx_bytes 197152 vnet3 rx_packets 3760 • Two interfaces defined and up vnet3 rx_errs 0 vnet3 rx_drop 0 • Only 1 interface has an address enabled on it. vnet3 tx_bytes 2848 vnet3 tx_packets 22 vnet3 tx_errs 0 vnet3 tx_drop 0

# virsh domif-getlink sles12s1b vnet4 # virsh domifstat vnet4 up sles12s1b vnet4 # virsh domif-getlink sles12s1b vnet3 vnet4 rx_bytes 0 vnet3 up vnet4 rx_packets 0 # virsh domiflist sles12s1b vnet4 rx_errs 0 Interface Type Source Model MAC vnet4 rx_drop 0 ------vnet4 tx_bytes 0 vnet3 network default virtio 52:54:00:7a:08:78 vnet4 tx_packets 0 vnet4 bridge vlans virtio 52:54:00:d4:c2:10 vnet4 tx_errs 0 vnet4 tx_drop 0 # virsh domifaddr sles12s1b Name MAC address Protocol Address ------vnet3 52:54:00:7a:08:78 ipv4 192.168.122.190/24

2019 IBM Systems Technical University 48 Guest/Domain interface dynamic detach/attach – CLI example

• Remove one of the two # virsh detach-interface sles12s1b bridge --mac 52:54:00:d4:c2:10 - -live existing interfaces Interface detached successfully

# virsh domiflist sles12s1b Interface Type Source Model MAC • Attach dynamically a virtio ------interface from network vnet3 network default virtio 52:54:00:7a:08:78 “vlans” # virsh net-list Name State Autostart Persistent ------default active yes yes vlans active yes yes

# virsh attach-interface sles12s1b --type network --model virtio - -source vlans --target vnet5 --live Interface attached successfully

# virsh domiflist sles12s1b Interface Type Source Model MAC ------vnet3 network default virtio 52:54:00:7a:08:78 vnet4 bridge vlans virtio 52:54:00:1b:0c:f2

2019 IBM Systems Technical University 49 Dynamically adding a macvtap device to a KVM guest

• You can dynamically add macvtap interfaces also. • This server has an existing NAT / default network interface • Click “Add Hardware” • Select existing host interface to perform macvtap with as the source

2019 IBM Systems Technical University 50 © Copyright IBM Corporation 2019 Dynamically adding a macvtap device to a KVM guest

• Select the desired “mode” for the macvtap interface to operate in. • Click Finish

2019 IBM Systems Technical University 51 © Copyright IBM Corporation 2019 Dynamically adding a macvtap device to a KVM guest

2019 IBM Systems Technical University 52 © Copyright IBM Corporation 2019 Agenda

1 The Value of Dynamically Provisioning and Deprovisioning Resources

2 Channels and ignored devices

3 Dynamically Adjusting Disk Storage Resources

4 Dynamically Adjusting Networking Resources

5 Dynamically Adjusting Memory Resources

6 Dynamically Adjusting CPU Resources

7 Live migrating the workload to the available resource

2019 IBM Systems Technical University Dynamically enabling memory in the Partition and KVM Host

• 1 TB online and 2 TB offline initially • 8 GB memory increment size

2019 IBM Systems Technical University 54 Dynamically enabling memory in the Partition and KVM host

• Memory can not be brought online by just using chmem

• Must go back to the HMC to dynamically enable memory before issuing chmem

• Must be a multiple of 8GB

2019 IBM Systems Technical University 55 Dynamically enabling memory in the Partition with DPM

• Increase the “Memory” allocation from 1 TB to 2 TB

• Apply the change and go back to KVM

2019 IBM Systems Technical University 56 Dynamically enabling memory in the Partition with DPM

• When you save the memory allocation change you will receive a confirmation

2019 IBM Systems Technical University 57 Dynamically enabling memory in KVM host

• After the HMC memory allocation is increased, • chmem can now be used to increase the allocation within KVM

• Large amounts of memory could take a brief period to bring online (Terabytes). “Patience you must have.”

2019 IBM Systems Technical University 58 Dynamically Provisioning Memory Resources – Image profile, standard PRSM mode

▪ Defining “Reserved” storage to the LPAR will allow you to dynamically add memory to a running Linux server running natively in a partition

2019 IBM Systems Technical University Add/Remove memory from KVM guest

Guest Domain XML memory configuration

root@ubuntuController:~# virsh dumpxml sles12s1b sles12s1b 1bc58ea5-fc01-4bc7-a873-d9f629fee4f3 Guest-System Suse Sles12 3048576 1524288 1 1 hvm

2019 IBM Systems Technical University 60 Add/Remove memory from KVM guest

• Guest shows ~ 1.5 GB initially • Free and meminfo only show the initial 1.5 GB and not the 3GB maximum • KVM uses a memory ballooning technology to grow and shrink the memory size of a Linux server • It is NOT the same as the CMM memory ballooning you may be familiar with.

2019 IBM Systems Technical University 61 Add/Remove memory from KVM guest

• “lsmem” only shows the maximum memory size and NOT the initial size

• Picture shows before and after (initial and maximum memory allocations)

• Effects of memory ballooning not shown here

2019 IBM Systems Technical University 62 Add/Remove memory from KVM guest

• Memory increased to ~ 3GB • Show via dominfo

2019 IBM Systems Technical University 63 Add/Remove memory from KVM guest

• Guest shows memory increased to ~ 3GB

2019 IBM Systems Technical University 64 Memory via Virt-Manager

• Memory can be set and dynamically adjusted via virt- manager • Can also make not dynamic changes, such as increasing the maximum memory value for next boot

2019 IBM Systems Technical University 65 © Copyright IBM Corporation 2019 Add/Remove memory from KVM guest

• Removal of memory the same as the addition • virsh setmem command • Memory needs to be “available” to be released.

2019 IBM Systems Technical University 66 Dynamically Provisioning – Huge Pages

• 1MB and 2GB huge pages are possible • Huge pages can be added permanently via hugepages= in the kernel parameter line of zipl.conf • Huge page information can be queried via /proc/meminfo HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 1024 kB • Also queried via /proc/sys/vm/nr_hugepages • Can be set dynamically via echo xxx > /proc/sys/vm/nr_hugepages • Hotplug memory allocated as moveable and can only be used by movable resources. • By default Large Pages are not allocated as movable resource but can be made to allocate from movable hotplug memory with: • # echo 1 > /proc/sys/vm/hugepages_treat_as_movable • Hotplug memory allocated to large pages can not be set offline until all large pages are released • For more information see Documentation/vm/hugetlbpage.txt • Middleware exploiters may require configuration also to exploit (For example Java or Database servers)

2019 IBM Systems Technical University Dynamically Provisioning Huge Pages

• Don’t forget to make dynamic changes permanent in zipl.conf kernel parameter

• Allocate your large pages as soon as possible to avoid fragmentation issues

2019 IBM Systems Technical University Agenda

1 The Value of Dynamically Provisioning and Deprovisioning Resources

2 Channels and ignored devices

3 Dynamically Adjusting Disk Storage Resources

4 Dynamically Adjusting Networking Resources

5 Dynamically Adjusting Memory Resources

6 Dynamically Adjusting CPU Resources

7 Live migrating the workload to the available resource

2019 IBM Systems Technical University Dynamically enabling CPUs in the Partition and the KVM host

Current State

• 2 threads per core • 8 CPUs (threads) online

• Lets add some more!

2019 IBM Systems Technical University 70 Dynamically enabling CPUs in the Partition

• From “Partition Details” increase the “Processors” allocation

2019 IBM Systems Technical University 71 Dynamically enabling CPUs in the Partition

• The new allocation is 6 processors

2019 IBM Systems Technical University 72 Dynamically enable CPUs in the Partition

• Save the change

2019 IBM Systems Technical University 73 Dynamically enable CPUs in the KVM Host

Before

Configured

Enabled

After

2019 IBM Systems Technical University 74 Dynamically enable CPUs in the KVM host

• 12 threads • 6 IFLs

• Capacity added dynamically

2019 IBM Systems Technical University 75 Dynamic resource management

• Dynamically add/remove CPUs from host • QEMU/KVM : cpu_shares, vcpu_period, vcpu_quota, emulator_period, emulator_quota, • Dynamically add/disable CPUs from guest iothread_quota, iothread_period • Dynamically add/remove memory from host • setvcpus domain count [--maximum] [[--config] [-- • Dynamically add/remove memory from guest live] | [--current]] • [--guest] [--hotpluggable] • Prioritize guest CPU • Change the number of virtual CPUs • CPU dispatching mode (polarization) Vertical vs active in a guest domain. By default, this Horizontal on first level Linux command works on active guest domains. To change the settings for an inactive guest domain, use the --config flag.

• Suspend / Resume • Migrate (live / dead)

2019 IBM Systems Technical University 76 Add / remove CPUs from KVM host

• Initial and standby/reserved CPUs defined at the HMC • Initially only 2 of 4 CPUs online • Dispatching mode is horizontal

2019 IBM Systems Technical University 77 Add/remove CPUs from KVM host

• “chcpu” can be used to enable additional CPUs just like any other Linux • 4 CPUs now online • Core, socket, book topology displayed • New CPUs follow existing polarization mode when enabled.

2019 IBM Systems Technical University 78 Adjust CPU dispatching mode on KVM host

• As KVM is running first level (in an LPAR) the polarization mode can also be changed dynamically

2019 IBM Systems Technical University 79 Add/remove CPU from KVM guest/domain

• Virtual CPUs for guests are defined in the domain xml

• The current domain information including the CPU configuration can be queried with virsh dominfo from the KVM host Only beginning of domain xml shown for readability

2019 IBM Systems Technical University 80 Add/remove CPUs from KVM guest/domain

• Domain xml modified for up to 4 virtual CPUs from domain • Only 1 CPU will be activate initially • virsh- vcpuinfo and vcpucount display the cpu configuration for the domain

2019 IBM Systems Technical University 81 Add/remove CPUs from KVM guest/domain

Active CPUs set to 4 - Guest and Host view

KVM Host KVM Guest

2019 IBM Systems Technical University 82 Add/remove CPUs from KVM guest/domain

• The guest can take CPUs offline, but they can NOT be deconfigured.

• Attempts to deconfigure result in an error message

2019 IBM Systems Technical University 83 Add/remove CPUs from KVM guest/domain

• CPUs can be taken offline from the KVM host

• virsh setvcpus --live

• QEMU guest agent is required to be installed and running in guest

2019 IBM Systems Technical University 84 Qemu Guest Agent

linux:~ # rpm -ivh qemu-guest-agent-2.3.1-32.11.s390x.rpm • Guest agent Preparing... ################################# [100%] installation and rpm Updating / installing... contents 1:qemu-guest-agent-2.3.1-32.11 ################################# [100%] • Systemd status of linux:~ # running agent in linux:~ # rpm -ql qemu-guest-agent guest /usr/bin/qemu-ga /usr/lib/systemd/system/qemu-ga.service /usr/lib/udev/rules.d/80-qemu-ga.rules linux:~ #

2019 IBM Systems Technical University QEMU Guest Agent

• Serial connection to guest agent must be defined in the domain XML • The QEMU guest agent exists to do more than just offline vcpus • Other virsh commands use it (guestvcpus for example) • qemu-agent-command - QEMU Guest Agent Command

2019 IBM Systems Technical University QEMU Guest Agent

• Query network interface information

2019 IBM Systems Technical University Remove CPUs from KVM guest/domain with QEMU agent

• Live vcpu count reduced

• Virsh vcpucount shows no change

• Virsh guestvcpus show the reduced cpus (from guest perspective)

• CPUs are brought offline in guest but NOT deconfigured.

• Can adjust current CPU allocation dynamically up to the maximum values • Can adjust maximum value for next restart of the guest/domain

KVM host has 4 CPUs, each virtual server has 4 CPUs

• shares The optional shares element specifies the proportional weighted share for the domain. If this is omitted, it defaults to the OS provided defaults. NB, There is no unit for the value, it's a relative measure based on the setting of other VM, e.g. A VM configured with value 2048 will get twice as much CPU time as a VM configured with value 1024.

• Not available in Virt-Manager

• Default “share” value is 1024 for all domains

• Changes to the share values can be made dynamically and/or persistently

• By changing guest2’s share, it now gets 4/5ths (80%) of the priority • The remaining server (guest1) gets its 1/5th (20%) of the CPU resource

• Live change of the cpu_share setting is made permament with –config • Alternatively a virsh edit could also be used

• Here you can see the cputune / share value of 4096 set in the domain xml to persist the setting across reboots

• This can not be set via Virt- Manager

period Some additional CPU related controls • The optional period element specifies the • global_period enforcement interval (unit: microseconds). Within period, each vCPU of the domain will not be allowed to • global_quota consume more than quota worth of runtime. The value • emulator_period should be in range [1000, 1000000]. A period with value 0 means no value. Only QEMU driver support • emulator_quota since 0.9.4, LXC since 0.9.10 • iothread_period quota • iothread_quota • The optional quota element specifies the maximum allowed bandwidth (unit: microseconds). A domain with quota as any negative value indicates that the domain has infinite bandwidth for vCPU threads, which means that it is not bandwidth controlled. The value should be in range [1000, 18446744073709551] or less than 0. A quota with value 0 means no value. You can use this feature to ensure that all vCPUs run at the same speed. Only QEMU driver support since 0.9.4, LXC since 0.9.10

• Still 4 CPUs in the host and 4 CPUs in the guest, but only a single guest • Initial configuration shown below with default period/quota values • Guest workload driving to 100% usage of its virtual CPU allocation

• Same guest with same looping application, but only using the 1% quota that was assigned

1 The Value of Dynamically Provisioning and Deprovisioning Resources

2 Channels and ignored devices

3 Dynamically Adjusting Disk Storage Resources

4 Dynamically Adjusting Networking Resources

5 Dynamically Adjusting Memory Resources

6 Dynamically Adjusting CPU Resources

7 Live migrating the workload to the available resource

2019 IBM Systems Technical University Virtual Server live migration

• Target system must have same or equivalent access to resources as the source system • Ensure you use device naming that identifies the same resource from source and target • If using Images files (ie qcow2), they need to be hosted on sharable filesystem storage (NFS, GFPS, GFS) • Cannot share macvtap OSA between source and target LPARs on the same CEC/CPC. • firewall-cmd --zone=public --add-port=49152-49215/tcp –permanent • Disk cache mode none needed kvm1# virsh migrate --live --verbose --tunnelled --p2p s12raw-1 qemu+ssh://lbskvm2/system • Not CTCs required Migration: [100 %] • Uses TCP networking kvm1# virsh list Id Name State ❑ ISM ------❑ RoCE kvm1# virsh list --all ❑ HiperSocket Id Name State ❑ OSA ------s12raw-1 shut off • No predefined host cluster required • Domain dynamically added to target • Works with ECKD or LUNs kvm2# virsh migrate --live --verbose --tunnelled --p2p s12raw-1 qemu+ssh://lbskvm1/system Migration: [100 %] kvm2#

• You need to specify the target/lowest # virsh domcapabilities machine model you will migrate to in [...] the domain xml z800-base z890.2-base • Use virsh domcapabilities to list the z9EC.2 available models z13.2 z990.5-base • Failure to plan for this likely results in a z9BC-base crashed virtual server on the target z890.2 z890 domain z9BC z13 z196 z13s z990.3 z13s-base z9EC z14ZR1-base z900.3-base z13.2-base z196.2-base zBC12-base

➢ Two phases: Live and Stopped phases ➢ If using ssh, key based authentication expected ➢ migrate-setmaxdowntime ➢ migrate-setspeed ➢ Sampling of the “migration” options: --live / --offline --undefine-source Undefine domain on source system --timeout xxx Timeout for autoconverge, suspends server after timeout to ensure migration --auto-converge Throttle down CPU for domain, so est. downtime is less than max specified --copy-storage-all Can copy storage when “unshared” storage is being used --unsafe If coherent clustered filesystem is used, and cache mode is not none --p2p libvirt connection from source to destination host and controls the migration Without the --p2p option, virsh handles the communication/control --tunneled libvirt migration data through the libvirtd communication socket. No extra ports are required to be opened on the destination Tunneled migration has a small performance impact Data is copied between the libvirt daemon and the QEMU

2019 IBM Systems Technical University 101 Migrating a running virtual server – in progress

2019 IBM Systems Technical University 102 Migrating a running virtual server - completed

2019 IBM Systems Technical University 103 Live migration statistics available

$ virsh migrate –live --auto-converge --auto-converge-initial 30 • You can monitor the progress / details of --auto-converge-increment 15 --p2p blockc1 qemu+ssh://lbskvm1/system a migration request

• Transfer rate details root@lbskvm3:~# virsh domjobinfo blockc1 Job type: Unbounded • Elapsed/remaining time Operation: Outgoing migration Time elapsed: 2070 ms • Dirty rate root@lbskvm3:~# virsh domjobinfo blockc1 Job type: Unbounded • Iterations Operation: Outgoing migration Time elapsed: 2656 ms Data processed: 33.681 MiB Data remaining: 1.113 GiB Data total: 1.953 GiB Memory processed: 33.681 MiB Memory remaining: 1.113 GiB • Running migration can be converted Memory total: 1.953 GiB from precopy to postcopy. Memory bandwidth: 7.379 MiB/s Dirty rate: 0 pages/s Page size: 4096 bytes Iteration: 1 Postcopy requests: 0 Constant pages: 212218 Normal pages: 8140 Normal data: 31.797 MiB Expected downtime: 300 ms Setup time: 31 ms

➢ Linux on System z Device Drivers, Features, and Commands SC33-8411 http://www.ibm.com/developerworks/linux/linux390/documentation_dev.html ➢ KVM Virtual Server Management SC34-2752-04 https://www.ibm.com/support/knowledgecenter/en/linuxonibm/liaaf/lnz_r_va.html

https://libvirt.org/formatdomain.html

2019 IBM Systems Technical University Thank you!

Richard Young Executive I.T. Specialist

[email protected] +1-262-893-8662

Please complete the Session Evaluation!

— © 2019 International Business Machines Corporation. No part of — Performance data contained herein was generally obtained in a this document may be reproduced or transmitted in any form controlled, isolated environments. Customer examples are without written permission from IBM. presented as illustrations of how those — U.S. Government Users Restricted Rights — use, duplication or — customers have used IBM products and the results they may have disclosure restricted by GSA ADP Schedule Contract with IBM. achieved. Actual performance, cost, savings or other results in other operating environments may vary. — Information in these presentations (including information relating to products that have not yet been announced by IBM) — References in this document to IBM products, programs, or has been reviewed for accuracy as of the date of services does not imply that IBM intends to make such products, initial publication and could include unintentional technical or programs or services available in all countries in which typographical errors. IBM shall have no responsibility to update IBM operates or does business. this information. This document is distributed “as is” without any warranty, either express or implied. In no event, shall IBM — Workshops, sessions and associated materials may have been be liable for any damage arising from the use of this prepared by independent session speakers, and do not necessarily information, including but not limited to, loss of data, business reflect the views of IBM. All materials and discussions are provided interruption, loss of profit or loss of opportunity. for informational purposes only, and are neither intended to, nor IBM products and services are warranted per the terms and shall constitute legal or other guidance or advice to any individual conditions of the agreements under which they are provided. participant or their specific situation. — IBM products are manufactured from new parts or new and used — It is the customer’s responsibility to insure its own compliance parts. with legal requirements and to obtain advice of competent legal In some cases, a product may not be new and may have been counsel as to the identification and interpretation of any previously installed. Regardless, our warranty terms apply.” relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to — Any statements regarding IBM's future direction, intent or take to comply with such laws. IBM does not provide legal advice product plans are subject to change or withdrawal without or represent or warrant that its services or products will ensure that notice. the customer follows any law.

— Information concerning non-IBM products was obtained from the suppliers — IBM, the IBM logo, ibm.com and [names of other referenced of those products, their published announcements or other publicly IBM products and services used in the presentation] are available sources. IBM has not tested those products about this publication trademarks of International Business Machines Corporation, and cannot confirm the accuracy of performance, compatibility or any other registered in many jurisdictions worldwide. Other product and claims related to non-IBM products. Questions on the capabilities of non- service names might be trademarks of IBM or other IBM products should be addressed to the suppliers of those products. companies. A current list of IBM trademarks is available on IBM does not warrant the quality of any third-party products, or the ability of the Web at "Copyright and trademark information" at: any such third-party products to interoperate with IBM’s products. IBM www.ibm.com/legal/copytrade.shtml expressly disclaims all warranties, expressed or implied, including but not limited to, the implied warranties of merchantability and fitness for a purpose. — The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right.