TurtlesTurtles allall thethe WayWay DownDown

Image ©2013 CC-BY-NC-SA by Tony Gray Turtles all the Way Down

Misadventures with and Thin

Steven Ellis Senior Solution Architect – NZ [email protected] / @StevensHat

Agenda

● SSDs + Trim + Thin ● Redundant Turtles

● Turtle ● KVM Serial Console

● Turtles

● Virtual Turtles

● Migrating Turtles

● Nesting Turtles

● Cows

SSD + Trim / Unmap / Discard

● Critical to effective use of SSDs

● Lack of Trim can impact

– Lifetime of disk

● Blocks aren’t released / re-allocated correctly

– Performance

● OS + File-system need to support Trim

– Delete requests are passed to SSD

Filesystem Turtles? LVM Partition Virtual Disk KVM Filesystem thinpool

Filesystem LVM LVM Filesystem LVM Encryption Encryption Partition Partition Partition Partition

SSD SSD SSD SSD

File-system Discard

Filesystem Partition

SSD

File-system Discard

● XFS / EXT4 / BRTFS / JFS

– fstrim

– periodic discard

● /etc/fstab options discard

● SWAP understands discard Filesystem ● F2FS for Android / IoT / Embedded Partition ● /etc/fstab options discard SSD ● Confirm working

– sudo lsblk -o MOUNTPOINT,DISC-MAX,FSTYPE

LVM Passthru

● Edit /etc/lvm/lvm.conf

– issue_discards = 1

● Filesystem Previous file-system tips apply LVM Partition

SSD

dmcrypt / Luks

● Edit /etc/crypttab

– luks-blah-blah-blah UUID=blah- blah-blah none luks,discard

● Recreate boot disks Filesystem

– –force LVM Encryption

● Reference Partition – http://blog.christophersmart.com/201 SSD 3/06/05/trim-on-lvm-on-luks-on-ssd

Thin LVM

Creating a thin pool on VG myVG Virtual Disk [root@t440s ~]# lvcreate -T -L 80G myVG/lv_thin Logical volume "lv_thin" created. thinpool LVM New VM Disk Image Encryption [root@t440s ~]# lvcreate -V40G -T myVG/lv_thin -n New_VM Partition Logical volume "New_VM" created. SSD

Trim + libvirt

Check your machine type > 2.1 hvm KVM

Add discard to your hard disks LVM Encryption Partition Add scsi controller model virtio-scsi SSD

Filesystem Trim in a VM LVM Partition Treat like a physical environment Virtual Disk

Same rules apply for KVM

● fstrim vs discard thinpool LVM ● lvm.conf Encryption ● Luks Partition

SSD

Encryption + Filesystem LVM Trim in a VM Partition Don’t.. Luks Virtual Disk

Just Don’t.. KVM thinpool Please please... LVM Encryption DO NOT DO IT. Partition

SSD

Filesystem Are you Trim? LVM Partition Boot and test your VM Virtual Disk [root@testvm ~]# lsblk -o MOUNTPOINT,DISC-MAX,FSTYPE MOUNTPOINT DISC-MAX FSTYPE KVM /boot 1G xfs thinpool [SWAP] 1G swap / 1G xfs LVM Run fstrim in your VM Encryption [root@testvm ~]# fstrim -v / Partition /: 10.4 GiB (11197124608 bytes) trimmed SSD

Filesystem Fat -> Thin LVM Partition Virtual Disk Image Thin Virtual Disk KVM thinpool LVM Encryption Partition

SSD

Physical Host

Fat -> Thin Filesystem LVM kpartx + fstrim Partition Virtual Disk Image Thin Virtual Disk KVM Painfully via kpartx + fstrim thinpool # kpartx -a /dev/myVG/myVMDisk LVM # cd /mnt; mkdir volume # mount /dev/mapper/myVG-myVMDisk_BASE1 /mnt/volume/ Encryption # fstrim -v volume Partition

boot: 356.5 MiB (373850112 bytes) trimmed SSD # umount volume/ Physical Host

Fat -> Thin Filesystem LVM kpartx + fstrim Partition Virtual Disk Image Thin Virtual Disk KVM Repeat for other filesystems thinpool mount /dev/guestVG/root /mnt/volume/ LVM # fstrim -v ./volume Encryption ./volume: 9.8 GiB (10481520640 bytes) trimmed # umount volume Partition SSD

Physical Host

Fat -> Thin Filesystem LVM kpartx + fstrim Partition Virtual Disk Image Thin Virtual Disk KVM Clean up VGs and kpartx /dev/mapper entries thinpool # vgchange -a n guestVG LVM 0 logical volume(s) in volume group "guestVG" now active Encryption # kpartx -d /dev/mapper/myVG-myVMDisk_BASE Partition

SSD

Physical Host

Fat -> Thin Filesystem LVM virt-sparsify Partition Virtual Disk Image Thin Virtual Disk KVM Existing FAT LVM thinpool # dd if=myVMDisk.img of=/dev/myVG/myVMDisk LVM # virt-sparsify --in-place /dev/myVG/myVMDisk Encryption

Fat image ->Thin LVM Partition # virt-sparsify myVMDisk.img /dev/myVG/myVMDisk SSD # lvs /dev/myVG/myVMDisk Physical Host

Filesystem Filesystem LVM Thin -> Thin LVM Partition Partition Thin Virtual Disk Thin Virtual Disk KVM ● Painful KVM thinpool ● Sparse thinpool

LVM ● With Compression LVM (Encryption) (Encryption) Partition Partition

SSD SSD

Host A Host B

Filesystem Filesystem LVM Thin -> Thin LVM Partition Partition Thin Virtual Disk Thin Virtual Disk KVM ● Painful KVM thinpool – dd via ssh or nc thinpool LVM ● Results in a Fat LV LVM (Encryption) – Use kpartx/fstrim or virt-sparsify at (Encryption) destination Partition Partition

SSD SSD

Host A Host B

Filesystem Filesystem LVM Thin -> Thin LVM Partition Partition Thin Virtual Disk Thin Virtual Disk KVM Sparse KVM thinpool [root@hostb]# nc -l 11900 | \ thinpool dd of=/dev/HostbVG/guestVG bs=16M conv=sparse LVM LVM

(Encryption) [root@hosta]# dd if=/dev/HostaVG/guestVG \ (Encryption) Partition bs=16M | nc hostb 11900 Partition

SSD SSD [root@hostb]# virt-sparsify –in-place \ Host A /dev/HostbVG/guestVG Host B

Filesystem Filesystem LVM Thin -> Thin LVM Partition Partition Thin Virtual Disk Thin Virtual Disk KVM Sparse + lzop KVM thinpool [root@hostb]# nc -l 11900 | lzop -d -c - \ thinpool dd of=/dev/HostbVG/guestVG bs=16M conv=sparse LVM LVM

(Encryption) [root@hosta]# dd if=/dev/HostaVG/guestVG \ (Encryption) Partition bs=16M | lzop --fast - | nc hostb 11900 Partition

SSD SSD [root@hostb]# virt-sparsify –in-place \ Host B /dev/HostbVG/guestVG Host B

Nesting Turtles?

CC by NC Christa Burns https://www.flickr.com/photos/christajoy42/5562852057/ Filesystem Nesting KVM LVM /etc/modprobe.d/kvm-intel.conf Partition Virtual Disk options kvm-intel nested=1 Virtual Disk KVM options kvm-intel enable_shadow_vmcs=1 Nested KVM thinpool options kvm-intel enable_apicv=1 (thinpool) options kvm-intel ept=1 LVM LVM OR (Encryption) (Encryption) Partition /etc/modprobe.d/kvm-amd.conf Partition options kvm-amd nested=1 SSD Virtual Disk

Physical Host Virtual Host

But I like (q)Cows?

Filesystem Trim a qcow2 VM LVM Partition Same rules as we covered KVM and LVM qcow2 image ● KVM KVM – Machine type / unmap / virtio-scsi (thinpool)

● Inside the VM LVM – fstrim vs discard (Encryption) – lvm.conf Partition

● virt-sparsify SSD – virt-sparsify disk.raw --convert qcow2 disk.qcow2

Raid and Trim?

Two key software technologies

● mdadm

● lvm raid Filesystem LVM Filesystem md0 LVM raid1

mdadm raid1 partition partition

SSD SSD SSD SSD

mdadm raid1

Linux software raid (mdadm)

● Discard is passed-thru correctly

– Subject to your kernel version / distro Filesystem ● Initialisation may write to all blocks LVM – Potentially impacting performance md0 mdadm raid1

SSD SSD

LVM raid1

LVM based raid 1

● Discard is passed-thru correctly

– Less initialisation overhead compared with mdadm

Filesystem LVM raid1 partition partition

SSD SSD

KVM Tips

1) Trim / discard with libvirt + KVM

2) Using virt-sparsify

3) Nesting with KVM

4) Enable KVM serial Console

KVM Tips

1) Trim / discard with libvirt + KVM

2) Using virt-sparsify

3) Nesting with KVM

4) Enable KVM serial Console (including Grub)

KVM serial console

● Need to modify /etc/default/grub

– GRUB_TERMINAL="serial console”

– GRUB_SERIAL_COMMAND="serial --speed=38400 --unit=0 --word=8 --parity=no –stop=1"

– GRUB_CMDLINE_LINUX

● remove “rhgb quiet”

● Add “console=tty0 console=ttyS0,38400n8”

● Rebuild grub2 config

– grub2-mkconfig -o /boot/grub2/grub.cfg

KVM serial console + Ansible

● Example playbook

– https://github.com/steven-ellis/ansible-playpen

● grub_console.yaml

– Currently RHEL/Centos/Fedora centric

● Pull requests welcomed

● Demo

Questions?

http://people.redhat.com/sellis Open?