Qubes OS Many things Many things related to QubesOS Author: Neowutran Contents 1 Wiping VM 2 1.1 Low level storage technologies .................. 2 1.1.1 Must read ......................... 2 1.1.2 TL;DR of my understanding of the issue ........ 2 1.1.3 Things that could by implemented by QubesOS .... 2 2 Create a Gaming HVM 2 2.1 References ............................. 2 2.2 Prerequise ............................. 3 2.3 Hardware ............................. 3 2.4 Checklist .............................. 4 2.5 IOMMU Group .......................... 4 2.6 GRUB modification ........................ 4 2.7 Patching stubdom-linux-rootfs.gz ................ 5 2.8 Pass the GPU ........................... 6 2.9 Conclusion ............................. 6 2.10 Bugs ................................ 6 3 Create a Linux Gaming HVM, integrated with QubesOS 7 3.1 Goals ................................ 7 3.2 Hardware used .......................... 7 3.3 Main steps summary ....................... 7 3.3.1 Detailled steps ...................... 8 3.3.2 Using a kernel provided by debian ............ 8 3.4 Xorg ................................ 8 3.4.1 Pulseaudio ......................... 11 3.5 Final notes ............................ 11 3.6 References ............................. 12 4 Nitrokey and QubeOS 12 5 Recovery: Mount disk 12 6 Disposable VM 13 6.1 Introduction ............................ 14 6.1.1 References ......................... 14 6.1.2 What is a disposable VM? ................ 14 6.2 Playing online video ....................... 14 6.3 Web browsing ........................... 15 6.4 Manipulating untrusted files/data ................ 16 1 6.5 Mounting LVM image ...................... 17 6.6 Replace sys-* VM ......................... 18 6.7 Replace some AppVMs ...................... 18 7 Building a new QubesOS package 18 7.1 References ............................. 18 7.2 Goal ................................ 18 7.3 The software ............................ 19 7.4 Packaging ............................. 19 7.5 Building .............................. 21 2 1 Wiping VM Before starting this section, it should be noted that I don’t have a prior low level knownledge of data wiping. This section is some thinkings about how can we assure that a file have been deleted. And in our context, the goal would be to wipe out a LVM image. 1.1 Low level storage technologies 1.1.1 Must read • Wei.pdf • how-can-i-reliably-erase-all-information-on-a-hard-drive • https://www.sans.org/blog/spin-stand-microscopy-of-hard-disk-data/ • https://www.vidarholen.net/ vidar/overwriting_hard_drive_data.pdf • https://wiki.archlinux.org/index.php/Securely_wipe_disk 1.1.2 TL;DR of my understanding of the issue Magnetic disk have non physical way of wiping data that are considered as relatively reliable. SSD are a nightmare to wipe anything, and in most of the cases, it is impossible. 1.1.3 Things that could by implemented by QubesOS TODO. Links to the ongoing discussion on the subjects. But basically my idea: encryption, and store the encryption key on a specific physical device that can be logically destroyed (HDD ?), or that can be easily physically destroyed (MicroSD ?) TODO 2 Create a Gaming HVM 2.1 References Everythings needed is referenced here 3 • Usefull technical details • Reddit thread of what is needed for GPU passthrough • Solution to have more than 3Go of RAM in the Windows HVM • Some old references 2.2 Prerequise You have a functional Windows HVM (Windows 7 or Windows 10). The "how to" for this part can be found on the Qubes OS documentation and here: Usefull github comment. However, few tips: • Do a backup (clone VM) of the Windows HVM BEFORE starting to install QWT (installing QWT is not required) 2.3 Hardware To have a Windows HVM for gaming, you must have: • A dedicated AMD GPU. By dedicated, it means: it is a secondary GPU, not the GPU used to display dom0. Nvidia GPU are not sup- ported (or maybe with a lot of tricks). • A really fast disk (M.2 disk) • A lot of RAM • A dedicated screen • Dedicated gaming mouse and keyboard In my case: • Secondary GPU: AMD RX580 • Primary GPU: Some Nvidia trash, used for dom0 • 32Go of RAM. 12Go of RAM will be dedicated for the Windows HVM • A fast M.2 disk 4 2.4 Checklist Short list of things to do to make the GPU passthrough work: • You verified and confirmed that the secondary GPU is alone in its IOMMU Group • In dom0, you edited the file /etc/default/grub or /boot/efi/EFI/qubes/xen.cfg to allow PCI hiding for your secondary GPU, and regenerated the grub if needed • You have patched stubdom-linux-rootfs.gz to allow to have more than 3Go of RAM for your HVM 2.5 IOMMU Group Warning: I am far from understanding the IOMMU group. Check online references on that subject. It seems that you can only do a successfull GPU passthough if you can passthrough everything that is in the IOMMU Group of the GPU. Also, you can’t see your IOMMU Group when you are using Xen (the information is hidden from dom0). So, what I did: I booted from a Linux Mint Live USB. In the grub I enabled the IOMMU (iommu=1 iommu_amd=on), and then displayed the folder structure of /sys/kernel/iommu_group tree /sys/kernel/iommu_group My secondary GPU was alone in its IOMMU group. 2.6 GRUB modification You must hide your secondary GPU from dom0. To do that, you have to edit the GRUB. In a dom0 Terminal, type: qvm-pci Then find the devices id for your secondary gpu. In my case, it is dom0:0a_00.0 and dom0:0a_00.1. Edit /etc/default/grub, and add the PCI hiding GRUB_CMDLINE_LINUX="... rd.qubes.hide_pci=0a:00.0,0a:00.1 " then regenerate the grub grub2-mkconfig -o /boot/grub2/grub.cfg 5 2.7 Patching stubdom-linux-rootfs.gz Follow the instructions here: github.com/QubesOS/qubes-issues/issues/4321 Copy-paste of the comment: This is caused by the default TOLUD (Top of Low Usable DRAM) of 3.75G provided by qemu not being large enough to accommodate the larger BARs that a graphics card typically has. The code to pass a custom max- ram-below-4g value to the qemu command line does exist in the libxl_dm.c file of xen, but there is no functionality in libvirt to add this parameter. It is possible to manually add this parameter to the qemu commandline by doing the following in a dom0 terminal: mkdir stubroot cp /usr/lib/xen/boot/stubdom-linux-rootfs ,! stubroot/stubdom-linux-rootfs.gz cd stubroot gunzip stubdom-linux-rootfs.gz cpio -i -d -H newc --no-absolute-filenames < ,! stubdom-linux-rootfs rm stubdom-linux-rootfs nano init Before the line "#$dm_args and $kernel are separated with \x1b to allow for spaces in arguments." add: SP=$'\x1b' dm_args=$(echo "$dm_args" \ | sed "s/-machine\\${SP}xenfv/-machine\ \\${SP}xenfv,max-ram-below-4g=3.5G/g") Then execute: find . -print0 | cpio --null -ov \ --format=newc | gzip -9 > ../stubdom-linux-rootfs sudo mv ../stubdom-linux-rootfs /usr/lib/xen/boot/ Note that this will apply the change to all HVMs, so if you have any other HVM with more than 3.5G ram assigned, they will not start without the adapter being passed through. Ideally to fix this libvirt should be extended to pass the max-ram-below-4g parameter through to xen, and then a calculation added to determine the correct TOLUD based on the total BAR size of the PCI devices are being passed through to the vm. 6 2.8 Pass the GPU In qubes settings for the windows HVM, go to the "devices" tab, pass the ID corresponding to your AMD GPU. (in my case, it was 0a:00.0 and 0a:00.1) And check the option for "nostrict reset" for those 2. In some case, you might also need to set the "permissive" flag to true (But I didn’t need that with the RX 580): qvm-pci attach windows-hvm dom0:0a_00.0 -o permissive=True -o ,! no-strict-reset=True qvm-pci attach windows-hvm dom0:0a_00.1 -o permissive=True -o ,! no-strict-reset=True 2.9 Conclusion Don’t forget to install the GPU drivers, you can install the official one from AMD website, no modification or trick to do. Nothing else is required to make it work (in my case at least, once I finish to fight to find those informations). If you have issues, you can refer to the links in the first sections. If it doesn’t work and you need to debug more things, you can go deeper. • Virsh (start, define, ...) • /etc/libvirt/libxl/ • xl • /etc/qubes/templates/libvirt/xen/by-name/ • /usr/lib/xen/boot/ • virsh -c xen:/// domxml-to-native xen-xm /etc/libvirt/libxl/... I am able to play games on my windows HVM with very good perfor- mances. And safely. 2.10 Bugs The AMD GPUs have a bug when used in HVM: each time you will reboot your windows HVM, it will get slower and slower. It is because the AMD GPUs is not correctly resetted when you restart your windows HVM Two solutions for that: • Reboot your computer 7 • In the windows HVM, use to windows option in the system tray to "safely remove devices", remove your GPU. Restart the HVM. This bug is referenced somewhere, but lost the link and too lazy to search for it. 3 Create a Linux Gaming HVM, integrated with QubesOS 3.1 Goals We want a qube with the following characteristics: • Standalone linux from a template • With GPU passthough • Full Qubes integration • Able to play video games from Steam 3.2 Hardware used The same as for my article "Create a Gaming HVM". Prerequise You already followed my article "Create a Gaming HVM" 3.3 Main steps summary 1. Create a standalone qube based on a debian template 2.