distri researching fast management Michael Stapelberg @zekjur 2020-10-11 Overview

● 1-minute introduction

● demo videos: arch vs. distri package installation speed

● Comparison with

● How does distri work? Introduction: Michael Stapelberg

● Developer for 7 years (2012-2019) ○ left Debian because of antique tooling and slow changes

● Using Arch Linux for 1 year ○ used Fedora and NixOS each for a few months ○ in a previous life, used Gentoo, and NetBSD

● Wrote the i3 in 2009

● other FOSS projects, too! Debian Code Search, RobustIRC, gokrazy, … demo: installing “ack”

Arch distri demo: installing “qemu”

Arch distri Updates/package install: faster in distri

● transport compression → Arch switched to zstd in 2020-01-04 

● mirror selection → Arch asks its users to maintain their mirror list  → Why can’t Arch default to a CDN that’s fast everywhere?

● no hooks/triggers: maximum parallelism Arch is moving from package hooks to pacman hooks (e.g. sysusers) 

● no unpacking stage: use images instead of archives Updates/package install: more robust in distri

● Arch does not support partial upgrades  → distri packages depend on the specific transitive closure, so can always be installed

● Arch upgrades frequently require manual intervention  → distri packages use separate hierarchies: file conflicts impossible :) → distri packages are hermetic: not easily broken by other packages on the system Debugging experience

● Installing gdb should be all that a user needs to do: Debug symbols and sources of any package should be fetched on demand!

● Arch does not (yet) provide debug infos for all packages  Arch does not (yet) transparently make available symbols  → will be solved with debuginfod

● (distri solves this on the level) Packaging experience

● quicker feedback → more engaging → more contributions

● isolating package builds from the host system should be the default Arch asks package maintainers to do manual management  Changes over time

● declarative packaging is key to make changes happen the Arch is a custom format, not defined anywhere → want auto-formatting → want machines to be able to make edits (→ monorepo?) → express intents/end states, not mechanisms How does distri work? package manager speed: install “ack” (Perl)*

distribution package data wall-clock time rate manager

Fedora 114 MB 33s 3.4 MB/s

Debian 16 MB 10s 1.6 MB/s

NixOS Nix 15 MB 5s 3.0 MB/s

Arch Linux pacman 6.5 MB 3s 2.1 MB/s

Alpine apk 10 MB 1s 10.0 MB/s

rate = data ÷ wall-clock time * standard installation, includes metadata & package download and dependencies → https://michael.stapelberg.ch/posts/2019-08-17-linux-package-managers-are-slow/ Why are package managers slow?

● 2 most widely used package formats: ○ (Debian package), (1) in (1) ○ rpm (Red Hat Package Manager), metadata around (1) ○ (Arch: tar(1) with metadata)

● task: make package contents available → e.g. pacman -S nginx results in /usr/bin/nginx

● traditionally: resolve deps, download, extract, configure → need to carefully fsync(2) to make I/O as safe as possible How can we go faster? append-only package store of immutable images

1. use an image format (e.g. SquashFS) instead of an archive format

2. each image under its own path (“separate hierarchies”): e.g. /ro/nginx-amd64-1.14.1/… e.g. /ro/zsh-amd64-5.6.2/…

3. (rest of the system as usual, e.g. /etc, /var/cache, …) advantages

● mount instead of extract → faster package installation → faster build environment composition

● append-only: can use unsafe I/O

● immutable: no longer possible to screw up your installation hermetic packages

● when run, use the same version of dependencies as when built

● a wrapper script sets e.g. LD_LIBRARY_PATH, PYTHONPATH, PERL5LIB, … separate hierarchies: exchange dirs

● packages exchange data via directories with well-known paths, e.g.: man(1) ⟷ nginx(1) via /usr/share/man gcc(1) ⟷ libusb(3) via /usr/include

● prudent approach: emulate well-known paths e.g.: /usr/include/jpeglib.h is a symlink to /ro/libjpeg-turbo-amd64-2.0.0/out/include/jpeglib.h separate hierarchies: exchange dirs (per package)

● loose coupling (global) vs. tight coupling (per package) → typically suitable for plugin mechanisms where ABI must match

● e.g. /ro/xorg-server-amd64-1.20.3/out/lib/xorg/modules/ separate hierarchies: advantages

● move conflicts from package installation to program execution → only need to resolve /bin/python (2.7 or 3?) when assembling /bin

● packages always co-installable e.g. zsh-amd64-5.6.2 and zsh-amd64-5.6.3 → partial updates/rollbacks easily possible

● package manager can be version-agnostic! → entirely eliminates a large source of slowness → no need for global metadata, package-specific metadata sufficient immutability

● package contents and exchange dirs are read-only

● rarely, programs expect the system to be writable e.g. GNOME’s gsettings wants a cache in the exchange directory

● such designs need to be improved upstream: 1. good caches are not required (fallback to slow path) 2. good caches are transparently created 3. good caches are automatically updated when needed no hooks/triggers (1)

● hook (or maintscript, postinst, …): program run after package installation trigger: program run after other package installation (e.g. man-db) → work at package-installation time which may be unnecessary

● preclude concurrent package installation (not implemented concurrency-safe)

● arbitrary code, can be slow no hooks/triggers (2)

● claim: we can build a functioning system without hooks/triggers

● 1. packages declare what they need (e.g. sysusers)

● 2. move work from package installation to program execution e.g. ssh needs a hostkey: create it in sshd(8) wrapper script

● very few exceptions: bootloader or firmware (need to install them outside of the file system) practicality

● FUSE file system for providing /ro → easier to implement than managing separate mounts, overlays, unions, … → faster (!), as kernel mounts are slow

● packages need to be built with --prefix=/ro/nginx-amd64-1.14.1 etc.

● a small number of packages need to be patched → path-related issues (e.g. service files, gcc, gobject, automake, …) → deep system integration (e.g. ) practicality (2)

● removal of hooks is not for everyone → configuration layers (debconf, YAST, …) might be a feature to some Why is distri faster?

● traditionally: resolve deps, download, extract, configure + careful fsync(2) to make I/O as safe as possible

● distri: resolve deps, download image, extract, configure (unsafe I/O okay) → scales to 12+ GB/s (!) on 100 Gbit links using Go’s net/http conclusion

1. append-only package stores are more elegant than mutable systems → simpler design, faster implementation

2. exchange directories make things seem normal to third-party software → can compile unpackaged software, can run closed-source binaries

3. all of these ideas are practical → live CDs (read-only) and cross-compilation paved the way project goals

● Not trying to build a community or user base!

● Instead, distri enables (my) research, with regular proof of concept releases

● Now that you know the pain points and how fast it could be, maybe you can improve things? :) ●

●

●

●