Software deployment with Nix Eelco Dolstra [email protected] Universiteit Utrecht, Faculty of Science, Department of Information and Computing Sciences March 27, 2006 Overview TraCE Project I Part of the NWO Jacquard program I Universiteit Utrecht Nix What it does: I Software deployment (“package management”) I Service deployment I Continuous integration and release management I Build management I NixOS Software Deployment I Software deployment: the art of transferring software (components) from one machine to another (and managing it). I “All activities that make a software system available for use” (Carzaniga et al. 1998) I Covers activities such as: I Packaging I Transferring I Installing I Configuring I Updating I Uninstalling Deployment Problems Software deployment (the act of transferring software to another system) is surprisingly hard. I It’s hard to ensure correctness (the software should work the same on the source and target systems). I It’s too much work. I Deployment systems tend to be inflexible. So why is this hard? I Difficult to have multiple versions; but we want this to I Test upgrades I Deal with conflicting dependencies I Support different user / service requirements gtk+ wxGTK zapping wxPython BitTorrent So why is this hard? I Difficult to have multiple versions; but we want this to I Test upgrades I Deal with conflicting dependencies I Support different user / service requirements gtk+ wxGTK zapping Requires gtk+-2.4 wxPython BitTorrent So why is this hard? I Difficult to have multiple versions; but we want this to I Test upgrades I Deal with conflicting dependencies I Support different user / service requirements gtk+ wxGTK zapping Fails with gtk+-2.4 Requires gtk+-2.4 wxPython BitTorrent So why is this hard? I Unreliable dependency information I What components are needed? I What versions? gtk+ wxGTK wxPython python BitTorrent So why is this hard? I Unreliable dependency information I What components are needed? I What versions? gtk+ wxGTK wxPython python Missing! BitTorrent So why is this hard? glibc-2.3.3 xextensions-1.0.1 libXau-0.1.1 libXtrans-0.1 xproto-6.6.1 renderext-0.8 libX11-6.2.1 freetype-2.1.5 expat-1.95.8 libICE-6.3.3 libXext-6.4.3 libXrender-0.8.4 fontconfig-2.2.3 libSM-6.0.3 coreutils-5.2.1 libXv-2.2.2 libXft-2.1.6 libXt-0.1.4-cvs perl-5.8.5 libjpeg-6b gcc-3.4.2 zlib-1.2.1 glib-2.2.3 xlib-1.0 glib-2.4.7 libtiff-3.6.1 libpng-1.2.7 python-2.3.4 atk-1.2.4 pango-1.2.5 pango-1.4.1 popt-1.7 atk-1.6.1 audiofile-0.2.3 libIDL-0.8.2 zvbi-0.2.8 gtk+-2.2.4 gtk+-2.4.13 libxml2-2.6.13 esound-0.2.32 ORBit2-2.8.3 wxGTK-2.4.2 libglade-2.0.1 GConf-2.4.0.1 libart_lgpl-2.3.16 libbonobo-2.4.2 wxPython-2.4.2.4 libgnomecanvas-2.4.0 gnome-vfs-2.4.2 bittorrent-3.4.2 libgnome-2.0.6 BitTorrent libbonoboui-2.4.1 rte-0.5.2 libgnomeui-2.4.0.1 Zapping zapping-0.7 Unresolved Component Dependencies Producer Site Application App Libraries When we deploy a LibA LibB I version 0.5 version 1.3 component. I . we have to ensure that all its dependencies are present on the target system Unresolved Component Dependencies Producer Site Application App Libraries When we deploy a LibA LibB I version 0.5 version 1.3 component. I . we have to ensure that all its dependencies are Consumer Site Application present on the App target system Libraries LibA version 0.3 ?! Component Interference Applications App1 App2 App3 Operations on a component (install, Libraries upgrade, remove) often LLiibbA1 LiibBb2 break other components (interference). E.g.: I Upgrade of App2 breaks App1 due to upgrade of LibB to LibB’ I Removal of App3 breaks App1 due to removal of LibA Component Interference Applications App1 App2 App3 Operations on a component (install, Libraries upgrade, remove) often LLiibbA1 LiibBb2 break other components (interference). E.g.: Upgrade of App2 I Upgrade of App2 breaks App1 due to Applications upgrade of LibB to App1 App2' App3 LibB’ Libraries I Removal of App3 LibA LibB' breaks App1 due to removal of LibA Component Interference Applications App1 App2 App3 Operations on a component (install, Libraries upgrade, remove) often LLiibbA1 LiibBb2 break other components (interference). E.g.: Upgrade of App2 Removal of App3 I Upgrade of App2 breaks App1 due to Applications Applications upgrade of LibB to App1 App2' App3 App1 App2 App3 LibB’ Libraries Libraries I Removal of App3 LibA LibB' L?i!b1 LiibBb2 breaks App1 due to removal of LibA Tool Support I Deployment was (is) often done in an ad hoc, undisciplined fashion. I Files installed in global locations (/usr/bin, C:/Windows/System32). I “DLL Hell” — overwriting of shared components with older/newer versions. I “Dependency Hell” — components may have gazillions of dependencies. I Each application has its own (un)installer (so no unified view on the system). I Interactive installers ⇒ considered harmful (hard to automate). I Packaging = lots of work. I Package managers manage software installations in a unified way: RPM, FreeBSD Ports/Packages, Depot, Debian apt-get/dpkg, ..., Nix. Requirements on a Deployment System I Support multiple versions, variants. I Handle dependencies. I Ensure safe upgrades / uninstalls. I Atomic upgrades/downgrades (e.g., important in server environments). I Provide a good composition mechanism. I Allow different “views” for multiple users. I Unique identification of configurations. I ... The Nix Deployment System I Central idea: store all components in isolation. I Unique paths: /nix/store/jjp9pirx8b3nqs9k...-firefox which is an SHA-256 hash of all inputs used to build the component: I Sources I Libraries I Compilers I Build scripts I Build parameters I System type I ... I Prevent undeclared build time dependencies. I Scan for runtime dependencies. I Deploy only closures under the depends-on relation. Nix store /nix/store bd6593219f8dcb63...-gtk+-2.2.4 lib libgtk-x11-2.0.so.0 ce2d7d2a41456bab...-wxGTK-2.4.2 lib libwx_gtk2-2.4.so e889db0595672287...-wxPython-2.4.2.4 (lots of Python bindings) 9ed8c4231bfde4af...-bittorrent-3.4.2 bin btdownloadgui.py 300ccc1a41af3abc...-gtk+-2.4.13 lib libgtk-x11-2.0.so.0 f51ec7d5663c735e-zapping-0.7.3 bin zapping Nix store /nix/store bd6593219f8dcb63...-gtk+-2.2.4 lib libgtk-x11-2.0.so.0 ce2d7d2a41456bab...-wxGTK-2.4.2 lib libwx_gtk2-2.4.so e889db0595672287...-wxPython-2.4.2.4 (lots of Python bindings) Unique paths for 9ed8c4231bfde4af...-bittorrent-3.4.2 different versions bin btdownloadgui.py 300ccc1a41af3abc...-gtk+-2.4.13 lib libgtk-x11-2.0.so.0 f51ec7d5663c735e-zapping-0.7.3 bin zapping Nix expressions hello/default.nix {stdenv, fetchurl, perl}: stdenv.mkDerivation { name = "hello-2.1.1"; builder = ./builder.sh; src = fetchurl { url = ftp://ftp.gnu.org/pub/gnu/hello/hello-2.1.1.tar.gz; md5 = "70c9ccf9fac07f762c24f2df2290784d"; }; inherit perl; } Nix expressions hello/default.nix {stdenv, fetchurl, perl}: Function arguments stdenv.mkDerivation { name = "hello-2.1.1"; builder = ./builder.sh; src = fetchurl { url = ftp://ftp.gnu.org/pub/gnu/hello/hello-2.1.1.tar.gz; md5 = "70c9ccf9fac07f762c24f2df2290784d"; }; inherit perl; } Nix expressions hello/default.nix {stdenv, fetchurl, perl}: Function arguments stdenv.mkDerivation { name = "hello-2.1.1"; builder = ./builder.sh; Build attributes src = fetchurl { url = ftp://ftp.gnu.org/pub/gnu/hello/hello-2.1.1.tar.gz; md5 = "70c9ccf9fac07f762c24f2df2290784d"; }; inherit perl; } Nix expressions hello/builder.sh source $stdenv/setup PATH=$perl/bin:$PATH tar xvfz $src cd hello-* ./configure --prefix=$out make make install Nix expressions hello/builder.sh source $stdenv/setup PATH=$perl/bin:$PATH tar xvfz $src cd hello-* Environment initially empty; pre- ./configure --prefix=$outvents undeclared dependencies make make install Nix expressions system/all-packages-generic.nix hello = (import ../applications/misc/hello/ex-1) { inherit fetchurl stdenv perl; }; perl = (import ../development/interpreters/perl) { inherit fetchurl stdenv; }; fetchurl = (import ../build-support/fetchurl) { inherit stdenv; ... }; stdenv = ...; Nix expressions system/all-packages-generic.nix hello = (import ../applications/misc/hello/ex-1) { inherit fetchurl stdenv perl; }; perl = (import ../development/interpreters/perl) { inherit fetchurl stdenv; }; fetchurl = (import ../build-support/fetchurl) { inherit stdenv; ... }; stdenv = ...; Variability bittorrent = (import ../tools/networking/bittorrent) { inherit fetchurl stdenv wxGTK; }; wxGTK = (import ../development/libraries/wxGTK) { inherit fetchurl stdenv pkgconfig; gtk = gtkLibs22.gtk; }; firefox = (import ../applications/browsers/firefox) { inherit fetchurl stdenv pkgconfig perl zip libIDL libXi; gtk = gtkLibs24.gtk; }; Variability { localServer, stdenv, fetchurl , openssl ? null, db4 ? null, ... }: assert localServer -> db4 != null; assert sslSupport -> openssl != null && && (httpServer -> httpd.openssl == openssl); stdenv.mkDerivation { name = "subversion-1.1.3"; builder = ./builder.sh; src = fetchurl {url=...}; ... } Finding runtime dependencies /nix/store bd6593219f8dcb63...-gtk+-2.2.4 lib libgtk-x11-2.0.so.0 ce2d7d2a41456bab...-wxGTK-2.4.2 lib libwx_gtk2-2.4.so e889db0595672287...-wxPython-2.4.2.4 (lots of Python bindings) 9ed8c4231bfde4af...-bittorrent-3.4.2 bin btdownloadgui.py 300ccc1a41af3abc...-gtk+-2.4.13 lib libgtk-x11-2.0.so.0 f51ec7d5663c735e-zapping-0.7.3 bin zapping Finding runtime dependencies /nix/store bd6593219f8dcb63...-gtk+-2.2.4 lib libgtk-x11-2.0.so.0 ce2d7d2a41456bab...-wxGTK-2.4.2 lib libwx_gtk2-2.4.so e889db05Contents95672287... of-wxPlibwx-gtk2-2.4.soython-2.4.2.4 (lots of Python bindings) 9ed8c42...31bfde4af...-bittorrent-3.4.2 2e 36 00 6c 69 62 73 74 64 63 2b 2b 2e 73 6f 2e |.6.libstdc++.so.| bin 36 00 6c 69 62 67 63 63 5f 73 2e 73 6f 2e 31 00 |6.libgcc_s.so.1.| bt6cdow 69nl 62oad 70gui 74.py 68 72 65 61 64 2e 73 6f 2e 30 00 |libpthread.so.0.| 6c 69 62 63 2e 73 6f 2e 36 00 5f 5f 63 78 61 5f |libc.so.6.__cxa_| 300ccc161a41 74af 653abc.