sign in sign up

<<

Software Deployment with Nix

Software Deployment with Nix

Software deployment with Nix

Eelco Dolstra [email protected]

Universiteit Utrecht, Faculty of Science, Department of Information and Computing Sciences

March 27, 2006 Overview

TraCE Project

I Part of the NWO Jacquard program

I Universiteit Utrecht

Nix What it does:

I Software deployment (“package management”)

I Service deployment

I Continuous integration and release management

I Build management

I NixOS Software Deployment

I Software deployment: the art of transferring software (components) from one machine to another (and managing it).

I “All activities that make a software system available for use” (Carzaniga et al. 1998)

I Covers activities such as:

I Packaging I Transferring I Installing I Conﬁguring I Updating I Uninstalling Deployment Problems

Software deployment (the act of transferring software to another system) is surprisingly hard.

I It’s hard to ensure correctness (the software should work the same on the source and target systems).

I It’s too much work.

I Deployment systems tend to be inﬂexible. So why is this hard?

I Difficult to have multiple versions; but we want this to I Test upgrades I Deal with conflicting dependencies I Support different user / service requirements

wxGTK zapping

wxPython

BitTorrent So why is this hard?

I Difficult to have multiple versions; but we want this to I Test upgrades I Deal with conflicting dependencies I Support different user / service requirements

gtk+

wxGTK zapping Requires gtk+-2.4

wxPython

BitTorrent So why is this hard?

I Difficult to have multiple versions; but we want this to I Test upgrades I Deal with conflicting dependencies I Support different user / service requirements

gtk+

wxGTK zapping Fails with gtk+-2.4 Requires gtk+-2.4

wxPython

BitTorrent So why is this hard?

I Unreliable dependency information

I What components are needed? I What versions?

gtk+

wxGTK

wxPython python

BitTorrent So why is this hard?

I Unreliable dependency information

I What components are needed? I What versions?

gtk+

wxGTK

wxPython python Missing!

BitTorrent So why is this hard?

glibc-2.3.3

xextensions-1.0.1 libXau-0.1.1 libXtrans-0.1 xproto-6.6.1

renderext-0.8 libX11-6.2.1 freetype-2.1.5 expat-1.95.8 libICE-6.3.3

libXext-6.4.3 libXrender-0.8.4 fontconfig-2.2.3 libSM-6.0.3 coreutils-5.2.1

libXv-2.2.2 libXft-2.1.6 libXt-0.1.4-cvs perl-5.8.5

libjpeg-6b gcc-3.4.2 zlib-1.2.1 glib-2.2.3 xlib-1.0 glib-2.4.7

libtiff-3.6.1 libpng-1.2.7 python-2.3.4 atk-1.2.4 pango-1.2.5 pango-1.4.1 popt-1.7 atk-1.6.1 audiofile-0.2.3 libIDL-0.8.2

zvbi-0.2.8 gtk+-2.2.4 gtk+-2.4.13 libxml2-2.6.13 esound-0.2.32 ORBit2-2.8.3

wxGTK-2.4.2 libglade-2.0.1 GConf-2.4.0.1 libart_lgpl-2.3.16 libbonobo-2.4.2

wxPython-2.4.2.4 libgnomecanvas-2.4.0 gnome-vfs-2.4.2

bittorrent-3.4.2 libgnome-2.0.6 BitTorrent

libbonoboui-2.4.1 rte-0.5.2

libgnomeui-2.4.0.1 Zapping zapping-0.7 Unresolved Component Dependencies

Producer Site Application App

Libraries When we deploy a LibA LibB I version 0.5 version 1.3 component. . .

I . . . we have to ensure that all its dependencies are present on the target system Unresolved Component Dependencies

Producer Site Application App

Libraries When we deploy a LibA LibB I version 0.5 version 1.3 component. . .

I . . . we have to ensure that all its dependencies are Consumer Site Application present on the App target system

Libraries LibA version 0.3 ?! Component Interference

Applications App1 App2 App3 Operations on a component (install, Libraries upgrade, remove) often LLiibbA1 LiibBb2 break other components (interference). E.g.:

I Upgrade of App2 breaks App1 due to upgrade of LibB to LibB’

I Removal of App3 breaks App1 due to removal of LibA Component Interference

Applications App1 App2 App3 Operations on a component (install, Libraries upgrade, remove) often LLiibbA1 LiibBb2 break other components (interference). E.g.: Upgrade of App2 I Upgrade of App2 breaks App1 due to Applications upgrade of LibB to App1 App2' App3 LibB’

Libraries I Removal of App3 LibA LibB' breaks App1 due to removal of LibA Component Interference

Applications App1 App2 App3 Operations on a component (install, Libraries upgrade, remove) often LLiibbA1 LiibBb2 break other components (interference). E.g.: Upgrade of App2 Removal of App3 I Upgrade of App2 breaks App1 due to Applications Applications upgrade of LibB to App1 App2' App3 App1 App2 App3 LibB’

Libraries Libraries I Removal of App3 LibA LibB' L?i!b1 LiibBb2 breaks App1 due to removal of LibA Tool Support

I Deployment was (is) often done in an ad hoc, undisciplined fashion.

I Files installed in global locations (/usr/bin, C:/Windows/System32).

I “DLL Hell” — overwriting of shared components with older/newer versions.

I “Dependency Hell” — components may have gazillions of dependencies.

I Each application has its own (un)installer (so no uniﬁed view on the system).

I Interactive installers ⇒ considered harmful (hard to automate). I Packaging = lots of work.

I Package managers manage software installations in a uniﬁed way: RPM, FreeBSD Ports/Packages, Depot, Debian apt-get/dpkg, ..., Nix. Requirements on a Deployment System

I Support multiple versions, variants.

I Handle dependencies.

I Ensure safe upgrades / uninstalls.

I Atomic upgrades/downgrades (e.g., important in server environments).

I Provide a good composition mechanism.

I Allow diﬀerent “views” for multiple users.

I Unique identiﬁcation of conﬁgurations.

I ... The Nix Deployment System

I Central idea: store all components in isolation.

I Unique paths:

/nix/store/jjp9pirx8b3nqs9k...-firefox which is an SHA-256 hash of all inputs used to build the component:

I Sources I Libraries I Compilers I Build scripts I Build parameters I System type I ...

I Prevent undeclared build time dependencies.

I Scan for runtime dependencies.

I Deploy only closures under the depends-on relation. Nix store

/nix/store bd6593219f8dcb63...-gtk+-2.2.4 lib libgtk-x11-2.0.so.0 ce2d7d2a41456bab...-wxGTK-2.4.2 lib libwx_gtk2-2.4.so e889db0595672287...-wxPython-2.4.2.4 (lots of Python bindings) 9ed8c4231bfde4af...-bittorrent-3.4.2 bin btdownloadgui.py 300ccc1a41af3abc...-gtk+-2.4.13 lib libgtk-x11-2.0.so.0 f51ec7d5663c735e-zapping-0.7.3 bin zapping Nix store

/nix/store bd6593219f8dcb63...-gtk+-2.2.4 lib libgtk-x11-2.0.so.0 ce2d7d2a41456bab...-wxGTK-2.4.2 lib libwx_gtk2-2.4.so e889db0595672287...-wxPython-2.4.2.4 (lots of Python bindings) Unique paths for 9ed8c4231bfde4af...-bittorrent-3.4.2 different versions bin btdownloadgui.py 300ccc1a41af3abc...-gtk+-2.4.13 lib libgtk-x11-2.0.so.0 f51ec7d5663c735e-zapping-0.7.3 bin zapping Nix expressions

hello/default.nix

{stdenv, fetchurl, perl}:

stdenv.mkDerivation { name = "hello-2.1.1"; builder = ./builder.sh; src = fetchurl { url = ftp://ftp.gnu.org/pub/gnu/hello/hello-2.1.1.tar.gz; md5 = "70c9ccf9fac07f762c24f2df2290784d"; }; inherit perl; } Nix expressions

hello/default.nix

{stdenv, fetchurl, perl}: Function arguments stdenv.mkDerivation { name = "hello-2.1.1"; builder = ./builder.sh; src = fetchurl { url = ftp://ftp.gnu.org/pub/gnu/hello/hello-2.1.1.tar.gz; md5 = "70c9ccf9fac07f762c24f2df2290784d"; }; inherit perl; } Nix expressions

hello/default.nix

{stdenv, fetchurl, perl}: Function arguments stdenv.mkDerivation { name = "hello-2.1.1"; builder = ./builder.sh; Build attributes src = fetchurl { url = ftp://ftp.gnu.org/pub/gnu/hello/hello-2.1.1.tar.gz; md5 = "70c9ccf9fac07f762c24f2df2290784d"; }; inherit perl; } Nix expressions

hello/builder.sh

source $stdenv/setup

PATH=$perl/bin:$PATH

tar xvfz $src cd hello-* ./configure --prefix=$out make make install Nix expressions

hello/builder.sh

source $stdenv/setup

PATH=$perl/bin:$PATH

tar xvfz $src cd hello-* Environment initially empty; pre- ./configure --prefix=$outvents undeclared dependencies make make install Nix expressions

system/all-packages-generic.nix

hello = (import ../applications/misc/hello/ex-1) { inherit fetchurl stdenv perl; };

perl = (import ../development/interpreters/perl) { inherit fetchurl stdenv; };

fetchurl = (import ../build-support/fetchurl) { inherit stdenv; ... };

stdenv = ...; Nix expressions

system/all-packages-generic.nix

hello = (import ../applications/misc/hello/ex-1) { inherit fetchurl stdenv perl; };

perl = (import ../development/interpreters/perl) { inherit fetchurl stdenv; };

fetchurl = (import ../build-support/fetchurl) { inherit stdenv; ... };

stdenv = ...; Variability

bittorrent = (import ../tools/networking/bittorrent) { inherit fetchurl stdenv wxGTK; };

wxGTK = (import ../development/libraries/wxGTK) { inherit fetchurl stdenv pkgconfig; gtk = gtkLibs22.gtk; };

firefox = (import ../applications/browsers/firefox) { inherit fetchurl stdenv pkgconfig perl zip libIDL libXi; gtk = gtkLibs24.gtk; }; Variability

{ localServer, stdenv, fetchurl , openssl ? null, db4 ? null, ... }:

assert localServer -> db4 != null; assert sslSupport -> openssl != null && && (httpServer -> httpd.openssl == openssl);

stdenv.mkDerivation { name = "subversion-1.1.3"; builder = ./builder.sh; src = fetchurl {url=...}; ... } Finding runtime dependencies

/nix/store bd6593219f8dcb63...-gtk+-2.2.4 lib libgtk-x11-2.0.so.0 ce2d7d2a41456bab...-wxGTK-2.4.2 lib libwx_gtk2-2.4.so e889db0595672287...-wxPython-2.4.2.4 (lots of Python bindings) 9ed8c4231bfde4af...-bittorrent-3.4.2 bin btdownloadgui.py 300ccc1a41af3abc...-gtk+-2.4.13 lib libgtk-x11-2.0.so.0 f51ec7d5663c735e-zapping-0.7.3 bin zapping Finding runtime dependencies

/nix/store bd6593219f8dcb63...-gtk+-2.2.4 lib libgtk-x11-2.0.so.0 ce2d7d2a41456bab...-wxGTK-2.4.2 lib libwx_gtk2-2.4.so e889db05Contents95672287... of-wxPlibwx-gtk2-2.4.soython-2.4.2.4 (lots of Python bindings) 9ed8c42...31bfde4af...-bittorrent-3.4.2 2e 36 00 6c 69 62 73 74 64 63 2b 2b 2e 73 6f 2e |.6.libstdc++.so.| bin 36 00 6c 69 62 67 63 63 5f 73 2e 73 6f 2e 31 00 |6.libgcc_s.so.1.| bt6cdow 69nl 62oad 70gui 74.py 68 72 65 61 64 2e 73 6f 2e 30 00 |libpthread.so.0.| 6c 69 62 63 2e 73 6f 2e 36 00 5f 5f 63 78 61 5f |libc.so.6.__cxa_| 300ccc161a41 74af 653abc. 78. 69.-gt 74k+ 00-2.4 5f.13 65 64 61 74 61 00 5f 5f |atexit._edata.__| lib 62 73 73 5f 73 74 61 72 74 00 2f 6e 69 78 2f 73 |bss_start./nix/s| 74 6f 72 65 2f 62 64 36 35 39 33 32 31 39 66 38 |tore/bd6593219f8| libgt64 63k-x1 621-2 36.0. 33so. 300 61 34 35 35 62 31 61 35 37 66 |dcb630a455b1a57f| f51ec7d536663c7 34 3635 33e-zappi 33 2dng-0. 67 747.3 6b 2b 2d 32 2e 32 2e 34 |64633-gtk+-2.2.4| 2f 6c 69 62 3a 2f 6e 69 78 2f 73 74 6f 72 65 2f |/lib:/nix/store/| bin 62 37 65 62 34 37 36 64 36 32 62 61 65 38 62 63 |b7eb476d62bae8bc| ... zapping Finding runtime dependencies

/nix/store bd6593219f8dcb63...-gtk+-2.2.4 lib libgtk-x11-2.0.so.0 ce2d7d2a41456bab...-wxGTK-2.4.2 lib libwx_gtk2-2.4.so e889db05Contents95672287... of-wxPlibwx-gtk2-2.4.soython-2.4.2.4 (lots of Python bindings) 9ed8c42...31bfde4af...-bittorrent-3.4.2 2e 36 00 6c 69 62 73 74 64 63 2b 2b 2e 73 6f 2e |.6.libstdc++.so.| bin 36 00 6c 69 62 67 63 63 5f 73 2e 73 6f 2e 31 00 |6.libgcc_s.so.1.| bt6cdow 69nl 62oad 70gui 74.py 68 72 65 61 64 2e 73 6f 2e 30 00 |libpthread.so.0.| 6c 69 62 63 2e 73 6f 2e 36 00 5f 5f 63 78 61 5f |libc.so.6.__cxa_| 300ccc161a41 74af 653abc. 78. 69.-gt 74k+ 00-2.4 5f.13 65 64 61 74 61 00 5f 5f |atexit._edata.__| lib 62 73 73 5f 73 74 61 72 74 00 2f 6e 69 78 2f 73 |bss_start./nix/s| 74 6f 72 65 2f 62 64 36 35 39 33 32 31 39 66 38 |tore/bd6593219f8| libgt64 63k-x1 621-2 36.0. 33so. 300 61 34 35 35 62 31 61 35 37 66 |dcb630a455b1a57f| f51ec7d536663c7 34 3635 33e-zappi 33 2dng-0. 67 747.3 6b 2b 2d 32 2e 32 2e 34 |64633-gtk+-2.2.4| 2f 6c 69 62 3a 2f 6e 69 78 2f 73 74 6f 72 65 2f |/lib:/nix/store/| bin 62 37 65 62 34 37 36 64 36 32 62 61 65 38 62 63 |b7eb476d62bae8bc| ... zapping Finding runtime dependencies

/nix/store bd6593219f8dcb63...-gtk+-2.2.4 lib libgtk-x11-2.0.so.0 ce2d7d2a41456bab...-wxGTK-2.4.2 lib libwx_gtk2-2.4.so e889db05Contents95672287... of-wxPlibwx-gtk2-2.4.soython-2.4.2.4 (lots of Python bindings) 9ed8c42...31bfde4af...-bittorrent-3.4.2 2e 36 00 6c 69 62 73 74 64 63 2b 2b 2e 73 6f 2e |.6.libstdc++.so.| bin 36 00 6c 69 62 67 63 63 5f 73 2e 73 6f 2e 31 00 |6.libgcc_s.so.1.| bt6cdow 69nl 62oad 70gui 74.py 68 72 65 61 64 2e 73 6f 2e 30 00 |libpthread.so.0.| 6c 69 62 63 2e 73 6f 2e 36 00 5f 5f 63 78 61 5f |libc.so.6.__cxa_| 300ccc161a41 74af 653abc. 78. 69.-gt 74k+ 00-2.4 5f.13 65 64 61 74 61 00 5f 5f |atexit._edata.__| lib 62 73 73 5f 73 74 61 72 74 00 2f 6e 69 78 2f 73 |bss_start./nix/s| 74 6f 72 65 2f 62 64 36 35 39 33 32 31 39 66 38 |tore/bd6593219f8| libgt64 63k-x1 621-2 36.0. 33so. 300 61 34 35 35 62 31 61 35 37 66 |dcb630a455b1a57f| f51ec7d536663c7 34 3635 33e-zappi 33 2dng-0. 67 747.3 6b 2b 2d 32 2e 32 2e 34 |64633-gtk+-2.2.4| 2f 6c 69 62 3a 2f 6e 69 78 2f 73 74 6f 72 65 2f |/lib:/nix/store/| bin 62 37 65 62 34 37 36 64 36 32 62 61 65 38 62 63 |b7eb476d62bae8bc| ... zapping Finding runtime dependencies

/nix/store bd6593219f8dcb63...-gtk+-2.2.4 lib libgtk-x11-2.0.so.0 ce2d7d2a41456bab...-wxGTK-2.4.2 lib libwx_gtk2-2.4.so e889db0595672287...-wxPython-2.4.2.4 (lots of Python bindings) 9ed8c4231bfde4af...-bittorrent-3.4.2 bin btdownloadgui.py 300ccc1a41af3abc...-gtk+-2.4.13 lib libgtk-x11-2.0.so.0 f51ec7d5663c735e-zapping-0.7.3 bin zapping Finding runtime dependencies

/nix/store bd6593219f8dcb63...-gtk+-2.2.4 lib libgtk-x11-2.0.so.0 ce2d7d2a41456bab...-wxGTK-2.4.2 lib libwx_gtk2-2.4.so e889db0595672287...-wxPython-2.4.2.4 (lots of Python bindings) 9ed8c4231bfde4af...-bittorrent-3.4.2 bin btdownloadgui.py 300ccc1a41af3abc...-gtk+-2.4.13 lib libgtk-x11-2.0.so.0 f51ec7d5663c735e-zapping-0.7.3 bin zapping Finding runtime dependencies

/nix/store bd6593219f8dcb63...-gtk+-2.2.4 lib libgtk-x11-2.0.so.0 ce2d7d2a41456bab...-wxGTK-2.4.2 lib libwx_gtk2-2.4.so e889db0595672287...-wxPython-2.4.2.4 (lots of Python bindings) 9ed8c4231bfde4af...-bittorrent-3.4.2 bin btdownloadgui.py 300ccc1a41af3abc...-gtk+-2.4.13 lib libgtk-x11-2.0.so.0 f51ec7d5663c735e-zapping-0.7.3 bin zapping User operations

I To build and install Hello: $ nix-env -if .../all-packages.nix hello

I When a new version comes along:

$ nix-env -uf .../all-packages.nix hello

I If it doesn’t work: $ nix-env --rollback

I Delete unused components:

$ nix-collect-garbage User operations

I To build and install Hello: $ nix-env -if .../all-packages.nix hello

I When a new version comes along:

$ nix-env -uf .../all-packages.nix hello

I If it doesn’t work: $ nix-env --rollback

I Delete unused components:

$ nix-collect-garbage User operations

I To build and install Hello: $ nix-env -if .../all-packages.nix hello

I When a new version comes along:

$ nix-env -uf .../all-packages.nix hello

I If it doesn’t work: $ nix-env --rollback

I Delete unused components:

$ nix-collect-garbage User operations

I To build and install Hello: $ nix-env -if .../all-packages.nix hello

I When a new version comes along:

$ nix-env -uf .../all-packages.nix hello

I If it doesn’t work: $ nix-env --rollback

I Delete unused components:

$ nix-collect-garbage User environments

/nix/store PATH eb3266df5c1a...-user-env bin Users can have firefox I /nix/links different sets of hello installed applications. current 90d0d25ee157...-hello-2.1.1 bin I nix-env operations 42 hello create new user e7cb4be9e7c4...-firefox-1.0 environments in the bin store. firefox

I We can atomically switch between them.

I These are roots of the garbage collector. User environments

/nix/store PATH eb3266df5c1a...-user-env bin Users can have firefox I /nix/links different sets of hello installed applications. current 90d0d25ee157...-hello-2.1.1 bin I nix-env operations 42 hello create new user e7cb4be9e7c4...-firefox-1.0 environments in the bin store. firefox d0b3495e5c73...-hello-2.1.2 I We can atomically bin switch between them. hello

I These are roots of the garbage collector.

(nix-env -u hello) User environments

/nix/store PATH eb3266df5c1a...-user-env bin Users can have firefox I /nix/links different sets of hello installed applications. current 90d0d25ee157...-hello-2.1.1 bin I nix-env operations 42 hello create new user e7cb4be9e7c4...-firefox-1.0 environments in the bin store. firefox d0b3495e5c73...-hello-2.1.2 I We can atomically bin switch between them. hello These are roots of the 83ca061e32cd...-user-env I bin garbage collector. hello firefox (nix-env -u hello) User environments

/nix/store PATH eb3266df5c1a...-user-env bin Users can have firefox I /nix/links different sets of hello installed applications. current 90d0d25ee157...-hello-2.1.1 bin I nix-env operations 42 hello create new user e7cb4be9e7c4...-firefox-1.0 environments in the 43 bin store. firefox d0b3495e5c73...-hello-2.1.2 I We can atomically bin switch between them. hello These are roots of the 83ca061e32cd...-user-env I bin garbage collector. hello firefox (nix-env -u hello) User environments

/nix/store PATH eb3266df5c1a...-user-env bin Users can have firefox I /nix/links different sets of hello installed applications. current 90d0d25ee157...-hello-2.1.1 bin I nix-env operations 42 hello create new user e7cb4be9e7c4...-firefox-1.0 environments in the 43 bin store. firefox d0b3495e5c73...-hello-2.1.2 I We can atomically bin switch between them. hello These are roots of the 83ca061e32cd...-user-env I bin garbage collector. hello firefox (nix-env -u hello) User environments

/nix/store PATH eb3266df5c1a...-user-env bin Users can have firefox I /nix/links different sets of hello installed applications. current 90d0d25ee157...-hello-2.1.1 bin I nix-env operations hello create new user e7cb4be9e7c4...-firefox-1.0 environments in the 43 bin store. firefox d0b3495e5c73...-hello-2.1.2 I We can atomically bin switch between them. hello These are roots of the 83ca061e32cd...-user-env I bin garbage collector. hello firefox (nix-env –remove-generations old) User environments

/nix/store PATH

Users can have I /nix/links diﬀerent sets of installed applications. current

I nix-env operations create new user e7cb4be9e7c4...-firefox-1.0 environments in the 43 bin store. firefox d0b3495e5c73...-hello-2.1.2 I We can atomically bin switch between them. hello These are roots of the 83ca061e32cd...-user-env I bin garbage collector. hello firefox (nix-collect-garbage) Deployment using Nix

I This is conceptually a source deployment model.

I We get binary deployment by sharing pre-built components.

I On the producer side:

$ nix-push $(nix-instantiate .../all-packages.nix) \ http://server/cache

I On the client side: $ nix-pull http://server/cache $ nix-env -if .../all-packages.nix hello

I Installation will now reuse pre-built components, iﬀ they are exactly the same. Deployment using Nix

I This is conceptually a source deployment model.

I We get binary deployment by sharing pre-built components.

I On the producer side:

$ nix-push $(nix-instantiate .../all-packages.nix) \ http://server/cache

I On the client side: $ nix-pull http://server/cache $ nix-env -if .../all-packages.nix hello

I Installation will now reuse pre-built components, iﬀ they are exactly the same. Deployment using Nix

I This is conceptually a source deployment model.

I We get binary deployment by sharing pre-built components.

I On the producer side:

$ nix-push $(nix-instantiate .../all-packages.nix) \ http://server/cache

I On the client side: $ nix-pull http://server/cache $ nix-env -if .../all-packages.nix hello

I Installation will now reuse pre-built components, iﬀ they are exactly the same. An example deployment policy: channels

I Channels allow Nix expressions to be updated automatically.

I Subscribe to a channel: $ nix-channel --add http://.../channels/nixpkgs-stable

I Fetch latest channel instance: $ nix-channel --update

I Update all installed packages:

$ nix-env -u ’*’ An example deployment policy: channels

I Channels allow Nix expressions to be updated automatically.

I Subscribe to a channel: $ nix-channel --add http://.../channels/nixpkgs-stable

I Fetch latest channel instance: $ nix-channel --update

I Update all installed packages:

$ nix-env -u ’*’ An example deployment policy: channels

I Channels allow Nix expressions to be updated automatically.

I Subscribe to a channel: $ nix-channel --add http://.../channels/nixpkgs-stable

I Fetch latest channel instance: $ nix-channel --update

I Update all installed packages:

$ nix-env -u ’*’ Service deployment

Services: sets of running programs that provide some useful facility on a system or network. Example: Subversion service Example: Issue tracking service Service deployment is hard

Service deployment involves a number of steps:

I Deploy software components (e.g., Apache, PostgreSQL, Subversion)

I Edit conﬁguration ﬁles (e.g., httpd.conf, viewcvs.conf)

I Initialise state (e.g., logging directories, database tables)

I Start/stop processes

I ... and all of this possibly on multiple machines / platforms Problems

I Poor reproducibility (bad CM)

I Hard to support parallel conﬁgurations

I Cross-cutting conﬁguration choices Problem 1: Poor reproducibility

I Goal: it should be possible to realise a service by running a single command.

I E.g., to move it to another machine I So no manual installing of missing software components, tweaking of conﬁguration ﬁles, creating missing directories, etc.

I Why is reproducibility hard?

I Admins often manually edit conﬁguration ﬁles and initialise state

I Service conﬁguration doesn’t express software component dependencies Example

configuration data

control /data/subversion/

httpd.conf

software

viewcvs

subversion perlBerkeleyDB

perl apache

expat openssl db4 Gap between package management and service conﬁguration

I Software components are typically deployed through package managers such as RPM

I Service conﬁguration is typically kept under version management

I However, there is no good way to express the dependencies of the service on the software components Problem 2: Parallel conﬁgurations

I It should be easy to create diﬀerent instances of a service

I Test vs. production servers (running on diﬀerent ports, using diﬀerent databases, etc.)

I Instantiations for diﬀerent users I Evolution through time (rollbacks)

I This is hard to support because there are typically lots of configuration files and control scripts that refer to lots of paths for components, state, static data files, etc.

I /etc/apache/httpd.conf, /etc/init.d/apache, /etc/apache/viewcvs.conf, ... Example

/etc/apache/httpd.conf for Subversion service (fragment) ServerRoot "/var/httpd" ServerName svn.cs.uu.nl:8080 LoadModule dav_svn_module /usr/lib/modules/mod_dav_svn.so AuthType Basic AuthDBMUserFile /data/subversion/db/svn-users ... SVNParentPath /data/subversion/repos ScriptAlias /viewcvs /usr/viewcvs/www/cgi/viewcvs.cgi Example

/etc/init.d/httpd for Subversion service (fragment) /usr/sbin/apachectl -k start -f /etc/apache/httpd.conf

Use cases

I Try out with a diﬀerent set of repositories.

I Try out a diﬀerent Apache.

I Try out a diﬀerent Subversion module. Problem 3: Cross-cutting conﬁguration choices

I Many configuration choices are cross-cutting, i.e., impact many different (parts of) configuration files, scripts, etc.

I Examples:

I Port numbers I Host names I Paths (major source of problems!)

I So a change to the conﬁguration choices must be realised in many diﬀerent places

I Lots of work

I Danger of inconsistency Example: port number

In /etc/init.d/httpd.conf ServerName www.example.org:12443 Listen 12443

In repoman.pl my $url = "https://www.example.org:12443/" print "... ..."; Solution

Treat all the static parts of conﬁgurations as Nix components:

I Conﬁguration ﬁles

I Control scripts

I Static data ﬁles (e.g., static web pages) But not mutable state, e.g.,

I Databases Continuous Integration and Release Management

I Building releases of components automatically involves many steps:

I Prepare the build environment(s) I Make sure that all tests succeed I Build a source distribution I Build binary distributions for a variety of platforms I Upload (publish) to a server I Update client machines

I ⇒ Requires a build farm. Build farm

Set of machines that automatically performs build actions from a version management repository. Nix is very useful for implementing a build farm:

I The Nix expression language is ideal for describing the build tasks.

I The Nix expression language makes it easy to describe variants.

I Nix manages the dependencies.

I Complete dependencies, thus reproducibility.

I Eﬃciency: only rebuild things that have actually changed.

I Builds can be made available through a channel. Example Example Example Example Conclusions

I Contributions: I Safe, automatic coexistance of versions/variants. I Reliable dependencies. I Multiple concurrent conﬁgurations. I Atomic upgrades/rollbacks. I Safe garbage collection. I Binary deployment is automatic. I Can accomodate many deployment policies. I Useful for service deployment. I Integrated continuous integration / release management.

I Available at http://www.cs.uu.nl/groups/ST/Trace/Nix. Further reading

ICSE’04 E. Dolstra, E. Visser, and M. de Jonge, Imposing a Memory Management Discipline on Software Deployment LISA’04 E. Dolstra, M. de Jonge, and E. Visser, Nix: A Safe and Policy-Free System for Software Deployment CBSE’05 E. Dolstra, Eﬃcient Upgrading in a Purely Functional Component Deployment Model ASE’05 E. Dolstra, Secure Sharing Between Untrusted Users in a Transparent Source/Binary Deployment Model PhD thesis E.Dolstra, The Purely Functional Software Deployment Model