The Purely Functional Software Deployment Model Het puur functionele softwaredeploymentmodel (met een samenvatting in het Nederlands) Proefschrift ter verkrijging van de graad van doctor aan de Universiteit Utrecht op gezag van de Rector Magnificus, Prof. dr. W. H. Gispen, ingevolge het besluit van het College voor Promoties in het openbaar te verdedigen op woensdag 18 januari 2006 des middags te 12.45 uur door Eelco Dolstra geboren op 18 augustus 1978, te Wageningen Promotor: Prof. dr. S. Doaitse Swierstra, Universiteit Utrecht Copromotor: Dr. Eelco Visser, Universiteit Utrecht The work in this thesis has been carried out under the auspices of the research school IPA (Institute for Programming research and Algorithmics). Dit proefschrift werd mogelijk gemaakt met financiële steun van CIBIT|SERC ICT Adviseurs. Cover illustration: Arthur Rackham (1867–1939), The Rhinegold & The Valkyrie (1910), illustrations for Richard Wagner’s Der Ring des Nibelungen: the gods enter Walhalla as the Rhinemaidens lament the loss of their gold. ISBN 90-393-4130-3 Acknowledgements This thesis could not have come about without the help of many individuals. Foremost I wish to thank my supervisor and copromotor Eelco Visser. He was also my master’s thesis supervisor, and I was quite happy that we were able to continue to work together on the Variability/TraCE project. He shared my eventual interest in configuration management- related issues, and package management in particular. His insights have improved this research and this thesis at every point. The Software Engineering Research Center (SERC), now known as CIBIT|SERC ICT Adviseurs, funded my PhD position. Gert Florijn initiated the variability project and the discussions with him were a valuable source of insights. The results of the present thesis are probably not what any of us had expected at the start; but then again, the nice thing about a term like “variability” is that it can take you in so many directions. My promotor Doaitse Swierstra gave lots of good advice—but I ended up not imple- menting his advice to keep this thesis short. I am grateful to the members of the reading committee—Jörgen van den Berg, Sjaak Brinkkemper, Paul Klint, Alexander Wolf and Andreas Zeller—for taking the time and effort to go through this book. Karl Trygve Kalle- berg, Armijn Hemel, Arthur van Dam and Martin Bravenboer gave helpful comments. Several people have made important contributions to Nix. Notorious workaholic Martin Bravenboer in particular was an early adopter who contributed to almost every aspect of the system. Armijn Hemel (“I am the itch programmers like to scratch”) contributed to Nixpkgs and initiated the NixOS, which will surely conquer the world. René de Groot coined the marketing slogan for NixOS—“Nix moet, alles kan!” Roy van den Broek has replaced my hacky build farm supervisor with something much nicer (and Web 2.0 compli- ant!). Eelco Visser, Rob Vermaas and Merijn de Jonge also contributed to Nixpkgs and the build farm. In addition, it is hard to overestimate the importance of the work done by the free and open source software community in providing a large body of non-trivial modular components suitable for validation. Thanks! The Software Technology group is a very pleasant work environment, and I thank all my current and former colleagues for that. I especially thank my roommates Dave Clarke (“The cause of the software crisis is software”), Frank Atanassow (who demanded “pointable research”), Merijn de Jonge, Martin Bravenboer, Rob Vermaas and Stefan Hold- ermans. Andres Löh, Bastiaan Heeren, Arthur Baars, Karina Olmos and Alexey Rodriguez were more than just good colleagues. Daan Leijen insisted on doing things the right way. The #klaplopers provided a nice work environment in virtual space—though whether IRC is a medium that increases productivity remains an unanswered empirical question. Arthur van Dam and Piet van Oostrum provided valuable assistance in wrestling with TEX. Atze Dijkstra, Bastiaan Heeren and Andres Löh had helpful advice (and code!) for the preparation of this thesis. And finally I would like to thank my family—Mom, Dad, Jitske and Menno—for their support and love through all these years. iii iv Contents I. Introduction 1 1. Introduction 3 1.1. Software deployment . 3 1.2. The state of the art . 6 1.3. Motivation . 13 1.4. The Nix deployment system . 14 1.5. Contributions . 14 1.6. Outline of this thesis . 16 1.7. Notational conventions . 17 2. An Overview of Nix 19 2.1. The Nix store . 19 2.2. Nix expressions . 25 2.3. Package management . 34 2.4. Store derivations . 39 2.5. Deployment models . 42 2.6. Transparent source/binary deployment . 44 II. Foundations 47 3. Deployment as Memory Management 49 3.1. What is a component? . 49 3.2. The file system as memory . 52 3.3. Closures . 55 3.4. A pointer discipline . 56 3.5. Persistence . 59 4. The Nix Expression Language 61 4.1. Motivation . 61 4.2. Syntax . 64 4.2.1. Lexical syntax . 66 4.2.2. Context-free syntax . 67 4.3. Semantics . 71 4.3.1. Basic values . 71 4.3.2. Compound values . 73 v Contents 4.3.3. Substitutions . 75 4.3.4. Evaluation rules . 76 4.4. Implementation . 81 5. The Extensional Model 87 5.1. Cryptographic hashes . 87 5.2. The Nix store . 90 5.2.1. File system objects . 90 5.2.2. Store paths . 92 5.2.3. Path validity and the closure invariant . 95 5.3. Atoms . 97 5.4. Translating Nix expressions to store derivations . 100 5.4.1. Fixed-output derivations . 106 5.5. Building store derivations . 108 5.5.1. The simple build algorithm . 109 5.5.2. Distributed builds . 115 5.5.3. Substitutes . 118 5.5.4. The parallel build algorithm . 121 5.6. Garbage collection . 124 5.6.1. Garbage collection roots . 125 5.6.2. Live paths . 127 5.6.3. Stop-the-world garbage collection . 128 5.6.4. Concurrent garbage collection . 131 5.7. Extensionality . 134 6. The Intensional Model 135 6.1. Sharing . 135 6.2. Local sharing . 139 6.3. A content-addressable store . 140 6.3.1. The hash invariant . 141 6.3.2. Hash rewriting . 143 6.4. Semantics . 145 6.4.1. Equivalence class collisions . 147 6.4.2. Adding store objects . 153 6.4.3. Building store derivations . 155 6.4.4. Substitutes . 157 6.5. Trust relations . 159 6.6. Related work . 159 6.7. Multiple outputs . 160 6.8. Assumptions about components . 162 III. Applications 165 vi Contents 7. Software Deployment 167 7.1. The Nix Packages collection . 167 7.1.1. Principles . 170 7.1.2. The standard environment . 174 7.1.3. Ensuring purity . 179 7.1.4. Supporting third-party binary components . 180 7.1.5. Experience . 181 7.2. User environments . 184 7.3. Binary deployment . 185 7.4. Deployment policies . 187 7.5. Patch deployment . 190 7.5.1. Binary patch creation . 193 7.5.2. Patch chaining . 194 7.5.3. Base selection . 196 7.5.4. Experience . 198 7.6. Related work . 200 8. Continuous Integration and Release Management 209 8.1. Motivation . 209 8.2. The Nix build farm . 212 8.3. Distributed builds . 217 8.4. Discussion and related work . 218 9. Service Deployment 221 9.1. Overview . 222 9.1.1. Service components . ..