Code Staging in GNU Guix Ludovic Courtès Inria Bordeaux, France Abstract means that software build processes are considered as pure GNU Guix is a “functional” package manager that builds functions: given a set of inputs (compiler, libraries, build upon earlier work on Nix. Guix implements high-level ab- scripts, and so on), a package’s build function is assumed stractions such as packages and operating system services to always produce the same result. Build results are stored as domain-specific languages (DSLs) embedded in Scheme. in an immutable persistent data structure, the store, imple- It also implements build actions and operating system or- mented as a single directory, /gnu/store. Each entry in /- chestration in Scheme. This leads to a multi-tier program- gnu/store has a file name composed of the hash of all the ming environment where embedded code snippets are staged build inputs used to produce it, followed by a symbolic name. for eventual execution. For example, /gnu/store/yr9rk90jf...-gcc-7.1.0identi- This paper presents G-expressions or “gexps”, the staging fies a specific build of GCC 7.1. A variant of GCC 7.1, for mechanism we devised for Guix. We explain our journey instance one using different build options or different de- from traditional Lisp S-expressions to G-expressions, which pendencies, would get a different hash. Thus, each store file augment the former with contextual information and en- name uniquely identifies build results, and build processes sure hygienic code staging. We discuss the implementation are referentially transparent. This simplifies reasoning on of gexps and report on our experience using them in a vari- complex package compositions, and also has nice properties ety of operating system use cases—from package build pro- such as supporting transactional upgrades and rollback “for cesses to system services. Gexps provide a novel way to free.” The Guix System Distribution (or GuixSD) and NixOS cover many aspects of OS configuration in a single, multi- extend the functional paradigm to whole operating system tier language, while facilitating code reuse and code shar- deployments [6]. ing. Guix implements this functional deployment paradigm pioneered by Nix but, as explained in previous work, its CCS Concepts • Software and its engineering → Source implementation departs from Nix in interesting ways [3]. code generation; Functional languages; System administra- First, while Nix relies on a custom domain-specific language tion; (DSL), the Nix language, Guix instead implements a set of Keywords Code staging, Scheme, Software deployment DSLs and data structures embedded in the general-purpose language Scheme. This simplifies the development of user ACM Reference Format: interfaces and tools, and allows users to benefit from every- Ludovic Courtès. 2017. Code Staging in GNU Guix. In Proceedings thing a general-purpose language brings: compiler, debug- of 16th ACM SIGPLAN International Conference on Generative Pro- gramming: Concepts and Experiences (GPCE’17). ACM, New York, ger, REPL, editor support, libraries, and so on. NY, USA, 9 pages. hps://doi.org/10.1145/3136040.3136045 (define hello (package arXiv:1709.00833v1 [cs.PL] 4 Sep 2017 1 Introduction (name "hello") Users of free operating systems such as GNU/Linux are used (version "2.8") (source (origin to package managers like Debian’s apt-get, which allow (method url-fetch) them to install, upgrade, and remove software from a large (uri (string-append "mirror://gnu/hello/hello-" 1 version ".tar.gz")) collection of free software packages. GNU Guix is a func- (sha256 (base32 "0wqd8...")))) tional package manager that builds upon the ideas devel- (build-system gnu-build-system) (arguments oped for Nix by Dolstra et al. [5]. The term “functional” ’(#:configure-flags ‘("--disable-color" 1https://gnu.org/software/guix ,(string-append "--with-gawk=" (assoc-ref %build-inputs "gawk"))))) Copyright ©2017 Ludovic Courtès. (inputs ‘(("gawk" ,gawk))) (synopsis "GNU Hello") Permission is granted to copy, distribute and/or modify this document un- (description "Example of a GNU package.") der the terms of the GNU Free Documentation License, Version 1.3 or any (home-page "https://gnu.org/software/hello/") later version published by the Free Software Foundation; with no Invari- (license gpl3+))) ant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is available at hps://www.gnu.org/licenses/gfdl.html. Figure 1. A package definition using the high-level inter- The source of this document is available from face. hps://git.sv.gnu.org/cgit/guix/maintenance.git. GPCE’17, October 23–24, 2017, Vancouver, Canada Ludovic Courtès (let* ((store (open-connection)) In Guix, high-level package definitions like the one shown (drv (package-derivation store imagemagick)) in Figure 1 are compiled to derivations, the low-level repre- (image (add-to-store store "image.png" #t "sha256" "./GuixSD.png")) sentation of build actions inherited from Nix. A derivation (build specifies: a command to run to perform the build (the build ’(let ((imagemagick (assoc-ref %build-inputs "imagemagick")) program), environment variables to be defined, and deriva- (image (assoc-ref %build-inputs "image"))) tions whose build result it depends on. Derivations are sent (mkdir %output) (system* (string-append imagemagick "/bin/convert") to a privileged daemon, which is responsible for building "-quality" "75%" them on behalf of clients. The build daemon creates isolated image (string-append %output "/image.jpg"))))) environments (containers in a chroot) in which it spawns (build-expression->derivation store "example" build the build program; isolated build environments ensure that #:inputs ‘(("imagemagick" ,drv) build programs do not depend on undeclared inputs. ("image" ,image)))) The second way in which Guix departs from Nix is by Figure 2. First attempt: build expressions as sexps. using the same language, Scheme, for all its functionality. While package definitions in Nix can embed Bash or Perl snippets to refine build steps, Guix package definitions in- the file GuixSD.png to /gnu/store.The variable build con- stead embed Scheme code. Consequently, we have two strata tains our build program as an sexp (the apostrophe is equiva- of Scheme code: the host code, which provides the package lent to quote; it introduces unevaluated code). Finally, build-- definition, and the build code, which is staged for later exe- expression->derivationtakesthebuild programandcom- cution by the build daemon. putes the corresponding derivation without building it. The This paper focuses on code staging in Guix. Our contribu- user can then make an RPC to the daemon asking it to build tion is twofold: we present G-expressions (or “gexps”), a new this derivation; only then will the daemon create an isolated code staging mechanism implemented through mere syntac- environment and run our build program. tic extensions of the Scheme language; we show the use of build-expression->derivation arranges so that the gexps in several areas of the “orchestration” programs of the build program, build, is evaluated in a context where two operating system. Section 2 discusses the early attempt at extra variables are defined: %build-inputs contains a list code staging in Guix, as mentioned in [3], and its shortcom- that maps labels, such as "imagemagick",to file names, such ings. Section 3 presents the design and implementation of as /gnu/store/...-imagemagick-6.9,and %output contains gexps. Section 4 reports on our experience using gexps in a the file name for the result of this derivation. Thus, the assoc-- variety of areas in Guix and GuixSD. Section 5 discusses lim- ref calls in build allow it to retrieve the file name of Im- itations and future work. Finally Section 6 compares gexps ageMagick. to related work and Section 7 concludes. 2.2 S-Expressions Are Not Enough Needless to say, this interface was extremely verbose and 2 Early Attempt inconvenient—fortunately, users usually only had to deal Scheme is a dialect of Lisp, and Lisp is famous for its its with the more pleasant package interface shown in Figure direct representation of code as a data structure using the 1. All these steps are necessary to define the derivation and same syntax. “S-expressions” or “sexps”, Lisp’s parenthecal its dependencies, but where does the verbosity come from? expressions, thus look like they lend themselves to code stag- First, we have to explicitly call package-derivation for ing. In this section we show how our early experience made each package the expression refers to. Second, we have to it clear that we needed an augmented version of sexps. specify the inputs with labels at the call site. Third, the build code has to use this assoc-ref call just to retrieve the /- gnu/store 2.1 Staging Build Expressions file name of its inputs. It is error-prone: if we omit the #:inputs parameter, of if we mispell an input la- In previous work [3], we presented our first attempt at writ- bel, we will only find out when we build the derivation. ing build expressions in Scheme, which relied solely on Lisp Another limitation not visible on a toy example but that quotation [2]. Figure 2 shows an example that creates a deriva- became clear as we developed GuixSD is the cost of carry- tion that, when built, converts the input image to JPEG, us- ing this #:inputs argument down to the call site. It forces ing the convertprogram from the ImageMagickpackage—this programmers to carry not only the build expression, build, is equivalent to a three-line makefile rule, but referentially but also the corresponding inputs argument, and makes it transparent. In this example, variable store represents the very hard to compose build expressions. connection to the build daemon. The package-derivation While quote allowed us to easily represent code, it clearly function takes the imagemagick package object and com- lacked some of the machinery that would make staging in putes its corresponding derivation, while the add-to-store Guix more convenient.