arXiv:1610.02742v1 [cs.DC] 10 Oct 2016 u otepriuaiiso ahHCevrnet tis it Fortuna environment, [ packages. software Gentoo’s abilit HPC specific the to offer Finally patches each to apply process. manager easily of package the the for particularities in desirable highly the packages of to rebuilds tree package without due full post change, force the dependencies and to of rebuilding time, able options be run configuration also be time, must when and It build dependencies, correct. as between is build package distinguish order build each to the for able that account options ensure into as configuration take well recent must of orde manager multitude In in package the reasons. the features users, consistency new satisfy or to of compatibility sof use of for versions releases specific packages make against upstream build to or latest releases, vendors software want the HPC users behind from while systems significantly operating lag core often the by Packag provided tools. packaging powerful requires environment HPC omng ag otaesak nHCenvironments. HPC in stacks software large manage to .Gno Linux Gentoo A. paths. non-standard into rmFeBD etoi o ae nayLnxdistribution, Linux any on manager, based not package is own Gentoo FreeBSD. its from uses customize own Gentoo [ his[her] Portage provides Gentoo. build it to of that user sense version the the for since in tools development metadistribution, necessary active a in is been It has then. i and Robbins Daniel 1999, founder of its December by environme released development first been excellent has an Gentoo it make that tools erful o aepriso oisalsfwr nosadr locat standard do [ into usually Prefix software users Gentoo install where to systems permission HPC have On not more. much and this rfi ntlaincnb sdt rs opl otaepac Gentoo software compile a Intel cross how the to for demonstrate used be we can Here a installation Prefix systems. Gentoo offers of operating Gen virtues The other the installing that to bringing task. of locations, tool way non-standard daunting a in flexible maintains systems otherwise and highly develops this project a Prefix to package Large is solution Portage task. Gentoo mature The challenging tools. from re a management manage, manager to is package difficult powerful packages are source ing software from built date stacks software to up of uligadmitiigalresfwr tc nan on stack software large a maintaining and Building etoLnx[ Linux Gentoo Abstract otg:Bign akr’Wso oScience to Wisdom Hackers’ Bringing Portage: 2 Poiigueso P ytm ihawd variety wide a with systems HPC of users —Providing ,witni yhnadisie yteprssystem ports the by inspired and Python in written ], 1

otg [ Portage ] R enPhi Xeon 3 losisaltoso etoenvironment Gentoo a of installations allows ] 1 sasuc-ae itiuinwt pow- with distribution source-based a is ] ˜ oPuoSaeUiest,Brazil University, State Paulo S˜ao .I I. 2 TM akg aae sal od all do to able is manager package ] NTRODUCTION nw sKihsCre,a elas well as Corner, Knights as known ulem Amadio Guilherme [email protected] etoLinux Gentoo Gentoo tware kages quir- ions, tely, to y the too nt. es n d r , , locmieisonsadr irr [ library C standard can own durin Prefix Gentoo its managed Linux, compile and On installed system. also the are of tools bootstrapping the other all but library, and package. Gentoo each of in building, options configuration available code, includi handled, automatically source packages dependencies their manager fetching software having package of installing Portage process privi- and Gentoo’s the administrator use automate have to to not simply i might or software user installing leges, the for where useful is systems This systems. different operating management—to tuning dependency performance package automated configurability, and high as Linux—such too .Gno rfi Project Prefix Gentoo B. [ ChromeOS Google’s and reS’ ot olcin otg ossso w anpa main two the of consists Portage collection. ports FreeBSD’s Manager Package MacPorts Portage Homebrew, C. as Gentoo based). X, such (binary OS Fink projects Mac and to On based) (source etc. on similar also Solaris, is but AIX, database. Linux, Prefix BSD, on Portage install X, only and not the OS installed Mac compile in be can to available th Prefix user install Gentoo packages and the sofware compiler allows temporary a other which a bootstrap system, to into it base manager uses script package shell then Portage a location, the by performed installs is that environment Prefix Gentoo the opl,adisalsfwr,a ela hi dependency their tha more as configure, has well repository download, ebuild as main to software, Gentoo’s requirements. how install SRPMs; in on and files compile, spec information to contain similar are Ebuilds they them). run to used Debian. on apt and The of ebuilds dpkg of that ebuild. that to and Hat, to similar Red is on interface yum emerge and the and rpm ebuild as between dependency servers relationship provides emerge and instructi while build management, packages, executing for installing responsible and is system ebuild u sisl h oto oua itiuin ieSabayon like [ CoreOS distributions as popular such systems of operating root innovative and the Linux, itself is but etoPexue h otsse’ enladsadr C standard and kernel system’s host the uses Prefix Gentoo otg saGL2pcaemngmn ytmbsdon based system management package GPLv2 a is Portage [ project Prefix Gentoo The otg akgsaeseilbs hl cit called scripts shell bash special are packages Portage ebuild ntt ecnue ihteeul olisl htis that itself tool ebuild the with confused be to (not h nvriyo oy,Japan Tokyo, of University The ytm n the and system, [email protected] etoLinux Gentoo ed Xu Benda 5 emerge ]. 3 rnstevruso Gen- of virtues the brings ] omn ieuiiy The utility. line command 6 .Isalto of Installation ]. gthe ng ons nto rts, 4 n g e ] , 27000 ebuilds available for a variety of architectures. Addi- The bootstrap process consists roughly of three stages: tional packages are available via official and unofficial package installing a temporary version of Portage, boostrapping a overlays that complement the main tree of packages. The compiler using the host’s toolchain, then bootstrapping a full special syntax used in ebuild scripts is standardized in the system with the bootstrapped compiler. On Linux, a C standard Package Manager Specification (PMS) [7] document. The library can also be part of the bootstrap process, in which PMS documents the behavior of Portage so that Gentoo case the kernel is the only piece shared between the host and packages can be managed by alternative package managers, the prefix environment. To have the C standard library from such as paludis [8] and pkgcore [9]. Gentoo, rename the script before invoking it. The Portage system has the concept of USE flags to allow $ mv bootstrap-prefix.sh bootstrap-rap.sh the user to configure compile-time options of software pack- $ ./bootstrap-rap.sh ages. USE flags affect which dependencies are required to (follow the instructions) build a package, which allows, for example, a headless server to be installed with a lighter system footprint, by stripping all The script works as an interactive installer; it will ask options for building a graphical environment. USE flags can the user for some information and discuss options based on be used in HPC systems to enable or disable MPI support, the system being bootstrapped on. Once the script finishes to choose which version of the Python interpreter a module running, it will create a startprefix script at the root of should target, to enable specific instruction sets for a given the bootstrapped system. Running that script brings the user architecture, among many others. into the prefix environment, where emerge and other tools For software packages that require specific versions, Portage become available to let the user install more packages. has the concept of package SLOTs [7, p. 27], which allow mul- $ ${EPREFIX}/startprefix tiple versions of a same package to be installed simultaneously Entering Gentoo Prefix /micfs/gentoo on the same system. This is very useful for, e.g., installing $ packages that depend on incompatible versions of the same library, or to have multiple compiler versions available for The following sections present several case studies to show development and testing. SLOTs can also be used in package the flexibility of Gentoo Prefix. dependencies to indicate when a package should be rebuilt A. Nested Job-Scheduling with Slurm should the installed version of its dependencies change. Although HPC programs are typically parallel, for some embarrassingly parallel tasks like data analysis and simulation, II. GENTOO PREFIX ON AN HPC ENVIRONMENT it is enough to have a grid infrastructure running many single- As Gentoo Prefix runs on all the GNU/Linux Distributions, threaded workers divided by input data. it supports GNU/Linux-based HPC systems off-the-shelf. It is Some supercomputing site allocates resources in units of also reported that Gentoo Prefix runs on Cray with some minor nodes and leaves finer scheduling within nodes to users. For site-specific modifications [10]. MPI-based parallel programs it is straight-forward, but for Gentoo Prefix uses ebuilds from Gentoo’s main repos- single-threaded programs it is tricky to efficiently and reliably itory (Section I-C) and receives cutting edge updates by use all the CPUs in an allocated node. hundreds of developers and thousands of user contributors. Instead of writing ad hoc shell scripts to launch multiple It is a general-purpose and full-featured environment, which single-threaded programs, slurm or torque available in Gentoo mitigates a big portion of the complexity and workload of could be used as a nested scheduler and allocate jobs in units of CPUs. HPC software maintenance. LATEX and pandoc for publication, emacs and vim for editing, vnc and xpra for remote display, to name a few, are all available. From the users’ point of view, Gentoo Prefix vastly expands the tools available for HPC. From the administrators’ point of view, Gentoo Prefix supports a diversity of user needs and at the same time keeps the core operating system stable and secure. Installation instructions for Gentoo Prefix can be found on the project website [3]. The bootstrap-prefix.sh script automates most of the process ($ shown below is the normal Fig. 1. User slurm is nested in site PBS Pro. The nodes are allocated by PBS user command line prompt). and each slurm worker runs on each node. Single-threaded jobs are controlled and queued by user slurm controller. Resources within a node, such as CPUs (download script from project website) and memory, are individually allocated. $ chmod 755 bootstrap-prefix.sh $ ./bootstrap-prefix.sh Assuming that the host scheduler is PBS Professional and (follow the instructions) has been installed by the site administrators, a nested slurm scheduler (Fig. 1) can be installed by a normal user with $ layman --add=kde Gentoo Prefix on a filesystem shared between all nodes. $ emerge kde-misc/massif-visualizer $ emerge sys-cluster/slurm and of course in addition to valgrind After editing EPREFIX/etc/slurm/slurm.conf to $ emerge dev-util/valgrind enable individual allocation of resources within a node, the To resolve the function names in the stack in valgrind slurm worker can be launched by slurmd and the slurm output, debug symbols are needed. A neat feature is controller by slurmctld. EPREFIX is the installation offset provided by Portage to control build environment in a of Gentoo Prefix. per package-basis. For example, to get memory profile It is recommended to use munge for authentication between of a quantile regression package quantreg of GNU R, slurmd and slurmctld. OpenRC, a dependency aware debug symbols of glibc, R and quantreg are needed. In service manager, is suitable to manage the relation between EPREFIX/etc/portage/env/debug-cflags.conf, munge and slurm daemons. We create named runlevels for add the slurm worker and controller, CFLAGS="-O2 -ggdb -pipe" $ mkdir EPREFIX/etc/runlevels/worker CXXFLAGS="${CFLAGS}" $ mkdir EPREFIX/etc/runlevels/controller FEATURES="${FEATURES} splitdebug" add services to the corresponding runlevels, and in EPREFIX/etc/portage/package.env, add $ rc-update add munged controller worker sci-CRAN/quantreg debug-cflags.conf $ rc-update add slurmctld controller dev-lang/R debug-cflags.conf $ rc-update add slurmd controller sys-libs/glibc debug-cflags.conf and start slurmctld on the controller node (or login node) with and rebuild the packages, $ openrc controller $ emerge sys-libs/glibc dev-lang/R \ * Starting munged ... > sci-CRAN/quantreg * Starting slurm control daemon ... To produce and view the memory profile, Then, we submit a PBS batch script to launch slurm workers, R -d "valgrind --tool=massif \ $ cat slurmd.sh > --massif-out-file=rq.prof" -f rq.R #!/bin/sh $ massif-visualizer rq.prof #PBS -l select=32:ncpus=36:mpiprocs=1 #PBS -l walltime=48:00:00 where rq.R is the R script to be profiled, its content being source EPREFIX/etc/profile irrelavent to this discussion. mpirun sh -c "openrc worker; sleep 2d" C. Cross Compiling Software for the Xeon PhiTM $ qsub slurmd.sh Portage can also be configured to cross compile software In the script above, 32 nodes, each having 36 CPUs, are for different platforms. The usual way of doing so is to follow requested for 2 days. mpirun is used to launch 1 slurm the Gentoo Embedded Handbook [13] to set up the appropriate worker per node. GCC toolchain and setup the right Portage profile. However, After the workers are started, single-threaded jobs can be cross compiling for the Xeon PhiTM is somewhat different, as submitted with slurm tools. only the Intel R C/C++ Compiler has full support for the Xeon PhiTM’s IMCI (Initial ManyCore Instructions) architecture, B. Memory Profile with Valgrind and massif-visualizer which, despite being quite similar to the x86 architecture, In the context of HPC, large scale problems could be- has some subtle differences. The fact that Knights Corner come memory-bounded, for example when fill-in happens to Xeon PhiTM cards are used as coprocessors and their boot a sparse matrix based algorithm. Valgrind is traditionally a process does not follow a conventional procedure is irrelevant memory debugging tool for memory leak detection. A heap for cross compiling. Once the Xeon PhiTM is properly set up, profiler, massif [11], is also distributed with valgrind. massif- it basically works as a headless server; users can just ssh visualizer [12] is a handful GUI visualizer for interpreting into it and run their programs. In our setup, a server with massif outputs besides the default CLI-based ms-print. CentOS 7 and Intel tools installed was used to host the Gentoo massif-visualizer is a KDE application. It depends on a Prefix installation. The host system also needs to have both recent KDE framework, which is a fairly large and complex the Intel R C/C++ compiler and Intel R ’s Manycore Platform software stack. Gentoo Prefix readily provides the whole KDE Software Stack (MPSS) in order to fulfill the requirements for stack thanks to the shared ebuilds with the main Gentoo cross compiling. repository. As massif-visualizer is already in the KDE overlay, Since Python is a popular language for scientific computing building it is straightforward, in general, we demonstrate how to cross compile Python 2.7 for the Xeon PhiTM. The first step was to create a root directory into the subdirectory dev-libs/libffi-3.2.1 under for Portage, where all the configuration files reside, and where /micfs/gentoo-mic/etc/portage/patches. If the all software packages are installed. version is ommitted, the patch will apply to all versions. In our system, we used a separate partition (/micfs) After everything has been properly configured as shown to hold the Gentoo Prefix (/micfs/gentoo) above, Python and all its dependencies can be cross compiled and the root of the tree of cross compiled with a single command /micfs/gentoo-mic software ( ). The directory $ emerge \ /micfs/gentoo-mic/etc/portage contains all > --root=/micfs/gentoo-mic \ files needed to configure Portage to use the Intel compiler, > --config-root=/micfs/gentoo-mic \ and the proper compilation flags to cross compile for the > --prefix=/ python:2.7 Xeon Phi. The first file needed is a bashrc: The command above installs in total 26 packages, including $ cd /micfs/gentoo-mic/etc/portage Python. To confirm that the software was properly cross $ cat > bashrc <<-EOF compiled, we can check the architecture of the generated . /opt/intel/bin/compilervars.sh intel64 libraries with readelf: EOF $ readelf -h \ This file is sourced by Portage, which gains access to icc by > /micfs/gentoo-mic/usr/bin/Python2.7 | placing it in its $PATH. > grep Machine Now, the most important file, make.conf, is used to Machine: IntelK1OM properly configure the compiler to produce binaries for the Xeon PhiTM: Python modules can now be installed with emerge too. Due to incompatibilities, not all software can be cross compiled this $ cat > make.conf <<-EOF way. Python 3.5, for example, needs further patching before it ACCEPT_KEYWORDS="˜amd64-linux" can be installed by Portage. However, for basic software like CONFIG_SHELL="/micfs/gentoo/bin/bash" bash, rsync, and many other tools that facilitate the users’ life when logged in on the Xeon PhiTM, Portage works quite CBUILD="x86_64-pc-linux-gnu" CHOST="x86_64-k1om-linux-gnu" well.

CC=icc III. ASSESSMENT OF PORTAGE FOR HPC CXX=icpc MPI_C=mpiicc With more than 10 years development, Portage has be- MPI_CXX=mpiicpc come an excellent and mature package management system. LD=xild It is general purpose and used in a diversity of scenarios. AR=xiar Nevertheless, it is still worthwhile to assess whether Portage CFLAGS="${CFLAGS} -mmic -O3" meets the specific needs of the HPC community. The technical CXXFLAGS="${CXXFLAGS} -mmic -O3" report from the software packaging working group of HEP FFLAGS="${FFLAGS} -mmic -O3" software fundation [16] has summarized the requirements of # append link flags, order is important! HPC from the point of view of high energy physics, which is # we inject RPATHs to avoid setting also applicable to other fields. # LD_LIBRARY_PATH later LDFLAGS="${LDFLAGS} -mmic \ As the core of Gentoo is a meta distribution, Portage -L/opt/intel/lib/mic \ naturally satisfies all the requirements: -Wl,--rpath,/opt/intel/lib/mic \ 1) Portability: Portage runs on GNU/Linux, Android/Linux, -L/micfs/gentoo-mic/usr/lib \ -Wl,--rpath,/micfs/gentoo-mic/usr/lib \ Solaris, AIX, Mac OS X, Windows and more (Section -limf -lsvml -lirng -lintlc" I-B). 2) Multiple versions: Portage supports building and in- USE_PYTHON="2.7" PYTHON_TARGETS="python2_7" stalling multiple stacks in parallel with the SLOT mech- anism (Section I-C). USE="-* unicode python ncurses readline threads xml" 3) Ease: Portage is self-contained, ebuilds for new packages EOF are easy to add (Section I-B), and the bootstraping To cross compile Python 2.7 [14], a patch [15] needs process on POSIX environments is simple. to be applied to one of its dependencies, libffi, in or- 4) Support: The Gentoo community is friendly, active and der for it to properly cross compile for the Xeon PhiTM. supportive. Documentaion of Gentoo is abundant and Portage supports user patches to be applied by simply informative. dropping them into EPREFIX/etc/portage/patches 5) Performance: Gentoo is a Linux distribution famous in a subdirectory corresponding to the package to be for keeping users’ CPU warm. Compiling thousands of patched, with an optional version. In this case, we packages is a daily exercise of an average Gentoo user. dropped the patch file libffi-3.2.1-k1om.patch Portage provides a mature build system scrutinized by hackers all around the world and tested in every extreme of software communities than to circumvent issues caused by setups. The vcs support (checking out sources from bad designs. cvs, svn, git, mercury, etc.) implemented by eclasses is Therefore supporting multiple versions and build configura- excellent. tions by default calls for more caution, especially for a healthy scientific ecosystem in the a long run. IV. ADEEPER LOOK AT MULTIPLE VERSIONS AND BUILD CONFIGURATIONS V. CONCLUSION Both the HSF technical report [16] and Gamblin et al. [17] Portage is a powerful package management system from argue that multiple versions and multiple build configurations Gentoo. It is robust, mature, well-documented and user- of the same package should be available at the same time to friendly. It is especially suitable in HPC setups. Gentoo Prefix support wide user groups in a shared HPC environment. Gam- is easy to bootstrap, and easier to bootstrap a new instance blin et al. [17] also put forward explicitly that software build from an existing one. It brings the power of a well-received is by its nature combinatorial, therefore a package manager metadistribution to the users, and at the same time reduces the should support all the combinations natively. Although from heavy maintanance burden of administrators. an administrator’s point of view the system can potentially Portage and Gentoo repositories are scrutinized by hundreds have all kinds of combinations, for a single user or a closely of developers worldwide, many of whom are software pro- collaborated group this is an overkill. fessionals and respected hackers. They grow out of a group If each user group maintains their own software stack, of people who engineer complex software. Their wisdom to support for multiple versions and build configurations is rarely handle complexity is invaluable to scientific communities. necessary. Therefore the advantage provided by multiple ver- Portage is a perfect bridge between hackers and scientists, sions and build configurations is limited to administrators. and gives an ultimate solution to the chaotic status quo of Only a few packages need explicit version tracking and HPC software stacks. specific build configurations, like compilers/interpreters (gcc- REFERENCES 4 vs. 5 vs. 6, python-2 vs. 3, ruby-2.2 vs. 2.3), MPI imple- mentations, blas/lapack implementations, autoconf/automake [1] “Gentoo Linux,” http://www.gentoo.org. [2] “Portage package manager,” https://wiki.gentoo.org/wiki/Project:Portage. versions, complete rewrites of frameworks (Geant-3 vs. 4). For [3] “Gentoo Prefix Project,” https://wiki.gentoo.org/wiki/Project:Prefix. the majority of relatively small tools the cost of complexity [4] “CoreOS,” http://coreos.com. exceeds the benefit of diversity of multiple versions and build [5] “Chromium OS,” https://www.chromium.org. [6] “Gentoo Prefix with libc,” https://wiki.gentoo.org/wiki/Prefix/libc. configurations. Ultimately it is a balance between simplicity [7] “Portage Package Manager Specification,” and diversity. Portage provides straightforward solutions for https://projects.gentoo.org/pms/6/pms.pdf. installing multiple package versions with SLOTs and for [8] “paludis,” http://paludis.exherbo.org. [9] “pkgcore,” https://github.com/pkgcore/pkgcore. selecting among alternative implementations with eselect, or [10] C. A. MacLean, “Maintaining Large Software Stacks in a Cray Ecosys- disabling all implementations by default and only enabling tem with Gentoo Portage,” Cray User Group, 2016. them after conscious and educated decisions. Conversely, [11] “massif heap profiler,” http://valgrind.org/docs/manual/ms-manual.html. [12] “massif visualizer,” https://github.com/KDE/massif-visualizer. installing a multitude of versions and build configurations of a [13] “Gentoo Embedded Handbook,” https://wiki.gentoo.org/wiki/Embedded Handbook. single package by default is pursuing diversity to the extreme. [14] “Cross Compiling Python 2.7 for the Intel R Xeon TM Project management in scientific software is far from Phi with Gentoo Prefix (YouTube Screencast),” https://www.youtube.com/watch?v=YmME7nfjxzo. mature. Authors of scientific packages often value instant [15] “Workaround for compiling libffi for gratification (publish of an article) more than long term K1OM architecture (Intel R Xeon PhiTM),” maintainability (robust build system, stable API, etc.). ABI https://gist.github.com/amadio/6066dabf2b7d98ee73aab28f0dc07bf1. [16] L. Sexton-Kennedy, B. Hegner, and B. Viren, “HSF Packaging Working and even API incompatibilities between minor updates are Group Report,” FNAL, CERN, BNL, Tech. Rep., 2016. common. Function calls from libraries could give different [17] Todd Gamblin and Matthew LeGendre and Michael R. Collette and results when linked against different versions of stardard c++ Gregory L. Lee and Adam Moody and Bronis R. de Supinski and Scott Futral, “The Spack Package Manager: Bringing Order to HPC Software runtime. It therefore becomes crucial for a package manager to Chaos,” International Conference for High Performance Computering, support multiple versions and build configurations to facilitate Networking, Storage and Analysis, 2015. reproducibility in research. Package managers with such a feature provide an instant solution to the chaos of scientific software like a painkiller. Providing such a feature by default may encourage bad practice. There is a growing consensus that computer programs are replacing mathematics to become the most used language of science. There will be more sophisticated scientific software in the future. The value of good practice in project management will be more conspicuous, most of which are already estab- lished in free and open source software communities. In the context of HPC, it is more important to draw on the experience