The Spack Package Manager: Bringing Order to HPC Soware Chaos Supercompung 2015 (SC15) Aus)n, Texas Todd Gamblin, Mahew LeGendre, Michael R. Colle@e, Gregory L. Lee, Adam Moody, Bronis R. de Supinski, and Sco@ Futral November 18, 2015 LLNL-PRES-803375 This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC hp://bit.ly/spack-git What is the producon environment for HPC? § Someone’s home directory? § LLNL? LANL? Sandia? ANL? LBL? TACC? — Environments at large-scale sites are very different. § Which MPI implementaon? § Which compiler? § Which dependencies? § Which versions of dependencies? — Many applicaons require specific dependency versions. Real answer: there isn’t a single production environment or a standard way to build. 2 LLNL-PRES-803375 hp://bit.ly/spack-git HPC soware is becoming increasingly complex § Not much standardizaon in HPC: every machine/app has a different so^ware stack § Sites share unique hardware among teams with very different requirements — Users want to experiment with many exo)c architectures, compilers, MPI versions — All of this is necessary to get the best performance § Example environment for some LLNL codes: 48 third party packages x 3 MPI versions x 3-ish Platforms mvapich mvapich2 OpenMPI Linux BlueGene Cray Up to 7 compilers Oh, and 2-3 versions of x Intel GCC XLC Clang x each package = ~7,500 combinations PGI Cray Pathscale We want an easy way to quickly sample the space, to build configurations on demand! 3 LLNL-PRES-803375 hp://bit.ly/spack-git Most exisDng tools do not support combinatorial versioning § Tradi)onal binary package managers — RPM, yum, APT, yast, etc. — Designed to manage a single stack. — Install one version of each package in a single prefix (/usr). — Seamless upgrades to a stable, well tested stack § Port systems — BSD Ports, portage, Macports, Homebrew, Gentoo, etc. — Minimal support for builds parameterized by compilers, dependency versions. § Virtual Machines and Linux Containers (Docker) — Containers allow users to build environments for different applicaons. — Does not solve the build problem (someone has to build the image) — Performance, security, and upgrade issues prevent widespread HPC deployment. 4 LLNL-PRES-803375 hp://bit.ly/spack-git How do HPC sites deal with combinatorial builds? § HPC so^ware is typically installed manually in a directory hierarchy. — Hierarchy o^en doesn’t give all needed informaon about a build. — Sites can run out of unique directory names quickly. Site Naming Convention LLNL / usr / global / tools / $arch / $package / $version / usr / local / tools / $package-$compiler-$build-$version Oak Ridge / $arch / $package / $version / $build TACC / $compiler-$comp_version / $mpi / $mpi_version / $package / $version 5 LLNL-PRES-803375 hp://bit.ly/spack-git Environment modules can help, but are hard to get right. $ module avail --------------------------- /opt/modules/modulefiles ---------------------------- acml-gnu/4.4 intel/12.0 mvapich2-pgi-ofa/1.7 acml-gnu_mp/4.4 intel/13.0 mvapich2-pgi-psm/1.7 acml-intel/4.4 intel/14.0(default) mvapich2-pgi-shmem/1.7... $ module load intel/13.0 $ module load mvapich2-pgi-shmem/1.7 § Advantages: — Swap different library versions dynamically, in a shell. — Abstracts a lot of environment complexity from the user. § Disadvantages: — Users must typically remember to load the same module that they built with. • Easy to load wrong module and break code. — Many sites and vendors deploy extremely bri@le, inconsistent modules. — Module systems do not build so^ware; they only change the environment. 6 LLNL-PRES-803375 hp://bit.ly/spack-git Spack handles combinatorial soware complexity. Dependency DAG § Each unique dependency graph is a unique mpi configuraon. mpileaks libdwarf callpath dyninst libelf § Each configuraon installed in a unique directory. — Configuraons of the same package can coexist. Installaon Layout Hash spack/opt/! § Hash of en)re directed acyclic graph (DAG) is linux-x86_64/! appended to each prefix. gcc-4.7.2/! mpileaks-1.1-0f54bf34cadk/! intel-14.1/! § hdf5-1.8.15-lkf14aq3nqiz/! Installed packages automacally find dependencies bgq/! — Spack embeds RPATHs in binaries. xl-12.1/! hdf5-1-8.16-fqb3a15abrwx/! — No need to use modules or set LD_LIBRARY_PATH ...! — Things work the way you built them 7 LLNL-PRES-803375 hp://bit.ly/spack-git `spack list` shows what packages are available $ spack list ==> 243 packages. activeharmony coreutils ghostscript leveldb libxslt netcdf ppl py-pychecker qt thrift adept-utils cppcheck git libarchive llvm netgauge protobuf py-pycparser qthreads tk apex cram glib libcircle llvm-lld netlib-blas py-basemap py-pyelftools R tmux arpack cscope glm libdrm lmdb netlib-lapack py-biopython py-pygments ravel tmuxinator asciidoc cube global libdwarf lua nettle py-cffi py-pylint readline trilinos atk czmq glog libelf lwgrp ompss py-cython py-pypar rose uncrustify atlas dbus gmp libevent lwm2 ompt-openmp py-dateutil py-pyparsing ruby util-linux autoconf docbook-xml gnutls libffi matio opari2 py-epydoc py-pyqt SAMRAI vim automaded doxygen gperf libgcrypt memaxes openmpi py-genders py-pyside samtools vtk automake dri2proto gperftools libgpg-error mesa openssl py-gnuplot py-python-daemon scalasca wget bear dtcmp graphlib libjpeg-turbo metis otf py-h5py py-pytz scorep wx bib2xhtml dyninst graphviz libjson-c Mitos otf2 py-ipython py-rpy2 scotch wxpropgrid binutils elfutils gtkplus libmng mpc pango py-libxml2 py-scientificpython scr xcb-proto bison extrae harfbuzz libmonitor mpe2 papi py-lockfile py-scikit-learn silo xz boost exuberant-ctags hdf5 libNBC mpfr paraver py-mako py-scipy snappy yasm bowtie2 fish hwloc libpciaccess mpibash paraview py-matplotlib py-setuptools spindle zeromq boxlib flex hypre libpng mpich parmetis py-mock py-shiboken sqlite zlib bzip2 flux icu libsodium mpileaks parpack py-mpi4py py-sip stat zsh cairo fontconfig icu4c libtiff mrnet pcre py-mx py-six sundials callpath freetype ImageMagick libtool munge petsc py-nose py-sphinx swig cblas gasnet isl libunwind muster pidx py-numpy py-sympy task cgm gcc jdk libuuid mvapich2 pixman py-pandas py-virtualenv taskd clang gdk-pixbuf jpeg libxcb nasm pkg-config py-pexpect py-yapf tau cloog geos launchmon libxml2 ncdu pmgr_collective py-pil python tcl cmake gflags lcms libxshmfence ncurses postgresql py-pmw qhull the_silver_searcher 8 LLNL-PRES-803375 hp://bit.ly/spack-git Spack provides a spec syntax to describe customized DAG configuraons $ spack install mpileaks unconstrained! ! $ spack install [email protected] @ custom version! ! $ spack install [email protected] %[email protected] % custom compiler! ! $ spack install [email protected] %[email protected] +threads +/- build option! ! $ spack install [email protected] =bgq = cross-compile! § Each expression is a spec for a par)cular configuraon — Each clause adds a constraint to the spec — Constraints are op)onal – specify only what you need. — Customize install on the command line! § Syntax abstracts details in the common case — Makes parameterizaon by version, compiler, and op)ons easy when necessary 9 LLNL-PRES-803375 hp://bit.ly/spack-git Spack Specs can constrain versions of dependencies mpi mpileaks libdwarf callpath dyninst libelf $ spack install mpileaks %[email protected] ^[email protected]! § Spack ensures one configuraon of each library per DAG — Ensures ABI consistency. — User does not need to know DAG structure; only the dependency names. § Spack can ensure that builds use the same compiler, or you can mix — Working on ensuring ABI compability when compilers are mixed. 10 LLNL-PRES-803375 hp://bit.ly/spack-git Spack handles ABI-incompaDble, versioned interfaces like MPI mpi mpileaks libdwarf callpath dyninst libelf § mpi is a virtual dependency § Install the same package built with two different MPI implementaons: $ spack install mpileaks ^[email protected]! $ spack install mpileaks ^[email protected]:! § Let Spack choose MPI version, as long as it provides MPI 2 interface: $ spack install mpileaks ^mpi@2! 11 LLNL-PRES-803375 hp://bit.ly/spack-git Spack packages are simple Python scripts. from spack import * class Dyninst(Package): """API for dynamic binary instrumentation.""” Metadata homepage = "https://paradyn.org" version('8.2.1', 'abf60b7faabe7a2e’, url="http://www.paradyn.org/release8.2/DyninstAPI-8.2.1.tgz") version('8.1.2', 'bf03b33375afa66f’, url="http://www.paradyn.org/release8.1.2/DyninstAPI-8.1.2.tgz") Versions and URLs version('8.1.1', 'd1a04e995b7aa709’, url="http://www.paradyn.org/release8.1/DyninstAPI-8.1.1.tgz") depends_on("libelf") depends_on("libdwarf") Dependencies depends_on("[email protected]:") Patches, variants (not shown) def install(self, spec, prefix): libelf = spec['libelf'].prefix libdwarf = spec['libdwarf'].prefix with working_dir('spack-build', create=True): cmake('..', '-DBoost_INCLUDE_DIR=%s' % spec['boost'].prefix.include, '-DBoost_LIBRARY_DIR=%s' % spec['boost'].prefix.lib, '-DBoost_NO_SYSTEM_PATHS=TRUE’ *std_cmake_args) Commands for installation make() make("install") @when('@:8.1') Access build config through def install(self, spec, prefix): configure("--prefix=" + prefix) the spec parameter. make() make("install") 12 LLNL-PRES-803375 hp://bit.ly/spack-git Dependencies in Spack may be oponal. § The user can define named variants: variant("python", default=False, “Build with python support”) depends_on("python", when="+python") § And use them to install: $ spack install vim +python $ spack install vim –python § Dependencies may be op)onal according to other condi)ons: