Spack @Spackpm

github.com/spack @spackpm

Spack: A Package Manager for HPC 2019 HPC-AI Advisory Council Stanford Conference

February 14, 2019 Todd Gamblin Stanford University Computer Scientist

LLNL-PRES-747560 This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE- AC52-07NA27344. Lawrence Livermore National Security, LLC Scientific software is becoming extremely complex

Nalu: Generalizeddealii Unstructured: C++ Finite MassivelyR Element Miner: Parallel R Data Library Mining Low Mach Library Flow github.com/spack 2 LLNL-PRES-747560 @spackpm Even proprietary codes are based on many open source libraries

▪ Half of this DAG is external (blue); more than half of it is open source

▪ Nearly all of it needs to be built specially for HPC to get the best performance

github.com/spack 3 LLNL-PRES-747560 @spackpm The Exascale Computing Project is building an entire ecosystem

5+ target architectures/platforms 15+ applications x 80+ software packages x Xeon Power KNL NVIDIA ARM Laptops?

Up to 7 compilers 10+ Programming Models 2-3 versions of each package + x Intel GCC Clang XL x OpenMPI MPICH MVAPICH OpenMP CUDA x external dependencies PGI Cray NAG OpenACC Dharma Legion RAJA Kokkos

= up to 1,260,000 combinations! ▪ Every application has its own stack of dependencies. ▪ Developers, users, and facilities dedicate (many) FTEs to building & porting. ▪ Often trade reuse and usability for performance.

We must make it easier to rely on others’ software!

github.com/spack 4 LLNL-PRES-747560 @spackpm How to install software on a Mac laptop, circa 2013

github.com/spack 5 LLNL-PRES-747560 @spackpm

How to install software on a supercomputer

Fight with compiler...

configure Tweak configure 1. Download all 16 make install tarballs you need cmake

2. Start building!

make

configure

make install

args

make make install

configure

......

make

make make

3. Run code 4. Segfault!? 5. Start over…

github.com/spack 6 LLNL-PRES-747560 @spackpm What about modules?

▪ Most supercomputers deploy some form of environment modules — TCL modules (dates back to 1995) and Lmod (from TACC) are the most popular

$ gcc - bash: gcc: command not found

$ module load gcc/7.0.1 $ gcc –dumpversion 7.0.1

▪ Modules don’t handle installation! — They only modify your environment (things like PATH, LD_LIBRARY_PATH, etc.)

▪ Someone (likely a team of people) has already installed gcc for you! — Also, you can only `module load` the things they’ve installed

github.com/spack 7 LLNL-PRES-747560 @spackpm What about containers?

▪ Containers provide a great way to reproduce and distribute an already-built software stack

▪ Someone needs to build the container! — This isn’t trivial — Containerized applications still have hundreds of dependencies

▪ Using the OS package manager inside a container is insufficient — Most binaries are built unoptimized — Generic binaries, not optimized for specific architectures

▪ Developing with an OS software stack can be painful — Little freedom to choose versions — Little freedom to choose compiler options, build options, etc. for packages We need something more flexible to build the containers

github.com/spack 8 LLNL-PRES-747560 @spackpm Spack is a flexible package manager for HPC

▪ How to install Spack (works out of the box): $ git clone https://github.com/spack/spack $ . spack/share/spack/setup-env.sh ▪ How to install a package:

$ spack install hdf5

▪ HDF5 and its dependencies are installed within the Spack directory. Visit spack.io ▪ Unlike typical package managers, Spack can also install many variants of the same build. github.com/spack/spack — Different compilers — Different MPI implementations — Different build options @spackpm

github.com/spack 9 LLNL-PRES-747560 @spackpm Spack provides the spec syntax to describe custom configurations

$ spack install mpileaks unconstrained $ spack install [email protected] @ custom version $ spack install [email protected] %[email protected] % custom compiler $ spack install [email protected] %[email protected] +threads +/- build option $ spack install [email protected] cxxflags="-O3 –g3” setting compiler flags $ spack install [email protected] os=cnl10 target=haswell setting target for X-compile $ spack install [email protected] ^[email protected] %[email protected] ^ dependency information

▪ Each expression is a spec for a particular configuration — Each clause adds a constraint to the spec — Constraints are optional – specify only what you need. — Customize install on the command line! ▪ Spec syntax is recursive — Full control over the combinatorial build space

github.com/spack 10 LLNL-PRES-747560 @spackpm `spack list` shows what packages are available

$ spack list ==> 3041 packages. abinit glew nalu py-fastaindex r-cairo r-viridislite abyss glfmultiples nalu-wind py-fasteners r-callr r-visnetwork accfft glib namd py-faststructure r-car r-vsn ack glibmm nano py-filelock r-caret r-webshot activeharmony glimmer nanoflann py-fiona r-category r-whisker adept-utils glm nanopb py-fiscalyear r-catools r-withr adios global nasm py-flake8 r-cdcfluview r-xde adios2 globalarrays nauty py-flake8-polyfill r-cellranger r-xgboost adlbx globus-toolkit ncbi-magicblast py-flask r-checkmate r-xlconnect adol-c glog ncbi-rmblastn py-flask-compress r-checkpoint r-xlconnectjars aegean gloo ncbi-toolkit py-flask-socketio r-chemometrics r-xlsx aida glpk nccl py-flexx r-chron r-xlsxjars albany glproto nccmp py-fn r-circlize r-xmapbridge albert glvis ncdu py-fparser r-class r-xml alglib gmake ncftp py-funcsigs r-classint r-xml2 allinea-forge gmap-gsnap ncl py-functools32 r-cli r-xnomial allinea-reports gmime nco py-future r-clipr r-xtable allpaths-lg gmodel ncurses py-futures r-cluster r-xts alquimia gmp ncview py-fypp r-clustergeneration r-xvector alsa-lib gmsh ndiff py-gdbgui r-clusterprofiler r-yaml aluminum gmt nek5000 py-genders r-cner r-yapsa amg gnat nekbone py-genshi r-coda r-yaqcaffy amg2013 gnu-prolog nekcem py-geopandas r-codetools r-yarn amp gnupg nektar py-gevent r-coin r-zlibbioc ampliconnoise gnuplot neovim py-git-review r-colorspace r-zoo amrex gnutls nest py-git2 r-combinat r3d amrvis go netcdf py-gnuplot r-complexheatmap racon andi go-bootstrap netcdf-cxx py-goatools r-compositions raft angsd gobject-introspection netcdf-cxx4 py-gpaw r-convevol ragel ant googletest netcdf-fortran py-greenlet r-corhmm raja antlr gotcha netgauge py-griddataformats r-corpcor randfold ants gource netgen py-guidata r-corrplot random123 ape gperf netlib-lapack py-guiqwt r-covr randrproto . . . ▪ Spack has over 3,000 builtin package recipes.

github.com/spack 11 LLNL-PRES-747560 @spackpm `spack find` shows what is installed

-- linux-rhel7-x86_64 / [email protected] ------— Don’t have to use them. [email protected] [email protected] libdwarf@20130729 [email protected] [email protected] [email protected] [email protected] [email protected]

github.com/spack 12 LLNL-PRES-747560 @spackpm Users can query the full dependency configuration of installed packages.

$ spack find callpath ==> 2 installed packages. -- linux-rhel7-x86_64 / [email protected] ———————— -- linux-rhel7-x86_64 / [email protected] [email protected] [email protected]

$ spack find -dl callpath ==> 2 installed packages. -- linux-rhel7-x86_64 / [email protected] ------linux-rhel7-x86_64 / [email protected] ------xv2clz2 [email protected] udltshs [email protected] ckjazss ^[email protected] rfsu7fb ^[email protected] 3ws43m4 ^boost^[email protected]@1.59.0 ybet64yybet64y ^boost^[email protected]@1.55.0 Expand dependencies ft7znm6 ^[email protected] aa4ar6i ^[email protected] qqnuet3 ^[email protected] tmnnge5 ^[email protected] with spack find -d 3ws43m4 ^boost^[email protected]@1.59.0 ybet64yybet64y ^boost^[email protected]@1.55.0 g65rdud ^libdwarf@20130729 g2mxrl2 ^libdwarf@20130729 cj5p5fk ^[email protected] ynpai3j ^[email protected] cj5p5fk ^[email protected] ynpai3j ^[email protected] g65rdud ^libdwarf@20130729 g2mxrl2 ^libdwarf@20130729 cj5p5fk ^[email protected] ynpai3j ^[email protected] cj5p5fk ^[email protected] ynpai3j ^[email protected] ft7znm6 ^[email protected] aa4ar6i ^[email protected]

▪ Architecture, compiler, versions, and variants may differ between builds.

github.com/spack 13 LLNL-PRES-747560 @spackpm Spack packages are templates They use a simple Python DSL to define how to build a spec

from spack import *

class Dyninst(Package): """API for dynamic binary instrumentation.""” Metadata at the class level homepage = "https://paradyn.org" url = "http://www.paradyn.org/release8.1.2/DyninstAPI-8.1.2.tgz"

version('8.2.1', 'abf60b7faabe7a2e’) version('8.1.2', 'bf03b33375afa66f’) Versions version('8.1.1', 'd1a04e995b7aa709’)

depends_on("cmake", type="build")

depends_on("libelf", type="link") Dependencies (note: they use the same spec syntax) depends_on("libdwarf", type="link") depends_on("boost @1.42: +multithreaded") Patches, variants, resources, conflicts, etc. def install(self, spec, prefix): (not shown) with working_dir('spack-build', create=True): cmake('-DBoost_INCLUDE_DIR=‘ + spec['boost'].prefix.include, '-DBoost_LIBRARY_DIR=‘ + spec['boost'].prefix.lib, '-DBoost_NO_SYSTEM_PATHS=TRUE’ Install logic in instance methods '..') make() make("install")

github.com/spack 14 LLNL-PRES-747560 @spackpm Spack handles combinatorial software complexity.

Dependency DAG ▪ Each unique dependency graph is a unique configuration.

▪ Each configuration installed in a unique directory. — Configurations of the same package can coexist.

Installation Layout Hash ▪ Hash of entire directed acyclic graph (DAG) is spack/opt/ linux-x86_64/ appended to each prefix. gcc-4.7.2/ mpileaks-1.1-0f54bf34cadk/ ▪ Installed packages automatically find dependencies intel-14.1/ hdf5-1.8.15-lkf14aq3nqiz/ — Spack embeds RPATHs in binaries. bgq/ — No need to use modules or set LD_LIBRARY_PATH xl-12.1/ hdf5-1-8.16-fqb3a15abrwx/ — Things work the way you built them ... No limit on the number of versions you can have installed.

github.com/spack 15 LLNL-PRES-747560 @spackpm Depend on interfaces (not implementations) with virtual dependencies

▪ mpi is a virtual dependency

▪ Install the same package built with two different MPI implementations:

$ spack install mpileaks ^mvapich Virtual dependencies can be versioned:

class Mpileaks(Package): depends_on("mpi@2:") dependent $ spack install mpileaks ^[email protected]:

class Mvapich(Package): provider ▪ Virtual deps are replaced with a valid provides("mpi@1” when="@:1.8") provides("mpi@2” when="@1.9:") implementation at resolution time. — If the user didn’t pick something and there are class Openmpi(Package): provider multiple options, Spack picks. provides("mpi@:2.2" when="@1.6.5:")

github.com/spack 16 LLNL-PRES-747560 @spackpm Concretization fills in missing parts of requested specs.

mpileaks ^[email protected]+debug ^[email protected] Concretize ▪ Workflow: 1. Users input only an abstract spec with some constraints 2. Spack makes choices according to policies (site/user/etc.) 3. Spack installs concrete configurations of package + dependencies

▪ Dependency resolution is an NP-complete problem! — Different versions/configurations of packages require different versions/configurations of dependencies — Concretizer searches for a configuration that satisfies all the requirements — This is basically a SAT/SMT solve Concrete spec is fully constrained and can be passed to install.

github.com/spack 17 LLNL-PRES-747560 @spackpm Dependency Resolution is an NP-hard problem!

▪ Different versions of packages require different versions of dependencies — Concretizer searches for a configuration that satisfies all the requirements — Can show that SAT/SMT solve is equivalent problem ▪ Resolution is NP-complete for *just* package and version metadata

— Concretization also includes compilers, variants, architecture, optional dependencies, virtual dependencies Unsatisfiable! — We have some leeway because multiple stacks can coexist within Spack (unlike system PMs) — Even within one DAG there can be issues! https://research.swtch.com/version-sat

github.com/spack 18 LLNL-PRES-747560 @spackpm Spack is used worldwide!

Over 3,000 software packages Over 150,000 downloads in the past year Over 1,100 monthly active users (on docs site) Over 350 contributors from labs, academia, industry

Plot shows sessions on spack.readthedocs.io for one month

github.com/spack 19 LLNL-PRES-747560 @spackpm Spack has a very active open source community

▪ Started Spack development in 2013 — Paper at SC15 — Tutorials at SC16, SC17, SC18 — GitHub community has grown steadily!

▪ 232 pull requests merged in lead-up to SC18! — By 74 contributors — We’ve been gradually increasing core contributors

github.com/spack 20 LLNL-PRES-747560 @spackpm Spack has benefitted tremendously from external contributions

▪ We try to make it easy to modify a package — spack edit — Pull request

▪ Contributors are HPC software developers as well as user support teams and admins

▪ We get contributions in the core as well as in packages

▪ LLNL still ha a majority of the core contributions, with significant help from others.

github.com/spack 21 LLNL-PRES-747560 @spackpm Spack is being used on many of the top HPC systems

▪ At HPC sites for software stack+ modules Summit (ORNL) — Reduced Summit deploy time from 2 weeks to 12 hrs. Sierra (LLNL) — EPFL deploys its software stack with Jenkins + Spack — NERSC, LLNL, ANL, other US DOE sites — SJTU in China

▪ Within ECP as part of their software release process — ECP-wide software distribution — SDK workflows

▪ Within High Energy Physics (HEP) community Cori (NERSC) — HEP (Fermi, CERN) have contributed many features to support their workflow

▪ Many others SuperMUC-NG (LRZ)

github.com/spack 22 LLNL-PRES-747560 @spackpm Spack v0.12.1 was just released

▪ New stuff: 1. Spack environments (covered today) 2. spack.yaml and spack.lock files for tracking dependencies (covered today) 3. Custom configurations via command line (covered today) 4. Better support for linking Python packages into view directories (pip in views) 5. Support for uploading build logs to CDash 6. Packages have more control over compiler flags via flag handlers 7. Better support for module file generation 8. Better support for Intel compilers, Intel MPI, etc. 9. Many performance improvements, improved startup time ▪ Spack is now permissively licensed under Apache-2.0 or MIT — previously LGPL ▪ Over 2,900 packages (800 added since last year) — This is from November; over 3,000 in latest develop branch

github.com/spack 23 LLNL-PRES-747560 @spackpm Spack has added environments and spack.yaml / spack.lock

Dependency packages install build project spack.yaml file with Lockfile describes names of required exact versions installed dependencies

▪ Allows developers to bundle Spack configuration with their repository

▪ Can also be used to maintain configuration together with Spack packages. — E.g., versioning your own local software stack with consistent compilers/MPI implementations Simple spack.yaml file

▪ Manifest / Lockfile model pioneered by Bundler is becoming standard — spack.yaml describes project requirements — spack.lock describes exactly what versions/configurations were installed, allows them to be reproduced.

github.com/spack 24 LLNL-PRES-747560 @spackpm Spack environments also help with building containers

▪ We recently started providing base images on DockerHub with Spack preinstalled.

▪ Very easy to build a container with some Spack packages in it:

spack-docker-demo/ Base image with Spack in PATH Dockerfile Copy in spack.yaml spack.yaml Then run spack install

Build with docker build . List of packages to install, Run with Singularity with constraints (or another tool)

github.com/spack 25 LLNL-PRES-747560 @spackpm What’s on the road map?

▪ Supporting the U.S. Exascale project with binary builds — Spack will be used to manage ECP software releases — In conjunction with ECP CI, start to generate prebuilt binaries for HPC facilities — Use the same relocatable binary packages for container deployment ▪ Spack stacks: Build on environments to enable more automated deployment at HPC centers. — Single YAML-file configuration for entire site stack — Install massive combinatorial package installations, modules, etc. with one command. ▪ Spack chains: — Allow user Spack instances to leverage facility and team installations — Hierarchical development flow ▪ Architecture-specific binaries — Better provenance for builds — Better support for matching optimized binary packages to machines ▪ Better dependency resolution — Handle newer C++ libraries better — More aggressive concretizer support — Support for depending on language levels/compiler features (e.g., C++14, lambdas, OpenMP@version)

github.com/spack 26 LLNL-PRES-747560 @spackpm Spack is the delivery platform for the ECP software stack

▪ U.S. Exascale Computing Project (ECP) will release software through Spack ▪ Software in ECP stack needs to run on ECP platforms, testbeds, clusters, laptops — Each new environment requires effort. ▪ ECP asks us to build a robust, reliable, and easy-to-use software stack ▪ We will provide the infrastructure necessary to make this tractable: 1. A dependency model that can handle HPC software 2. A hub for coordinated software releases (like xSDK) 3. Build and test automation for large packages across facility 4. Hosted binary and source software distributions for all ECP HPC platforms

github.com/spack 27 LLNL-PRES-747560 @spackpm Through ECP, we are working with Onyx Point to deliver continuous integration for HPC centers

Trusted runners at HPC facility Setuid runner Batch runner

Fast mirroring User checks out / commits code

Two-factor authentication

▪ CI at HPC centers is notoriously difficult — Security concerns prevent most CI tools from being run by staff or by users — HPC centers really need to deploy trusted CI services for this to work ▪ We are developing a secure CI system for HPC centers: — Setuid runners (run CI jobs as users); Batch integration (similar, but parallel jobs); multi-center runner support ▪ Onyx Point will upstream this support into GitLab CI — Initial rollout in FY19 at ECP labs: ANL, ORNL, NERSC, LLNL, LANL, SNL — Upstream GitLab features can be used by anyone!

github.com/spack 28 LLNL-PRES-747560 @spackpm We are building CI infrastructure for source and binary distribution

Source archives Amazon S3 source mirror

Binary packages

HPC Centers

Pull requests Amazon S3 binary mirror

User

github.com/spack 29 LLNL-PRES-747560 @spackpm Spack stacks: entire facility deployments in a single YAML file

▪ Allow users to easily express a huge cross- product of specs — All the packages needed for a facility — Generate modules tailored to the site — Generate a directory layout to browse the packages

▪ Build on the environments workflow — Manifest + lockfile — Lockfile enables reproducibility

▪ Relocatable binaries allow the same binary to be used in a stack, regular install, or container build. — Difference is how the user interacts with the stack — Single-PATH stack vs. modules.

github.com/spack 30 LLNL-PRES-747560 @spackpm Specific target information in specs – In progress

▪ As an HPC package manager, we want to provide optimized builds — Code level choices (O2, O3) — Architecture specific choices (-mcpu=cortex-a7, -march=haswell) ▪ Architectures vary as to how much they expose features to users — x86 exposes feature sets in /proc/cpuinfo — Arm hides many features behind revision number ▪ Methods for accessing architecture optimizations — Vary by both compiler and architecture • Gcc –mcpu vs. –march, for example • Relies on architectures providing a programmatic way to get information ▪ We want to expose the names users understand — Thunderx2, cortex-a7 for arm — Power8, power9 for IBM — Haswell, skylake for Intel

github.com/spack 31 LLNL-PRES-747560 @spackpm The Spack community is growing rapidly

▪ Spack simplifies HPC software for: — Users — Developers — Cluster installations — The largest HPC facilities ▪ Spack is central to ECP’s software strategy — Enable software reuse for developers and users — Allow the facilities to consume the entire ECP stack ▪ The roadmap is packed with new features: Visit spack.io — Building the ECP software distribution — Better workflows for building containers — Stacks for facilities github.com/spack/spack — Chains for rapid dev workflow — Optimized binaries — Better dependency resolution @spackpm

github.com/spack 32 LLNL-PRES-747560 @spackpm Disclaimer This document was prepared as an account of work sponsored by an agency of the United States government. Neither the United States government nor Lawrence Livermore National Security, LLC, nor any of their employees makes any warranty, expressed or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States government or Lawrence Livermore National Security, LLC. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States government or Lawrence Livermore National Security, LLC, and shall not be used for advertising or product endorsement purposes.