Software Packaging Techniques: A Comparison Pengfei Ding DUNE DAQ/SC Meeting June 15th, 2020 Common Features of Packaging Technique • Central repository of available packages; • Local database of installed packages; • Metadata for package description (dependency, install instruction etc); • Mechanism to install/reinstall/upgrade/verify package; • Digital signature of package. 2 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting Focus of Our Packaging Needs • Package dependencies: - Built-in package dependencies; - Automatic dependency checks; - Automatic dependency install (activate/setup). • Package Versions: - Installations of multiple versions of a package; - Installations of multiple variants of a package; - Quick switch between variants and versions. • Building and distribution: - Multi-package development environment. 3 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting Packaging techniques compared • UPS (Unix Product Support) (with UPD) https://cdcvs.fnal.gov/redmine/projects/ups/wiki/Getting_Started_Using_UPS - Developed by Fermilab, currently used by many (if not all) Fermilab experiments; - Decades old, plans to migrate to other modern tools (rumor is spack as the top candidate). • Spack https://spack.io - Initially developed by LLNL for HPCs, adopted by the community and spread to other platforms; - Used by LArSoft, and several other HEP software projects. • RPM (with YUM and DNF) https://rpm.org/documentation.html - Primarily used by Fedora and RedHat. - Used by ATLAS TDAQ in the HEP community. 4 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting Package dependencies • UPS: - Dependencies specified in a “table” (text) file; - If needed, local overwrite can be easily done by editing the table file; - Automatic activation of dependent package; - Tools available for listing the dependency tree; - Consistency check when activating a package, abort if conflicts in the dependency tree is found; • Spack: - Dependencies specified in ”package.py” file for each package; - If needed, local overwrite can be done by editing the “package.py” file; - Automatic installation of dependent package; - Programing functionality available when specifying dependencies (e.g. specifying different dependencies for different usage case); - Tools available for dependency listing and consistency check. • RPM: - Dependencies specified in package metadata, hard to overwrite locally; - Automatic installation of dependent package; - Tools available for dependency listing and consistency check. 5 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting Package Versions • UPS: - Support multiple versions of a package; - Support different builds of a package: • Different Linux distro and versions; • Different compilers and compiler flags used (debug/opt); - Local installations of multiple versions or variants; - Quick switch between different versions/variants; can be activated if a build is available for the targeted OS. • Spack: - Support different platforms: macOS, HPC, various Linux distros; - Quick switch among multiple locally-installed versions/variants; • RPM: - Primarily on Fedora or Redhat Linux distros; - Support of different versions and variants; - Less convenient to have multiple versions installed locally and do quick switch between them. 6 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting Build and Distribute (I) • UPS and RPM packages are essentially tarballs: - After unrolling the tarballs, files in UPS package lives under a dedicated UPS products directory, separated from OS system files, lib and include path got added to environment variables upon activation of a package; - Files in RPM packages get put into directories same as those used/specified while building; can be directories mixed with system files or dedicated directories; additional instructions (modifying envs) can be added in RPM’s post- script. - Central installations are available for UPS packages; - RPM prefers local installation for all packages. • Spack: - Central ”repository” is lean, hosts a package’s “package.py” file: • Specifying the source of the package: URL to a source tarball or a URL to its repo; • Initial installation pulls down the source package and build it; • Depends on the dependencies, the initial installation may take hours to finish. 7 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting Build and Distribute (II) • Spack (cont’d): - Central installations are available; - Build cache is available (tarballs of pre-built package); - No use of envs, but use built-in rpath in binary files (libs and executables). • Multi-package development environment: - UPS plus SRT (depreciated) or MRB; - Spack with Spack-dev; - RPM: subpackages. 8 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting Build and distribute a UPS package (I) • Can be built from costumed scripts with providing a table file and follow certain directory structure; • Ssibuildshims and build-framework is available for continence. 9 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting Build and distribute an UPS package (II) • Supports many build systems (to my knowledge, bazel is currently the only one we have problem with). • Package is distributed via tarballs; • Can be uploaded to a central web server (scisoft.fnal.gov); • Or being declared to Fermilab UPD (central repo for UPS packages); • Multi-package development environment for cmake based pacakges can be achieved via cet-build-tools (part of MRB). 10 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting Build and distribute a spack package (I) • Build systems supported: Cmake, Makefile, Autotools, PythonPackage, WafPackage, RPackage and custom build systems. 11 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting Build and distribute a spack package (II) • Build instructions specified in package.py; • By default, package is built from source; • Central installation can be setup for clusters; • Pre-built tarballs can be uploaded to a build cache to save recompile time. • Development environment for multiple packages can be achieved via spack-dev. 12 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting Build and distribute an RPM package • Supports various build systems; • Can distribute source package and pre-built package; • Easy to setup a central repo (YUM repo); • Prefers local over central installations of all packages. • Tools available for building rpm packages (essentially ch-root behind the scenes); • Normally requires root privilege for package installation/patching or version switch. • Starting from RHEL 8 (CentOS 8), YUM is replaced by DNF; which has trouble supporting multiple versions of packages (e.g. python, perl etc). 13 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting My personal take • UPS is soon to be deprecated; • RPM with YUM or DNF has unresolved supporting issues on CentOS 8 for multiple versions of a package, will need to develop a set of tools for managing DAQ software releases (or adopt from ATLAS TDAQ) - RPM prefers local installation and root privilege for installation/modification to a package. • Spack is the top candidate in my personal view. • Before we commit to spack, the following things need to be demonstrated in the context of DUNE DAQ software: - The use of build cache for pre-built tarballs; - Central installation of packages. 14 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting.