Software Packaging Techniques: A Comparison
Pengfei Ding DUNE DAQ/SC Meeting June 15th, 2020 Common Features of Packaging Technique
• Central repository of available packages; • Local database of installed packages; • Metadata for package description (dependency, install instruction etc); • Mechanism to install/reinstall/upgrade/verify package; • Digital signature of package.
2 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting Focus of Our Packaging Needs • Package dependencies: - Built-in package dependencies; - Automatic dependency checks; - Automatic dependency install (activate/setup).
• Package Versions: - Installations of multiple versions of a package; - Installations of multiple variants of a package; - Quick switch between variants and versions.
• Building and distribution: - Multi-package development environment.
3 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting Packaging techniques compared • UPS (Unix Product Support) (with UPD) https://cdcvs.fnal.gov/redmine/projects/ups/wiki/Getting_Started_Using_UPS - Developed by Fermilab, currently used by many (if not all) Fermilab experiments; - Decades old, plans to migrate to other modern tools (rumor is spack as the top candidate).
• Spack https://spack.io - Initially developed by LLNL for HPCs, adopted by the community and spread to other platforms; - Used by LArSoft, and several other HEP software projects.
• RPM (with YUM and DNF) https://rpm.org/documentation.html - Primarily used by Fedora and RedHat. - Used by ATLAS TDAQ in the HEP community.
4 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting Package dependencies
• UPS:
- Dependencies specified in a “table” (text) file;
- If needed, local overwrite can be easily done by editing the table file;
- Automatic activation of dependent package;
- Tools available for listing the dependency tree;
- Consistency check when activating a package, abort if conflicts in the dependency tree is found; • Spack:
- Dependencies specified in ”package.py” file for each package;
- If needed, local overwrite can be done by editing the “package.py” file;
- Automatic installation of dependent package;
- Programing functionality available when specifying dependencies (e.g. specifying different dependencies for different usage case);
- Tools available for dependency listing and consistency check. • RPM:
- Dependencies specified in package metadata, hard to overwrite locally;
- Automatic installation of dependent package;
- Tools available for dependency listing and consistency check.
5 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting Package Versions
• UPS: - Support multiple versions of a package; - Support different builds of a package:
• Different Linux distro and versions;
• Different compilers and compiler flags used (debug/opt); - Local installations of multiple versions or variants; - Quick switch between different versions/variants; can be activated if a build is available for the targeted OS. • Spack: - Support different platforms: macOS, HPC, various Linux distros; - Quick switch among multiple locally-installed versions/variants; • RPM: - Primarily on Fedora or Redhat Linux distros; - Support of different versions and variants; - Less convenient to have multiple versions installed locally and do quick switch between them.
6 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting Build and Distribute (I) • UPS and RPM packages are essentially tarballs: - After unrolling the tarballs, files in UPS package lives under a dedicated UPS products directory, separated from OS system files, lib and include path got added to environment variables upon activation of a package; - Files in RPM packages get put into directories same as those used/specified while building; can be directories mixed with system files or dedicated directories; additional instructions (modifying envs) can be added in RPM’s post- script. - Central installations are available for UPS packages; - RPM prefers local installation for all packages.
• Spack: - Central ”repository” is lean, hosts a package’s “package.py” file: • Specifying the source of the package: URL to a source tarball or a URL to its repo; • Initial installation pulls down the source package and build it; • Depends on the dependencies, the initial installation may take hours to finish.
7 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting Build and Distribute (II) • Spack (cont’d): - Central installations are available; - Build cache is available (tarballs of pre-built package); - No use of envs, but use built-in rpath in binary files (libs and executables).
• Multi-package development environment: - UPS plus SRT (depreciated) or MRB; - Spack with Spack-dev; - RPM: subpackages.
8 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting Build and distribute a UPS package (I)
• Can be built from costumed scripts with providing a table file and follow certain directory structure; • Ssibuildshims and build-framework is available for continence.
9 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting Build and distribute an UPS package (II)
• Supports many build systems (to my knowledge, bazel is currently the only one we have problem with). • Package is distributed via tarballs; • Can be uploaded to a central web server (scisoft.fnal.gov); • Or being declared to Fermilab UPD (central repo for UPS packages);
• Multi-package development environment for cmake based pacakges can be achieved via cet-build-tools (part of MRB).
10 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting Build and distribute a spack package (I)
• Build systems supported: Cmake, Makefile, Autotools, PythonPackage, WafPackage, RPackage and custom build systems.
11 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting Build and distribute a spack package (II)
• Build instructions specified in package.py; • By default, package is built from source; • Central installation can be setup for clusters; • Pre-built tarballs can be uploaded to a build cache to save recompile time.
• Development environment for multiple packages can be achieved via spack-dev.
12 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting Build and distribute an RPM package
• Supports various build systems; • Can distribute source package and pre-built package; • Easy to setup a central repo (YUM repo); • Prefers local over central installations of all packages.
• Tools available for building rpm packages (essentially ch-root behind the scenes); • Normally requires root privilege for package installation/patching or version switch.
• Starting from RHEL 8 (CentOS 8), YUM is replaced by DNF; which has trouble supporting multiple versions of packages (e.g. python, perl etc).
13 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting My personal take • UPS is soon to be deprecated; • RPM with YUM or DNF has unresolved supporting issues on CentOS 8 for multiple versions of a package, will need to develop a set of tools for managing DAQ software releases (or adopt from ATLAS TDAQ) - RPM prefers local installation and root privilege for installation/modification to a package.
• Spack is the top candidate in my personal view. • Before we commit to spack, the following things need to be demonstrated in the context of DUNE DAQ software: - The use of build cache for pre-built tarballs; - Central installation of packages.
14 06/15/2020 Pengfei Ding | DUNE DAQ/SC Meeting