Easybuild Documentation Release 20150714.01
Total Page:16
File Type:pdf, Size:1020Kb
EasyBuild Documentation Release 20150714.01 Ghent University Tue, 14 Jul 2015 12:47:12 Contents 1 Introductory topics 3 1.1 What is EasyBuild?.........................................3 1.2 Concepts and terminology......................................4 1.3 Typical workflow example: building and installing WRF......................6 2 Getting started 9 2.1 Installing EasyBuild.........................................9 2.2 Configuring EasyBuild........................................ 16 3 Basic usage topics 27 3.1 Using the EasyBuild command line................................. 27 3.2 Writing easyconfig files: the basics................................. 36 3.3 Understanding EasyBuild logs.................................... 42 4 Advanced usage topics 45 4.1 Experimental features........................................ 45 4.2 Including additional Python modules (--include-*)...................... 45 4.3 Integration with GitHub....................................... 47 4.4 Manipulating dependencies..................................... 49 4.5 Packaging support.......................................... 50 4.6 Partial installations.......................................... 53 4.7 Submitting jobs using --job .................................... 57 4.8 Using external modules....................................... 63 5 Other topics 65 5.1 Code style.............................................. 65 5.2 Unit tests............................................... 65 5.3 Useful scripts............................................. 69 5.4 Deprecated functionality....................................... 71 5.5 Removed functionality........................................ 73 6 Getting help 81 7 Lists and tables 83 8 Appendices 85 8.1 Changelog for EasyBuild documentation.............................. 85 8.2 Configuration Legacy........................................ 88 8.3 Available easyconfig parameters................................... 89 8.4 Available easyconfig parameters for EB_WRF........................... 91 8.5 Getting help, --help ........................................ 93 8.6 List of easyblocks.......................................... 98 8.7 List of known toolchains....................................... 101 i 8.8 Alternative installation methods................................... 102 8.9 Installing environment modules without root permissions...................... 105 8.10 Installing Lmod without root permissions.............................. 106 8.11 Useful links.............................................. 107 8.12 Sphinx and Read the Docs reference documents online....................... 107 8.13 Third party Sphinx examples, themes and possible extensions................... 107 8.14 EasyBuild release notes....................................... 108 ii EasyBuild Documentation, Release 20150714.01 Welcome to the documentation of EasyBuild, a software build and installation framework that allows you to manage (scientific) software on High Performance Computing (HPC) systems in an efficient way. This documentation is intended for EasyBuild version 2.2.0dev, and was last rebuilt on Tue, 14 Jul 2015 12:47:11. Contents 1 EasyBuild Documentation, Release 20150714.01 2 Contents CHAPTER 1 Introductory topics 1.1 What is EasyBuild? EasyBuild is a software build and installation framework that allows you to manage (scientific) software on High Performance Computing (HPC) systems in an efficient way. It is motivated by the need for a tool that combines the following features: •a flexible framework for building/installing (scientific) software • fully automates software builds • divert from the standard configure / make / make install with custom procedures • allows for easily reproducing previous builds • keep the software build recipes/specifications simple and human-readable • supports co-existence of versions/builds via dedicated installation prefix and module files • enables sharing with the HPC community (win-win situation) • automagic dependency resolution • retain logs for traceability of the build processes Some key features of EasyBuild: • build & install (scientific) software fully autonomously – also interactive installers, code patching, generating module file, ... • easily configurable: config file/environment/command line – including aspects like module naming scheme • thorough logging and archiving (see Understanding EasyBuild logs) – entire build process is logged thoroughly, logs are stored in install directory; – easyconfig file used for build is archived (install directory + file/svn/git repo) • automatic dependency resolution (see Enabling dependency resolution, –robot / -r and –robot-paths) – build entire software stack with a single command, using --robot • building software in parallel • robust and thoroughly tested code base, fully unit-tested before each release • thriving, growing community Take a look at our HUST‘14 workshop paper Modern Scientific Software Management Using EasyBuild and Lmod (download PDF here) and use that as a reference in case you present academic work mentioning EasyBuild. 3 EasyBuild Documentation, Release 20150714.01 1.2 Concepts and terminology EasyBuild consists of a collection of Python modules and packages that interact with each other, dynamically picking up additional Python modules as needed for building and installing a (stack of) software package(s) specified via simple specification files. Or, in EasyBuild terminology: the EasyBuild framework leverages easyblocks to automatically build and install software using particular compiler toolchains, as specified by one or multiple easyconfig files. 1.2.1 EasyBuild framework The EasyBuild framework embodies the core of the tool, providing functionality commonly needed when in- stalling scientific software on HPC systems. For example, it deals with downloading, unpacking and patching of sources, loading module files for dependencies, setting up the build environment, autonomously running (interac- tive) shell commands, creating module files that match the specification files, etc. Included in the framework is an abstract implementation of a software build and install procedure, which is split up into different steps: • unpacking sources • configuration • build • installation • module generation • etc. Most of these steps, i.e., the ones which are generally more-or-less analogous across different software packages, have appropriate (default) implementations. The only exceptions are the configuration, build and installation steps that are purposely left unimplemented (since there is no common procedure for them). Each of the steps can be tweaked and steered via different parameters known to the framework, for which values are either obtained from the provided specification files or set to reasonable default values. See Easyconfig files. In EasyBuild version 2.2.0dev the framework source code consists of about 19000 lines of code, organized across about 125 Python modules in roughly a dozen Python package directories, next to almost 7000 lines of code for tests. This provides some notion of the size of the EasyBuild framework and the amount of supporting function- ality it has to offer. 1.2.2 Easyblocks The implementation of a particular software build and install procedure is done in a Python module, which is aptly referred to as an easyblock. Each easyblock ties in with the framework API by defining (or extending/replacing) one or more of the step func- tions that are part of the abstract procedure used by the EasyBuild framework. Easyblocks typically heavily rely on the supporting functionality provided by the framework, for example for (autonomously) executing (interactive) shell commands and obtaining the command’s output and exit code. A distinction is made between software-specific and generic easyblocks. Software-specific easyblocks implement a build and install procedure which is entirely custom to one particular software package (e.g., WRF), while generic easyblocks implement a procedure using standard tools (e.g., CMake). Since easyblocks are implemented in an object-oriented scheme, the step methods implemented by a particular easyblock can be reused in others via inheritance, enabling code reuse across build procedure implementations. For each software package being built, the EasyBuild framework will determine which easyblock should be used, based on the name of the software package or the value of the easyblock specification parameter. In case an easyblock specification is not provided and no (software-specific) easyblock matching the software name could 4 Chapter 1. Introductory topics EasyBuild Documentation, Release 20150714.01 be found, a fallback mechanism will resort to using the generic ConfigureMake easyblock, which implements the common configure / make / make install procedure. The most recent release of EasyBuild (version 2.2.0dev) includes 139 software-specific easyblocks and 20 generic easyblocks (see also List of easyblocks), providing support for automatically installing a wide range of software packages. Examples range from fairly easy-to-build programs like gzip, other basic tools like compilers, various MPI stacks and commonly used libraries, primarily for x86_64 architecture systems, to large scientific software packages that are notorious for their involved and tedious install procedures, such as: CP2K, NWChem, Open- FOAM, QuantumESPRESSO, WRF. 1.2.3 Toolchains EasyBuild employs so-called compiler toolchains or, simply toolchains for short, which are a major concept in handling the build and installation processes. A typical toolchain consists of