EasyBuild Documentation Release 20210907.0 Ghent University Tue, 07 Sep 2021 08:55:25 Contents 1 What is EasyBuild? 3 2 Concepts and terminology 5 2.1 EasyBuild framework..........................................5 2.2 Easyblocks................................................6 2.3 Toolchains................................................7 2.3.1 system toolchain.......................................7 2.3.2 dummy toolchain (DEPRECATED) ..............................7 2.3.3 Common toolchains.......................................7 2.4 Easyconfig files..............................................7 2.5 Extensions................................................8 3 Typical workflow example: building and installing WRF9 3.1 Searching for available easyconfigs files.................................9 3.2 Getting an overview of planned installations.............................. 10 3.3 Installing a software stack........................................ 11 4 Getting started 13 4.1 Installing EasyBuild........................................... 13 4.1.1 Requirements.......................................... 14 4.1.2 Using pip to Install EasyBuild................................. 14 4.1.3 Installing EasyBuild with EasyBuild.............................. 17 4.1.4 Dependencies.......................................... 19 4.1.5 Sources............................................. 21 4.1.6 In case of installation issues. .................................. 22 4.2 Configuring EasyBuild.......................................... 22 4.2.1 Supported configuration types................................. 22 4.2.2 Overview of current configuration (--show-config, --show-full-config).... 27 4.2.3 Available configuration settings................................ 27 4.3 Common toolchains........................................... 34 4.3.1 Definition and motivation.................................... 34 4.3.2 Versioning scheme for common toolchains........................... 35 4.3.3 Update cycle for common toolchains.............................. 36 4.3.4 Overview of common toolchains................................ 36 4.3.5 Customizing common toolchains................................ 37 4.3.6 More information about toolchains............................... 37 i 5 Basic usage topics 39 5.1 Using the EasyBuild command line................................... 39 5.1.1 Specifying builds........................................ 39 5.1.2 Commonly used command line options............................ 42 5.2 Writing easyconfig files: the basics................................... 51 5.2.1 What is an easyconfig (file)?.................................. 52 5.2.2 Available easyconfig parameters................................ 53 5.2.3 Mandatory easyconfig parameters............................... 53 5.2.4 Common easyconfig parameters................................ 53 5.2.5 Tweaking existing easyconfig files............................... 65 5.2.6 Dynamic values for easyconfig parameters........................... 65 5.2.7 Version-specific documentation relevant to easyconfigs.................... 65 5.2.8 Contributing easyconfigs.................................... 66 5.3 Understanding EasyBuild logs...................................... 66 5.3.1 Basic information........................................ 66 5.3.2 Navigating log files....................................... 67 6 Advanced usage topics 69 6.1 Archived easyconfigs........................................... 69 6.1.1 Toolchain deprecation...................................... 69 6.1.2 Using --consider-archived-easyconfigs ..................... 70 6.2 Backing up of existing modules (--backup-modules)....................... 70 6.2.1 Disabling automatic backup of modules............................ 71 6.2.2 Example............................................. 71 6.3 Generating container recipes & images................................. 72 6.3.1 Requirements.......................................... 73 6.3.2 Usage.............................................. 73 6.3.3 Configuration.......................................... 81 6.3.4 ‘Stacking’ container images.................................. 83 6.3.5 Seeding in source files for container build process....................... 84 6.4 Contributing............................................... 84 6.4.1 How to contribute........................................ 85 6.4.2 Pull requests........................................... 87 6.4.3 Review process for contributions................................ 92 6.5 Controlling compiler optimization flags................................. 95 6.5.1 Controlling target architecture specific optimizations via --optarch ............ 96 6.6 EasyBuild on Cray............................................ 98 6.6.1 Test systems........................................... 98 6.6.2 EasyBuild toolchains...................................... 98 6.6.3 What works already?...................................... 99 6.6.4 Required EasyBuild configuration............................... 99 6.6.5 Major supported/tested applications.............................. 100 6.7 Detection of loaded modules....................................... 101 6.7.1 Motivation............................................ 101 6.7.2 Detection mechanism...................................... 102 6.7.3 Action to take if loaded modules are detected......................... 102 6.7.4 Allowing particular loaded modules.............................. 104 6.7.5 Checking of $EBROOT* environment variables........................ 104 6.8 Local variables in easyconfig files.................................... 106 6.8.1 Motivation & context...................................... 106 6.8.2 Changes in EasyBuild v4.0 w.r.t. local variables in easyconfig files.............. 107 6.8.3 Recommended naming scheme for local variables in easyconfig files............. 107 6.8.4 Warning for local variables that do not follow the recommended naming scheme....... 107 ii 6.8.5 Specifying what should be done when non-confirming local variables are found via --local-var-naming-check .............................. 108 6.8.6 Renaming local variables to match the recommended naming scheme using eb --fix-deprecated-easyconfigs ........................... 108 6.9 Using an index to speed up searching for easyconfigs.......................... 109 6.9.1 Creating an index (--create-index)........................... 110 6.9.2 Updating an existing index (--create-index --force)................ 110 6.9.3 Using index files........................................ 111 6.9.4 Ignoring indices (--ignore-index)............................ 111 6.9.5 Controlling how long the index is valid (--index-max-age)............... 111 6.9.6 Index included with EasyBuild releases............................ 111 6.9.7 Should I create an index?.................................... 112 6.10 Easystack files.............................................. 112 6.10.1 The basics............................................ 112 6.10.2 Usage.............................................. 112 6.10.3 Structure of an easystack file.................................. 113 6.10.4 To be developed......................................... 114 6.11 Experimental features.......................................... 114 6.12 Extended dry run............................................. 114 6.12.1 Important notes......................................... 115 6.12.2 Overview of dry run mechanism................................ 116 6.12.3 Guidelines for easyblocks.................................... 123 6.12.4 Example output......................................... 127 6.13 Hooks................................................... 127 6.13.1 What are hooks?........................................ 127 6.13.2 Configuring EasyBuild to use your hook implementations................... 128 6.13.3 Available hooks......................................... 128 6.13.4 Implementing hooks...................................... 129 6.13.5 Caveats............................................. 130 6.13.6 Examples of hook implementations.............................. 131 6.14 Implementing easyblocks........................................ 132 6.14.1 The basics............................................ 133 6.14.2 Easyblocks vs easyconfigs................................... 134 6.14.3 Naming scheme for easyblocks................................. 134 6.14.4 Structure of an easyblock.................................... 136 6.14.5 Deriving from existing (generic) easyblocks.......................... 136 6.14.6 Specific aspects of easyblocks................................. 136 6.14.7 Using new/custom easyblocks................................. 140 6.14.8 Testing easyblocks....................................... 141 6.14.9 Use case: an easyblock for Tensorflow............................. 141 6.15 Including additional Python modules (--include-*)........................ 141 6.15.1 General aspects of --include-* options.......................... 141 6.15.2 Including additional easyblocks (--include-easyblocks)............... 142 6.15.3 Including additional module naming schemes (--include-module-naming-schemes) 143 6.15.4 Including additional toolchains (--include-toolchains)................ 143 6.16 Integration with GitHub......................................... 144 6.16.1 Requirements.......................................... 145 6.16.2 Configuration.......................................... 145 6.16.3 Checking status of GitHub integration (--check-github)................. 147 6.16.4 Using easyconfigs from pull requests (--from-pr)....................