Reframe Documentation Release 3.8.2
Total Page：16
File Type：pdf, Size：1020Kb
ReFrame Documentation Release 3.8.2 CSCS Sep 22, 2021 TABLE OF CONTENTS 1 Use Cases 3 2 Publications 5 2.1 Getting Started..............................................5 2.2 ReFrame Tutorials............................................7 2.3 Configuring ReFrame for Your Site................................... 90 2.4 Advanced Topics............................................. 104 2.5 Use Cases................................................. 118 2.6 Migrating to ReFrame 3......................................... 121 2.7 ReFrame Manuals............................................ 124 Python Module Index 237 Index 239 i ii ReFrame Documentation, Release 3.8.2 ReFrame is a high-level framework for writing regression tests for HPC systems. The goal of the framework is to abstract away the complexity of the interactions with the system, separating the logic of a regression test from the low- level details, which pertain to the system configuration and setup. This allows users to write easily portable regression tests, focusing only on the functionality. Regression tests in ReFrame are simple Python classes that specify the basic parameters of the test. The framework will load the test and will send it down a well-defined pipeline that will take care of its execution. The stages of this pipeline take care of all the system interaction details, such as programming environment switching, compilation, job submission, job status query, sanity checking and performance assessment. ReFrame also offers a high-level and flexible abstraction for writing sanity and performance checks for your regression tests, without having to care about the details of parsing output files, searching for patterns and testing against reference values for different systems. Writing system regression tests in a high-level modern programming language, like Python, poses a great advantage in organizing and maintaining the tests. Users can create their own test hierarchies or test factories for generating multiple tests at the same time and they can also customize them in a simple and expressive way. Finally, ReFrame offers a powerful and efficient runtime for running and managing the execution of tests, as well as integration with common logging facilities, where ReFrame can send live data from currently running performance tests. TABLE OF CONTENTS 1 ReFrame Documentation, Release 3.8.2 2 TABLE OF CONTENTS CHAPTER ONE USE CASES A pre-release of ReFrame has been in production at the Swiss National Supercomputing Centre since early December 2016. The first public release was in May 2017 and it is being actively developed since then. Several HPC cen- ters around the globe have adopted ReFrame for testing and benchmarking their systems in an easy, consistent and reproducible way. You can read a couple of use cases here. 3 ReFrame Documentation, Release 3.8.2 4 Chapter 1. Use Cases CHAPTER TWO PUBLICATIONS • Slides [pdf]@ 6th EasyBuild User Meeting 2021. • Slides [pdf]@ 5th EasyBuild User Meeting 2020. • Slides [pdf]@ HPC System Testing BoF, SC’19. • Slides [pdf]@ HUST 2019, SC’19. • Slides [pdf]@ HPC Knowledge Meeting ‘19. • Slides [pdf]& Talk@ FOSDEM’19. • Slides [pdf]@ 4th EasyBuild User Meeting. • Slides [pdf]@ HUST 2018, SC’18. • Slides [pdf]@ CSCS User Lab Day 2018. • Slides [pdf]@ HPC Advisory Council 2018. • Slides [pdf]@ SC17. • Slides [pdf]@ CUG 2017. 2.1 Getting Started 2.1.1 Requirements • Python 3.6 or higher. Python 2 is not supported. • The required Python packages are the following: archspec==0.1.2 argcomplete==1.12.3 coverage==5.5 importlib_metadata==4.0.1; python_version<'3.8' jsonschema==3.2.0 lxml==4.6.3 pytest==6.2.3 pytest-forked==1.3.0 pytest-parallel==0.1.0 PyYAML==5.4.1 requests==2.25.1 semver==2.13.0 setuptools==56.0.0 (continues on next page) 5 ReFrame Documentation, Release 3.8.2 (continued from previous page) wcwidth==0.2.5 #+pygelf%pygelf==0.4.0 Note: Changed in version 3.0: Support for Python 3.5 has been dropped. 2.1.2 Getting the Framework Stable ReFrame releases are available through different channels. Spack ReFrame is available as a Spack package: spack install reframe There are the following variants available: • +docs: This will install the man pages of ReFrame. • +gelf: This will install the bindings for handling Graylog log messages. EasyBuild ReFrame is available as an EasyBuild package: eb ReFrame-VERSION.eb -r This will install the man pages as well as the Graylog bindings. PyPI ReFrame is available as a PyPI package: pip install reframe-hpc This is a bare installation of the framework. It will not install the documentation, the tutorial examples or the bindings for handling Graylog log messages. Github Any ReFrame version can be very easily installed directly from Github: pushd /path/to/install/prefix git clone -q --depth1 --branch VERSION_TAG https://github.com/eth-cscs/reframe.git pushd reframe&& ./bootstrap.sh&& popd export PATH=$(pwd)/bin:$PATH popd 6 Chapter 2. Publications ReFrame Documentation, Release 3.8.2 The VERSION_TAG is the version number prefixed by v, e.g., v3.5.0. The ./bootstrap.sh script will fetch ReFrame’s requirements under its installation prefix. It will not set the PYTHONPATH, so it will not affect the user’s Python installation. The ./bootstrap.sh has two additional variant options: • +docs: This will also build the documentation. • +pygelf: This will install the bindings for handling Graylog log messages. Note: New in version 3.1: The bootstrap script for ReFrame was added. For previous ReFrame versions you should install its requirements using pip install -r requirements.txt in a Python virtual environment. 2.1.3 Enabling auto-completion New in version 3.4.1. You can enable auto-completion for ReFrame by sourcing in your shell the corresponding script in <install_prefix>/share/completions/reframe.<shell>. Auto-completion is supported for Bash, Tcsh and Fish shells. Note: Changed in version 3.4.2: The shell completion scripts have been moved under share/completions/. 2.1.4 Where to Go from Here The easiest way to start with ReFrame is to go through Tutorial 1: Getting Started with ReFrame, which will guide you step-by-step in both writing your first tests and in configuring ReFrame. The Configuring ReFrame for Your Site page provides more details on the basic configuration aspects of ReFrame. Advanced Topics explain different aspects of the framework whereas ReFrame Manuals provide complete reference guides for the command line interface, the configuration parameters and the programming APIs for writing tests. 2.2 ReFrame Tutorials 2.2.1 Tutorial 1: Getting Started with ReFrame New in version 3.1. This tutorial will give you a first overview of ReFrame and will acquaint you with its basic concepts. We will start with a simple “Hello, World!” test running with the default configuration and we will expand the example along the way. We will also explore performance tests and port our tests to an HPC cluster. The examples of this tutorial can be found under tutorials/basics/. 2.2. ReFrame Tutorials 7 ReFrame Documentation, Release 3.8.2 Getting Ready All you need to start off with this tutorial is to have installed ReFrame. If you haven’t done so yet, all you need is Python 3.6 and above and to follow the steps below: git clone https://github.com/eth-cscs/reframe.git cd reframe ./bootstrap.sh ./bin/reframe -V We’re now good to go! The “Hello, World!” test As simple as it may sound, a series of “naive” “Hello, World!” tests can reveal lots of regressions in the programming environment of HPC clusters, but the bare minimum of those also serves perfectly the purpose of starting this tutorial. Here is its C version: cat tutorials/basics/hello/src/hello.c #include <stdio.h> int main() { printf("Hello, World!\n"); return 0; } And here is the ReFrame version of it: cat tutorials/basics/hello/hello1.py import reframe as rfm import reframe.utility.sanity as sn @rfm.simple_test class HelloTest(rfm.RegressionTest): valid_systems=[' *'] valid_prog_environs=[' *'] sourcepath='hello.c' @sanity_function def assert_hello(self): return sn.assert_found(r'Hello, World\!', self.stdout) Regression tests in ReFrame are specially decorated classes that ultimately derive from RegressionTest. The @simple_test decorator registers a test class with ReFrame and makes it available to the framework. The test variables are essentially attributes of the test class and can be defined directly in the class body. Each test must always set the valid_systems and valid_prog_environs attributes. These define the systems and/or system parti- tions that this test is allowed to run on, as well as the programming environments that it is valid for. A programming environment is essentially a compiler toolchain. We will see later on in the tutorial how a programming environment can be defined. The generic configuration of ReFrame assumes a single programming environment named builtin which comprises a C compiler that can be invoked with cc. In this particular test we set both these attributes to ['*'], essentially allowing this test to run everywhere. 8 Chapter 2. Publications ReFrame Documentation, Release 3.8.2 A ReFrame test must either define an executable to execute or a source file (or source code) to be compiled. In this example, it is enough to define the source file of our hello program. ReFrame knows the executable that was produced and will use that to run the test. Finally, every regression test must always decorate a member function as the test’s @sanity_function. This decorated function is converted into a lazily evaluated expression that asserts the sanity of the test. In this particular case, the specified sanity function checks that the executable has produced the desired phrase into the test’s standard output stdout. Note that ReFrame does not determine the success of a test by its exit code. Instead, the assessment of success is responsibility of the test itself. Before running the test let’s inspect the directory structure surrounding it: tutorials/basics/hello hello1.py src hello.c Our test is hello1.py and its resources, i.e., the hello.c source file, are located inside the src/ subdirectory.