https://doi.org/10.5194/gmd-2020-100 Preprint. Discussion started: 11 August 2020 c Author(s) 2020. CC BY 4.0 License. ESM-Tools Version 4.0: A modular infrastructure for stand-alone and coupled Earth System Modelling (ESM) Dirk Barbi1, Nadine Wieters1, Paul Gierz1, Fatemeh Chegini1, 3, Sara Khosravi2, and Luisa Cristini1 1Alfred Wegener Institute Helmholtz Center for Polar and Marine Research, Bremerhaven, Germany 2Alfred Wegener Institute Helmholtz Center for Polar and Marine Research, Potsdam, Germany 3Max Planck Institute for Meteorology, Hamburg, Germany Correspondence: Luisa Cristini ([email protected]) Abstract. Earth system and climate modelling involves the simulation of processes on a wide range of scales and within and across var- ious components of the Earth system. In practice, component models are often developed independently by different research groups and then combined using a dedicated coupling software. This procedure not only leads to a strongly growing number of 5 available versions of model components and coupled setups, but also to model- and system-dependent ways of obtaining and operating them. Therefore, implementing these Earth System Models (ESMs) can be challenging and extremely time consum- ing, especially for less experienced modellers, or scientists aiming to use different ESMs as in the case of inter-comparison projects. To assist researchers and modellers by reducing avoidable complexity, we developed the ESM-Tools software, which pro- 10 vides a standard way for downloading, configuring, compiling, running and monitoring different models - coupled ESMs and stand-alone models alike - on a variety of High Performance Computing (HPC) systems.1 With the ESM-Tools, the user is only required to provide a short script consisting of only the experiment specific definitions, while the software executes all the phases of a simulation in the correct order. The software, which is well documented and easy to install and use, currently supports four ocean models, three atmosphere models, two biogeochemistry models, an ice sheet model, an isostatic adjust- 15 ment model, a hydrology model and a land-surface model. ESM-Tools has been entirely re-coded in a high level programming language (Python) and provides researchers with an even more user-friendly interface for Earth system modelling lately. The ESM-Tools were developed within the framework of the project Advanced Earth System Model Capacity, supported by the Helmholtz Association. 1 Introduction 20 Earth System Models (ESMs) are widely used for studying past, present and future climates. They can include several model components (e.g. atmosphere, ocean, land and sea ice, land biosphere, ocean biogeochemistry) which are developed indepen- 1The ESM-Tools are equally applicable and helpful for stand-alone as for coupled models. In fact, the ESM-Tools are used as standard compile and runtime infrastructure for FESOM2, and currently also applied for ECHAM and ICON standalone simulations. As coupled ESMs are technically the more challenging tasks, we will focus on coupled setups, always implying that stand-alone models can benefit in the same way. 1 https://doi.org/10.5194/gmd-2020-100 Preprint. Discussion started: 11 August 2020 c Author(s) 2020. CC BY 4.0 License. dently by different research groups. Various ESMs have been developed in recent years by using different model components and couplers. Since the ESMs are usually written for a specific purpose, they often lack modularity. Therefore, in order to be able to apply these models, the users need to acquire knowledge of various model and system dependent parameters. This often 25 proves time consuming and challenging for scientists who are new to numerical modelling or are approaching a numerical model they are not familiar with. Furthermore, the lack of modularity in coupled systems can make it very inefficient when it comes to replacing certain model components. This issue can be further enhanced by lack of training in software development, often (but not always) amongst early career scientists, making the procedure non-trivial and error prone. To address these issues, we developed the ESM-Tools — a software that considerably reduces the difficulty of applying ESMs, by providing a modular 30 external modelling infrastructure that allows scientists to work with the coupled setup in a very intuitive and straightforward manner. The complete workflow for ESM applications includes tasks such as obtaining and compiling all source codes, managing input data and configuration files for model setup, submitting the executable to a multi-processor system, monitoring and logging the process and managing/post-processing output data. These tasks require knowledge of the parameters used by the 35 ESM such as build environment, configuration files (e.g. Fortran namelists) and I/O structure as well as knowledge of the specific High Performance Computing (HPC) environment (e.g. installed libraries and batch system). This often results in an ESM modeller having to deal with numerous, often unnecessary, technical issues. The ESM-Tools software provides standard solutions to typical tasks occurring within the workflow of Earth system modelling, such as calendar operations, data post- processing and monitoring, sanity checks, sorting and archiving of output, and script-based coupling. The ESM-Tools facilitate 40 Earth system modelling by providing a standardized framework to download, configure, compile, run, and analyze/monitor a variety of ESMs on various HPC systems. A variety of software and tools have been developed to assist Earth system modellers with running coupled models. However, most of them have been developed for specific purposes and lack adaptability. – Probably closest to the ESM-Tools, both by functionality and code design, is the ScriptEngine2 simulation task generating 45 software developed at SMHI primarily for the EC-Earth community. Indeed, ScriptEngine and ESM-Tools both use a combination of YAML configuration files and Python methods, but while the ESM-Tools use YAML to store the information about models, setups etc., ScriptEngine goes more into the direction of defining a Domain Specific Language (DSL), allowing the user to describe commands and simple routines in YAML format. This approach also is very elegant and leads to a simple and user-friendly way to formulate what used to be run scripts. To our knowledge, it is not in an 50 advanced state yet, and only contains functions for the EC-Earth model at this point. Also to our understanding it does not contain download and compile functionality, as well as a natural concept of modularity of model components. We hope to be able to cooperate with the ScriptEngine developers in the future, as both software package can be seen as natural extensions of each other. 2https://pypi.org/project/scriptengine 2 https://doi.org/10.5194/gmd-2020-100 Preprint. Discussion started: 11 August 2020 c Author(s) 2020. CC BY 4.0 License. – The Modular Earth Submodel System, MESSy3, has been developed as an infrastructure with generalized interfaces for 55 implementation of ESMs (Jöckel et al., 2005). The focus of MESSy, even though it comes with its own scripts and tools for compiling and running, is internal infrastructure, meaning the definition of reusable codes ("building blocks", "dwarfs", ...) as operators, automatic management of memory layout, and internal 3-D coupling of fields through all compartents of a coupled setup. MESSy and ESM-Tools don’t interfere at all, as one is for the internal, and the other for external infrastructure, and can in fact be used together. We already profit from a frequent exchange and cooperation 60 with the MESSy developers in the context of the Helmholtz-project ESM, and have implemented first steps towards integrating full MESSy support into the ESM-Tools, hoping to complete this process by release 5.0 (planned for October 2020). – The Modular System for Shelves and Coasts, MOSSCO4 provides a modular system for domain and process coupling of applications in the coastal ocean (Lemmen et al., 2017). It is designed to enable integrated regional coastal modelling 65 and is targeted towards the coupling of model components with different orders of magnitude of spatial and temporal resolution. The framework allows for the seamless replacement of individual model components. In contrast to the ESM-Tools, MOSSCO is a coupling framework rather than a framework for building and running ESMs. – The Make Experiments tools (Mkexp5) contain a set of tools for preparing experiments with the Earth system models developed at the Max-Planck Institute for Meteorology. It was primarily dedicated to generating run- and postprocessing 70 scripts for the MPI-ESM coupled Earth System Model from configuration files. One main difference between Mkexp and ESM-Tools is that the direct output of Mkexp are lengthy shell scripts, which are then submitted by the user to HPC systems. Even though Mkexp was successfully adapted to other coupled ESMs, lacks in our opinion the needed modularity required for optimal coupling of ESMs. The main problem of Mkexp seems to be that modellers don’t use it as intended though, distributing and editing the very long and unintuitive shell scripts, rather than generating new ones. 75 Out of this experience we decided that the ESM-Tools should not generate (complex) scripts, but rather interpret short and simple ones. – ESM-Tools should not be confused with the Earth System Model Evaluation Tool (ESMValTool6) which is a community diagnostics and performance metrics tool, used to compare one or several models against observations or their previous versions (Righi et al.; Eyring et al., 2015). ESM-Tools and ESMValTool are not related. 80 To our knowledge, there currently is no modular infrastructure that assists modellers in operating ESMs, incorporates a variety of stand-alone and coupled systems, is extendable and flexible, and is open to a larger community of researchers. The ESM- Tools emphasize the needs of researchers for such a software and fills this gap by fulfilling the specified criteria.