WRF-Cmake: Integrating Cmake Support Into the Advanced Research WRF (ARW) Modelling System

WRF-CMake: integrating CMake support into the Advanced Research WRF (ARW) modelling system

M. Riechert1 and D. Meyer1

1 Independent scholar DOI: 10.21105/joss.01468 Software • Review Summary • Repository • Archive The Weather Research and Forecasting model (WRF1) model (Skamarock et al., 2019) is an Submitted: 14 May 2019 atmospheric modelling system widely used in operational forecasting and atmospheric research Published: 09 September 2019 (Powers et al., 2017). WRF is released as a free and open-source software and officially sup- License ported to run on Unix and Unix-like operating systems and on several hardware architectures Authors of papers retain from single-core computers to multi-core supercomputers. Its current build system relies on copyright and release the work several bespoke hand-written Makefiles, and Perl and Shell scripts that have been supported under a Creative Commons and extended during the many years of development. Attribution 4.0 International License (CC-BY). The use of build script generation tools, that is, tools that generate files for native build systems from specifications in a high-level language, rather than manually maintaining build scripts for different environments and platforms, can be useful to reduce code duplication and to minimize issues with code not building correctly (Hoffman, Cole, & Vines, 2009), to make software more accessible to a broader audience, and the support less expensive (Heroux & Willenbring, 2009). As such, a common build script generation tool is CMake. Today, CMake is employed in several projects such as HDF5, EnergyPlus, and ParaView to build modern software written in C, C++, and Fortran in high performance computing (HPC) environments and, by CERN, to allow users to easily set-up and build several million lines of C++ and Python code used in the offline software of the ATLAS experiment at the Large Hadron Collider (LHC) (Elmsheuser, Krasznahorkay, Obreshkov, & Undrus, 2017). WRF-CMake aims at helping model developers and end-users by adding CMake support to the latest version of WRF and the WRF Processing System (WPS), while coexisting with the existing build set-up. The main goals of WRF-CMake are to simplify the build process involved in developing and building WRF and WPS, add support for automated testing using continuous integration (CI), and the generation of pre-built binary releases for Linux, macOS, and Windows thus allowing non-expert users to get started with their simulations in a few minutes, or integrating WRF and WPS into other software (see, for example, the GIS4WRF project (Meyer & Riechert, 2019)). The WRF-CMake project provides model developers, code maintainers, and end-users wishing to build WRF and WPS on their system several advantages such as robust incremental rebuilds, dependency analysis of Fortran code, flexible library dependency discovery, automatic construction of compiler command-lines based on the detected compiler, and integrated support for MPI and OpenMP. Furthermore, by using a single language to control the build, CMake removes the need to write and support several hand-written Makefiles, and Perl and Shell scripts. The current WRF-CMake set-up onGitHub offers model developers and code maintainers an automated testing infrastructure (see Testing) for Linux, macOS, and Windows, and allows end-users to directly download pre-built binaries

1By WRF, we specifically mean the Advanced Research WRF (ARW). The Non-hydrostatic Mesoscale Model (NMM) dynamical core, WRF-DA, WRFPLUS, WRF-Chem, and WRF-Hydro are not currently supported in WRF-CMake.

Riechert et al., (2019). WRF-CMake: integrating CMake support into the Advanced Research WRF (ARW) modelling system. Journal of Open 1 Source Software, 4(41), 1468. https://doi.org/10.21105/joss.01468 for common configurations and architectures from the project’s website (experimental). WRF- CMake is available as a free and open-source project on GitHub at https://github.com/ WRF-CMake and currently includes CMake support for the main Advanced Research WRF (ARW) core and WPS.

Testing

A fundamental aspect of software development is testing. Ideally, model components should be tested individually and under several testing methodologies (Feathers, 2004). Here, however, as the WRF framework does not offer a way to unit test its components, we instead run separate build and regression tests to evaluate the effects of our changes. While build tests are used to check the absence of compilation errors, regression tests are used to estimate the size of simulation errors resulting from our change. Build tests are performed for all supported build variants (Table 1) using CI services at every commit. As noted by Hodyss and Majumdar (2007), and Geer (2016), the high sensitivity to initial conditions of dynamical systems, such as the ones used in weather models, can lead to large differences in skill between any two forecasts. It is this high sensitivity to initial conditions that can obscure the source of model error, whether this originates from a change in compiler or architecture, an actual coding error, or indeed, the intrinsic nature of the dynamical system employed.

Table 1: Build variants used in build and regression tests. Make: original WRF build system files, CMake: this paper; Debug: compiler optimizations disabled, Release: enabled; serial: single processor, dmpar: multiple with distributed memory (MPI), smpar: multiple with shared memory (OpenMP), dm_sm: multiple with MPI and OpenMP.

Variant OS Linux, macOS, Windows Build tool Make, CMake Build type Debug, Release Mode serial, dmpar, smpar, dm_sm

As a result, the impact of our changes are evaluated using the range-normalized relative percentage error (δx) and range-normalized root-mean-square percentage error (NRMSPE; Appendix A). These are computed per domain for all grid points, and for all vertical levels. The errors are assessed by (a) comparing the outputs of prognostic-variable outputs (Table 2) from WRF (Make) against those from WRF-CMake and (b) comparing the outputs for all build variants (for both Make and CMake) against a reference build variant defined as Linux/Make/Debug/serial. These tests are then run for all supported build variants (Table 1) using the WRF-CMake Automated Testing Suite (WATS), and a subset of namelists2 from the official WRF Testing Framework, using CI services at major code changes (e.g. before merging pull requests), and for 1 hour of simulation time, to constrain computing resources. Here, we report summary results for the domain showing the greatest error (i.e. innermost; domain 2) after simulating 60 minutes. Values of δx are aggregated for all quantities reported in Table 2 and referred to as δ.

2See https://github.com/WRF-CMake/wats/tree/master/cases/wrf

Riechert et al., (2019). WRF-CMake: integrating CMake support into the Advanced Research WRF (ARW) modelling system. Journal of Open 2 Source Software, 4(41), 1468. https://doi.org/10.21105/joss.01468 Table 2: WRF prognostic variables evaluated during regression tests.

Symbol Name Unit p Air pressure Pa ϕ Surface geopotential m2 s−2 θ Air potential temperature K u Zonal component of wind velocity m s−1 v Meridional component of wind velocity m s−1 w Vertical component of wind velocity m s−1

At the start of the simulation, the NRMSPE between WRF (Make) and WRF-CMake is zero (Appendix B, Figure 5), but small, when comparing WRF build variants (both Make and CMake) against a reference variant (Linux/Make/Debug/serial; Appendix B, Figure 6), thus suggesting an expected variability of outputs when running WRF across different platforms. After 60 minutes (simulation time), WRF-CMake produces, on average, small values of δ, with mean close to zero, and most of the error (99.8 %) between -0.05 and 0.05 % (Figure 1). On Linux, the only build variants showing no error are for Debug/serial and Debug/dmpar (Figure 1 and 2). For NRMSPE (Figure 2), values of w show to be the most sensitive, however, the largest errors are shown for all components of wind velocity, on both, Linux and macOS. Differences, in particular for Release build variants, most likely arise from an inconsistent use of compiler optimization options in WRF (Make) across its C and Fortran files, whereas in WRF-CMake, such options are centrally and consistently applied. Given that Debug/serial and Debug/dmpar show no error, we would expect the same to be true for the OpenMP variants Debug/smpar and Debug/dm_sm. Further investigation is required to establish the source of these differences. When comparing both Make and CMake versions against the reference build variant (i.e. Linux/Make/Debug/serial; Figure 3 and 4), the errors appear to be of equal, or greater, magnitude than those shown when comparing WRF (Make) against WRF-CMake for both δ (Figure 3) and NRMSPE (Figure 4), thus indicating that the variability across build variants may be more important and may also be an inherent feature of WRF. The choice of operating system has the greatest impact on both δ and NRMSPE (Figure 3 and 4) over compiler optimization strategies and build tool used. A change in build tool to CMake appears to produce values of δ and NRMSPE consistent with those obtained from versions of WRF built with the original build scripts3. The largest errors are shown for wind velocity and, specifically for u and w. Larger values of δ and NRMSPE between operating systems appear to be a general property of WRF (i.e. with/without CMake support) and should be investigated further.

3Comparison on Windows is not made as Windows support is only available in WRF-CMake.

Riechert et al., (2019). WRF-CMake: integrating CMake support into the Advanced Research WRF (ARW) modelling system. Journal of Open 3 Source Software, 4(41), 1468. https://doi.org/10.21105/joss.01468 macOS/Release/smpar

macOS/Release/serial

macOS/Release/dm_sm -1.3 1.3

macOS/Release/dmpar

macOS/Debug/smpar -4.2 3.4

macOS/Debug/serial

macOS/Debug/dm_sm -6.2 6.2

macOS/Debug/dmpar

Trial Linux/Release/smpar -3.6 2.7

Linux/Release/serial -3.6 2.7

Linux/Release/dm_sm -3.6 2.7

Linux/Release/dmpar -3.6 2.7

Linux/Debug/smpar -4.2 3.1

Linux/Debug/serial

Linux/Debug/dm_sm -3.4 5.1

Linux/Debug/dmpar

0.075 0.050 0.025 0.000 0.025 0.050 0.075 in %

Figure 1: WRF (Make) vs WRF-CMake: extended box plots of range-normalized relative percentage errors (δ) for the domain with highest errors only (domain 2) after 60 minutes (simulation time). Extended boxplots show minimum, maximum, median, and percentiles at [99.9, 99, 75, 25, 5, 1, 0.1].

macOS/Release/smpar 0 0 0 0 0 0

macOS/Release/serial 0 0 0 0 0 0

macOS/Release/dm_sm 0.000232 1.15e-05 0.000828 0.00511 0.00405 0.00318 0.020 macOS/Release/dmpar 0 0 0 0 0 0

macOS/Debug/smpar 0.000299 4.8e-05 0.000852 0.0019 0.00167 0.019

macOS/Debug/serial 0 0 0 0 0 0

0.015 macOS/Debug/dm_sm 0.000455 5.9e-05 0.00139 0.00682 0.00494 0.0246

macOS/Debug/dmpar 0 0 0 0 0 0

Trial Linux/Release/smpar 0.000308 4.34e-05 0.000834 0.00178 0.00156 0.0193 NRMSPE in % Linux/Release/serial 0.000308 4.34e-05 0.000834 0.00178 0.00156 0.0193 0.010

Linux/Release/dm_sm 0.000387 4.52e-05 0.00129 0.00647 0.00438 0.0196

Linux/Release/dmpar 0.000308 4.34e-05 0.000834 0.00178 0.00156 0.0193

Linux/Debug/smpar 0.000338 5.28e-05 0.000889 0.00196 0.00186 0.0205 0.005

Linux/Debug/serial 0 0 0 0 0 0

Linux/Debug/dm_sm 0.00043 5.82e-05 0.00128 0.00569 0.00444 0.0246

Linux/Debug/dmpar 0 0 0 0 0 0 0.000 p u v w Quantity

Figure 2: WRF (Make) vs WRF-CMake: range-normalized root mean-square percentage error (NRM- SPE) for the domain with highest errors only (domain 2) after 60 minutes (simulation time).

Riechert et al., (2019). WRF-CMake: integrating CMake support into the Advanced Research WRF (ARW) modelling system. Journal of Open 4 Source Software, 4(41), 1468. https://doi.org/10.21105/joss.01468 Windows/CMake/Release/smpar -17.9 39.5 Windows/CMake/Release/serial -17.9 39.5 Windows/CMake/Release/dm_sm -17.9 39.5 Windows/CMake/Release/dmpar -17.9 39.5 Windows/CMake/Debug/smpar -17.9 39.5 Windows/CMake/Debug/serial -17.9 39.5 Windows/CMake/Debug/dm_sm -17.9 39.5 Windows/CMake/Debug/dmpar -17.9 39.5 macOS/Make/Release/smpar -31.3 21.8 macOS/Make/Release/serial -31.3 21.8 macOS/Make/Release/dm_sm -31.3 21.8 macOS/Make/Release/dmpar -31.3 21.8 macOS/Make/Debug/smpar -20.2 15.2 macOS/Make/Debug/serial -20.2 15.2 macOS/Make/Debug/dm_sm -20.2 15.2 macOS/Make/Debug/dmpar -20.2 15.2 macOS/CMake/Release/smpar -31.3 21.8 macOS/CMake/Release/serial -31.3 21.8 macOS/CMake/Release/dm_sm -31.3 21.8 macOS/CMake/Release/dmpar -31.3 21.8 Trial macOS/CMake/Debug/smpar -20.2 15.2 macOS/CMake/Debug/serial -20.2 15.2 macOS/CMake/Debug/dm_sm -20.2 15.2 macOS/CMake/Debug/dmpar -20.2 15.2 Linux/Make/Release/smpar -3.9 2.8 Linux/Make/Release/serial -3.9 2.8 Linux/Make/Release/dm_sm -3.9 2.8 Linux/Make/Release/dmpar -3.9 2.8 Linux/Make/Debug/smpar -4.3 4.7 Linux/Make/Debug/dm_sm -3.9 4.1 Linux/Make/Debug/dmpar Linux/CMake/Release/smpar -3.9 2.8 Linux/CMake/Release/serial -3.9 2.8 Linux/CMake/Release/dm_sm -3.9 2.8 Linux/CMake/Release/dmpar -3.9 2.8 Linux/CMake/Debug/smpar -4.2 4.8 Linux/CMake/Debug/serial Linux/CMake/Debug/dm_sm -4.0 4.6 Linux/CMake/Debug/dmpar

0.6 0.4 0.2 0.0 0.2 0.4 0.6 in %

Figure 3: WRF (Make and CMake) vs reference build variant: extended box plots of range-normalized relative percentage errors (δ) against the reference build variant (Linux/Make/Debug/serial) for the domain with highest errors only (domain 2) after 60 minutes (simulation time). Extended boxplots show minimum, maximum, median, and percentiles at [99.9, 99, 75, 25, 5, 1, 0.1].

Windows/CMake/Release/smpar 0.000867 0.000315 0.00448 0.0308 0.0916 0.103 Windows/CMake/Release/serial 0.000874 0.000315 0.00447 0.0309 0.0916 0.103 Windows/CMake/Release/dm_sm 0.000863 0.000315 0.00449 0.0308 0.0916 0.103 Windows/CMake/Release/dmpar 0.000874 0.000315 0.00447 0.0309 0.0916 0.103 0.10 Windows/CMake/Debug/smpar 0.000861 0.000314 0.00447 0.0309 0.0916 0.103 Windows/CMake/Debug/serial 0.000872 0.000315 0.00446 0.0308 0.0916 0.103 Windows/CMake/Debug/dm_sm 0.00087 0.000314 0.00454 0.0313 0.0917 0.103 Windows/CMake/Debug/dmpar 0.000872 0.000315 0.00446 0.0308 0.0916 0.103 macOS/Make/Release/smpar 0.000909 0.000344 0.00497 0.0358 0.101 0.109 macOS/Make/Release/serial 0.000909 0.000344 0.00497 0.0358 0.101 0.109 macOS/Make/Release/dm_sm 0.00091 0.000344 0.00503 0.0363 0.101 0.109 0.08 macOS/Make/Release/dmpar 0.000909 0.000344 0.00497 0.0358 0.101 0.109 macOS/Make/Debug/smpar 0.000913 0.000328 0.00478 0.0346 0.0789 0.0999 macOS/Make/Debug/serial 0.000921 0.000328 0.00479 0.0346 0.0789 0.1 macOS/Make/Debug/dm_sm 0.000934 0.000328 0.00478 0.0347 0.079 0.101 macOS/Make/Debug/dmpar 0.000921 0.000328 0.00479 0.0346 0.0789 0.1 macOS/CMake/Release/smpar 0.000909 0.000344 0.00497 0.0358 0.101 0.109 macOS/CMake/Release/serial 0.000909 0.000344 0.00497 0.0358 0.101 0.109 0.06 macOS/CMake/Release/dm_sm 0.000914 0.000344 0.00502 0.0362 0.101 0.109 macOS/CMake/Release/dmpar 0.000909 0.000344 0.00497 0.0358 0.101 0.109 Trial macOS/CMake/Debug/smpar 0.000914 0.000328 0.00477 0.0346 0.0789 0.0996 macOS/CMake/Debug/serial 0.000921 0.000328 0.00479 0.0346 0.0789 0.1

macOS/CMake/Debug/dm_sm 0.000912 0.000327 0.00477 0.0348 0.079 0.0993 NRMSPE in % macOS/CMake/Debug/dmpar 0.000921 0.000328 0.00479 0.0346 0.0789 0.1 Linux/Make/Release/smpar 0.000339 5.08e-05 0.00103 0.00208 0.0018 0.0201 0.04 Linux/Make/Release/serial 0.000339 5.08e-05 0.00103 0.00208 0.0018 0.0201 Linux/Make/Release/dm_sm 0.000414 5.25e-05 0.00149 0.00672 0.0051 0.0204 Linux/Make/Release/dmpar 0.000339 5.08e-05 0.00103 0.00208 0.0018 0.0201 Linux/Make/Debug/smpar 0.00035 4.99e-05 0.000927 0.00203 0.00191 0.0212 Linux/Make/Debug/dm_sm 0.000387 4.44e-05 0.00113 0.00455 0.00335 0.0182 Linux/Make/Debug/dmpar 0 0 0 0 0 0 Linux/CMake/Release/smpar 0.000332 5e-05 0.000961 0.00194 0.00153 0.0188 0.02 Linux/CMake/Release/serial 0.000332 5e-05 0.000961 0.00194 0.00153 0.0188 Linux/CMake/Release/dm_sm 0.000398 5.14e-05 0.00127 0.00571 0.00381 0.019 Linux/CMake/Release/dmpar 0.000332 5e-05 0.000961 0.00194 0.00153 0.0188 Linux/CMake/Debug/smpar 0.0003 4.14e-05 0.00101 0.00203 0.00174 0.0192 Linux/CMake/Debug/serial 0 0 0 0 0 0 Linux/CMake/Debug/dm_sm 0.000435 5.49e-05 0.00114 0.00479 0.00386 0.0219 Linux/CMake/Debug/dmpar 0 0 0 0 0 0 0.00 p u v w Quantity

Figure 4: WRF (Make and CMake) vs reference build variant: range-normalized root mean-square percentage errors (NRMSPE) against the reference build variant (Linux/Make/Debug/serial) for the domain with highest errors only (domain 2) after 60 minutes (simulation time).

Riechert et al., (2019). WRF-CMake: integrating CMake support into the Advanced Research WRF (ARW) modelling system. Journal of Open 5 Source Software, 4(41), 1468. https://doi.org/10.21105/joss.01468 Concluding remarks

We introduce WRF-CMake as a modern replacement for the existing WRF build system. Its main goals are to simplify the build process involved in developing and building WRF and WPS, add support for automated testing using CI, and automate the generation of pre-built binary releases for Linux, macOS, and Windows. Results from regression tests indicate that, when evaluating outputs of prognostic variables, errors between WRF and WRF-CMake are generally small, or smaller, then errors originating from a change in optimization strategy (e.g. Debug, Release) or a change in platform (e.g. Linux to macOS). These larger errors appear to be a general property of WRF (i.e. with/without CMake support) and should be investigated further. Depending on feedback and general uptake by the community, future work may involve adding support for WRF-DA, WRFPLUS, WRF-Chem, and WRF-Hydro.

Acknowledgements

We thank A. J. Geer at the European Centre for Medium-Range Weather Forecasts (ECMWF) for the useful discussion and feedback concerning the topic of error growth in dynamical systems. We also thank the reviewers I. Beekman and A. Hilboll for their time and useful contributions to both paper and software.

Appendix A Statistics

The vector of range-normalized relative percentage error (δx) between two vectors x1 and x2 of paired quantities x1 and x2 is defined as:

x1 − x2 δx := × 100 %, (1) Rx1

where Rx1 is the range of x1. Similarly, the range-normalized root-mean-square percentage error (NRMSPE) is defined as:

RMSE NRMSPE := × 100 %, (2) Rx1

with the root-mean-square-error (RMSE) defined as: √ ∑ N (x − x )2 RMSE := i=1 1,i 2,i , (3) N

and N is the size of the vector.

Riechert et al., (2019). WRF-CMake: integrating CMake support into the Advanced Research WRF (ARW) modelling system. Journal of Open 6 Source Software, 4(41), 1468. https://doi.org/10.21105/joss.01468 Appendix B Supplementary figures

macOS/Release/smpar 0 0 0 0 0 0

macOS/Release/serial 0 0 0 0 0 0 0.08

macOS/Release/dm_sm 0 0 0 0 0 0

macOS/Release/dmpar 0 0 0 0 0 0

macOS/Debug/smpar 0 0 0 0 0 0 0.04 macOS/Debug/serial 0 0 0 0 0 0

macOS/Debug/dm_sm 0 0 0 0 0 0

macOS/Debug/dmpar 0 0 0 0 0 0 0.00 Trial Linux/Release/smpar 0 0 0 0 0 0 NRMSPE in % Linux/Release/serial 0 0 0 0 0 0

Linux/Release/dm_sm 0 0 0 0 0 0 0.04 Linux/Release/dmpar 0 0 0 0 0 0

Linux/Debug/smpar 0 0 0 0 0 0

Linux/Debug/serial 0 0 0 0 0 0

Linux/Debug/dm_sm 0 0 0 0 0 0 0.08

Linux/Debug/dmpar 0 0 0 0 0 0

p u v w Quantity

Figure 5: WRF (Make) vs WRF-CMake: range-normalized root mean-square percentage error (NRM- SPE) at 0 minutes (simulation time).

Windows/CMake/Release/smpar 2.2e-05 3e-06 4.94e-06 7.77e-06 7.44e-06 6.15e-06 Windows/CMake/Release/serial 2.2e-05 3e-06 4.94e-06 7.77e-06 7.44e-06 6.15e-06 Windows/CMake/Release/dm_sm 2.2e-05 3e-06 4.94e-06 7.77e-06 7.44e-06 6.15e-06 Windows/CMake/Release/dmpar 2.2e-05 3e-06 4.94e-06 7.77e-06 7.44e-06 6.15e-06 Windows/CMake/Debug/smpar 2.2e-05 3e-06 4.94e-06 7.77e-06 7.44e-06 6.15e-06 Windows/CMake/Debug/serial 2.2e-05 3e-06 4.94e-06 7.77e-06 7.44e-06 6.15e-06 0.000020 Windows/CMake/Debug/dm_sm 2.2e-05 3e-06 4.94e-06 7.77e-06 7.44e-06 6.15e-06 Windows/CMake/Debug/dmpar 2.2e-05 3e-06 4.94e-06 7.77e-06 7.44e-06 6.15e-06 macOS/Make/Release/smpar 2.35e-05 3e-06 4.94e-06 7.79e-06 7.43e-06 6.15e-06 macOS/Make/Release/serial 2.35e-05 3e-06 4.94e-06 7.79e-06 7.43e-06 6.15e-06 macOS/Make/Release/dm_sm 2.35e-05 3e-06 4.94e-06 7.79e-06 7.43e-06 6.15e-06 macOS/Make/Release/dmpar 2.35e-05 3e-06 4.94e-06 7.79e-06 7.43e-06 6.15e-06 macOS/Make/Debug/smpar 2.35e-05 3e-06 4.94e-06 7.79e-06 7.43e-06 6.15e-06 0.000016 macOS/Make/Debug/serial 2.35e-05 3e-06 4.94e-06 7.79e-06 7.43e-06 6.15e-06 macOS/Make/Debug/dm_sm 2.35e-05 3e-06 4.94e-06 7.79e-06 7.43e-06 6.15e-06 macOS/Make/Debug/dmpar 2.35e-05 3e-06 4.94e-06 7.79e-06 7.43e-06 6.15e-06 macOS/CMake/Release/smpar 2.35e-05 3e-06 4.94e-06 7.79e-06 7.43e-06 6.15e-06 macOS/CMake/Release/serial 2.35e-05 3e-06 4.94e-06 7.79e-06 7.43e-06 6.15e-06 macOS/CMake/Release/dm_sm 2.35e-05 3e-06 4.94e-06 7.79e-06 7.43e-06 6.15e-06 macOS/CMake/Release/dmpar 2.35e-05 3e-06 4.94e-06 7.79e-06 7.43e-06 6.15e-06 0.000012 Trial macOS/CMake/Debug/smpar 2.35e-05 3e-06 4.94e-06 7.79e-06 7.43e-06 6.15e-06 macOS/CMake/Debug/serial 2.35e-05 3e-06 4.94e-06 7.79e-06 7.43e-06 6.15e-06

macOS/CMake/Debug/dm_sm 2.35e-05 3e-06 4.94e-06 7.79e-06 7.43e-06 6.15e-06 NRMSPE in % macOS/CMake/Debug/dmpar 2.35e-05 3e-06 4.94e-06 7.79e-06 7.43e-06 6.15e-06 Linux/Make/Release/smpar 0 0 0 0 0 0 Linux/Make/Release/serial 0 0 0 0 0 0 0.000008 Linux/Make/Release/dm_sm 0 0 0 0 0 0 Linux/Make/Release/dmpar 0 0 0 0 0 0 Linux/Make/Debug/smpar 0 0 0 0 0 0 Linux/Make/Debug/dm_sm 0 0 0 0 0 0 Linux/Make/Debug/dmpar 0 0 0 0 0 0 Linux/CMake/Release/smpar 0 0 0 0 0 0 Linux/CMake/Release/serial 0 0 0 0 0 0 0.000004 Linux/CMake/Release/dm_sm 0 0 0 0 0 0 Linux/CMake/Release/dmpar 0 0 0 0 0 0 Linux/CMake/Debug/smpar 0 0 0 0 0 0 Linux/CMake/Debug/serial 0 0 0 0 0 0 Linux/CMake/Debug/dm_sm 0 0 0 0 0 0 Linux/CMake/Debug/dmpar 0 0 0 0 0 0 0.000000 p u v w Quantity

Figure 6: WRF (Make and CMake) vs reference build variant: range-normalized root mean-square percentage errors (NRMSPE) against the reference build variant (Linux/Make/Debug/serial) for the domain with highest errors only (domain 2) at 0 minutes (simulation time). Extended boxplots show minimum, maximum, median, and percentiles at [99.9, 99, 75, 25, 5, 1, 0.1].

Riechert et al., (2019). WRF-CMake: integrating CMake support into the Advanced Research WRF (ARW) modelling system. Journal of Open 7 Source Software, 4(41), 1468. https://doi.org/10.21105/joss.01468 References

Elmsheuser, J., Krasznahorkay, A., Obreshkov, E., & Undrus, A. (2017). Large scale software building with CMake in ATLAS. Journal of Physics: Conference Series, 898, 072010. doi:10. 1088/1742-6596/898/7/072010 Feathers, M. (2004). Working Effectively with Legacy Code (p. 456). New Jersey: Prentice Hall. Geer, A. J. (2016). Significance of changes in medium-range forecast scores. Tellus A: Dynamic Meteorology and Oceanography, 68(1), 30229. doi:10.3402/tellusa.v68.30229 Heroux, M. A., & Willenbring, J. M. (2009). Barely sufficient software engineering: 10 practices to improve your CSE software. In 2009 ICSE workshop on software engineering for computational science and engineering. IEEE. doi:10.1109/secse.2009.5069157 Hodyss, D., & Majumdar, S. J. (2007). The contamination of ‘data impact’ in global models by rapidly growing mesoscale instabilities. Quarterly Journal of the Royal Meteorological Society, 133(628), 1865–1875. doi:10.1002/qj.157 Hoffman, B., Cole, D., & Vines, J. (2009). Software process for rapid development ofHPC software using CMake. In 2009 DoD high performance computing modernization program users group conference. IEEE. doi:10.1109/hpcmp-ugc.2009.62 Meyer, D., & Riechert, M. (2019). Open source QGIS toolkit for the advanced research WRF modelling system. Environmental Modelling & Software, 112, 166–178. doi:10.1016/j. envsoft.2018.10.018 Powers, J. G., Klemp, J. B., Skamarock, W. C., Davis, C. A., Dudhia, J., Gill, D. O., Coen, J. L., et al. (2017). The Weather Research and Forecasting Model: Overview, System Efforts, and Future Directions. Bulletin of the American Meteorological Society, 98(8), 1717–1737. doi:10.1175/BAMS-D-15-00308.1 Skamarock, W. C., Klemp, J. B., Dudhia, J., Gill, D. O., Liu, Z., Berner, J., Wang, W., et al. (2019). A description of the advanced research wrf model version 4 (p. 145). NCAR Technical Note NCAR/TN-556+STR. doi:10.5065/1dfh-6p97

Riechert et al., (2019). WRF-CMake: integrating CMake support into the Advanced Research WRF (ARW) modelling system. Journal of Open 8 Source Software, 4(41), 1468. https://doi.org/10.21105/joss.01468