Geosci. Model Dev., 14, 821–842, 2021 https://doi.org/10.5194/gmd-14-821-2021 © Author(s) 2021. This work is distributed under the Creative Commons Attribution 4.0 License. Shyft v4.8: a framework for uncertainty assessment and distributed hydrologic modeling for operational hydrology John F. Burkhart1, Felix N. Matt1,2, Sigbjørn Helset2, Yisak Sultan Abdella2, Ola Skavhaug3, and Olga Silantyeva1 1Department of Geosciences, University of Oslo, Oslo, Norway 2Statkraft AS, Lysaker, Norway 3Expert Analytics AS, Oslo, Norway Correspondence: John F. Burkhart ([email protected]) Received: 12 February 2020 – Discussion started: 4 May 2020 Revised: 14 October 2020 – Accepted: 16 October 2020 – Published: 5 February 2021 Abstract. This paper presents Shyft, a novel hydrologic 1 Introduction modeling software for streamflow forecasting targeted for use in hydropower production environments and research. The software enables rapid development and implementa- Operational hydrologic modeling is fundamental to several tion in operational settings and the capability to perform dis- critical domains within our society. For the purposes of flood tributed hydrologic modeling with multiple model and forc- prediction and water resource planning, the societal benefits ing configurations. Multiple models may be built up through are clear. Many nations have hydrological services that pro- the creation of hydrologic algorithms from a library of well- vide water-related data and information in a routine manner. known routines or through the creation of new routines, each The World Meteorological Organization gives an overview defined for processes such as evapotranspiration, snow accu- of the responsibilities of these services and the products they mulation and melt, and soil water response. Key to the de- provide to society, including monitoring of hydrologic pro- sign of Shyft is an application programming interface (API) cesses, provision of data, water-related information includ- that provides access to all components of the framework ing seasonal trends and forecasts, and, importantly, decision (including the individual hydrologic routines) via Python, support services (World Meteorological Organization, 2006). while maintaining high computational performance as the Despite the abundantly clear importance of such opera- algorithms are implemented in modern C++. The API al- tional systems, implementation of robust systems that are lows for rapid exploration of different model configurations able to fully incorporate recent advances in remote sensing, and selection of an optimal forecast model. Several differ- distributed data acquisition technologies, high-resolution ent methods may be aggregated and composed, allowing di- weather model inputs, and ensembles of forecasts remains rect intercomparison of models and algorithms. In order to a challenge. Pagano et al.(2014) provide an extensive review provide enterprise-level software, strong focus is given to of these challenges, as well as the potential benefits afforded computational efficiency, code quality, documentation, and by overcoming some relatively simple barriers. The Hydro- test coverage. Shyft is released open-source under the GNU logic Ensemble Prediction EXperiment (https://hepex.irstea. Lesser General Public License v3.0 and available at https: fr/, last access: 22 November 2020) is an activity that has //gitlab.com/shyft-os (last access: 22 November 2020), facil- been ongoing since 2004, and there is extensive research on itating effective cooperation between core developers, indus- the importance of the role of ensemble forecasting to reduce try, and research institutions. uncertainty in operational environments (e.g., Pappenberger et al., 2016; Wu et al., 2020). As most operational hydrological services are within the public service, government policies and guidelines influence the area of focus. Recent trends show efforts towards increas- ing commitment to sustainable water resource management, Published by Copernicus Publications on behalf of the European Geosciences Union. 822 J. F. Burkhart et al.: Shyft v4.8 disaster avoidance and mitigation, and the need for integrated ple of the scale of the challenge is well-defined in Zappa water resource management as climatic and societal changes et al.(2008) in which the authors’ contributions to the re- are stressing resources. sults of the Demonstration of Probabilistic Hydrological and For hydropower production planning, operational hydro- Atmospheric Simulation of flood Events in the Alpine re- logic modeling provides the foundation for energy market gion (D-PHASE) project under the Mesoscale Alpine Pro- forecasting and reservoir management, addressing the inter- gramme (MAP) of the WMO World Weather Research Pro- ests of both power plant operators and governmental reg- gram (WWRP) are highlighted. In particular, they had the ulations. Hydropower production accounts for 16 % of the goal to operationally implement and demonstrate a new gen- world’s electricity generation and is the leading renewable eration of flood warning systems in which each catchment source for electricity (non-hydro-renewable and waste sum had one or more hydrological models implemented. How- up to about 7 %). Between 2007 and 2015, the global hy- ever, following the “demonstration” period, “no MAP D- dropower capacity increased by more than 30 % (World En- PHASE contributor was obviously able to implement its hy- ergy Council, 2016). In many regions around the globe, hy- drological model in all basins and couple it with all avail- dropower is therefore playing a dominant role in the regional able deterministic and ensemble numerical weather predic- energy supply. In addition, as energy production from renew- tion (NWP) models”. This presumably resulted from the able sources with limited managing possibilities (e.g., from complexity of the configurations required to run multiple wind and solar) grows rapidly, hydropower production sites models with differing domain configurations, input file for- equipped with pump storage systems provide the possibility mats, operating system requirements, and so forth. to store energy efficiently at times when total energy produc- There is an awareness in the hydrologic community re- tion surpasses demands. Increasingly critical to the growth garding the nearly profligate abundance of hydrologic mod- of energy demand is the proper accounting of water use and els. Recent efforts have proposed the development of a information to enable water resource planning (Grubert and community-based hydrologic model (Weiler and Beven, Sanders, 2018). 2015). The WRF-Hydro platform (Gochis et al., 2018) is a Great advances in hydrologic modeling are being made first possible step in that direction, along with the Structure in several facets: new observations are becoming available for Unifying Multiple Modelling Alternatives (SUMMA) through novel sensors (McCabe et al., 2017), numerical (Clark et al., 2015a), a highly configurable and flexible plat- weather prediction (NWP) and reanalysis data are increas- form for the exploration of structural model uncertainty. ingly reliable (Berg et al., 2018), detailed estimates of quan- However, the WRF-Hydro platform is computationally ex- titative precipitation estimates (QPEs) are available as model cessive for many operational requirements, and SUMMA inputs (Moreno et al., 2012, 2014; Vivoni et al., 2007; Ger- was designed with different objectives in mind than what mann et al., 2009; Liechti et al., 2013), there are improved al- has been developed within Shyft. For various reasons (see gorithms and parameterizations of physical processes (Kirch- Sect. 1.2) the development of Shyft was initiated to fill a gap ner, 2006), and, perhaps most significantly, we have greatly in operational hydrologic modeling. advanced in our understanding of uncertainty and the quan- Shyft is a modern cross-platform open-source toolbox that tification of uncertainty within hydrologic models (West- provides a computation framework for spatially distributed erberg and McMillan, 2015; Teweldebrhan et al., 2018b). hydrologic models suitable for inflow forecasting for hy- Anghileri et al.(2016) evaluated the forecast value of long- dropower production. The software is developed by Statkraft term inflow forecasts for reservoir operations using ensemble AS, Norway’s largest hydropower company and Europe’s streamflow prediction (ESP) (Day, 1985). Their results show largest generator of renewable energy, in cooperation with that the value of a forecast using ESP varies significantly as a the research community. The overall goal for the toolbox function of the seasonality, hydrologic conditions, and reser- is to provide Python-enabled high-performance components voir operation protocols. Regardless, having a robust ESP with industrial quality and use in operational environments. system in place allows operational decisions that will create Purpose-built for production planning in a hydropower envi- value. In a follow-on study, Anghileri et al.(2019) showed ronment, Shyft provides tools and libraries that also aim for that preprocessing of meteorological input variables can also domains other than hydrologic modeling, including model- significantly benefit the forecast process. ing energy markets and high-performance time series calcu- A significant challenge remains, however, in environments lations, which will not be discussed herein. that have operational requirements. In such an environment, In order to target hydrologic modeling, the software