Hydrol. Earth Syst. Sci., 21, 4841–4859, 2017 https://doi.org/10.5194/hess-21-4841-2017 © Author(s) 2017. This work is distributed under the Creative Commons Attribution 3.0 License. Complex relationship between seasonal streamflow forecast skill and value in reservoir operations Sean W. D. Turner1,2, James C. Bennett3,4, David E. Robertson3, and Stefano Galelli5 1Pacific Northwest National Laboratory, College Park, MD, USA 2SUTD-MIT International Design Centre, Singapore University of Technology and Design, 487372, Singapore 3CSIRO, Melbourne, Clayton, Victoria 3168, Australia 4Institute for Marine and Antarctic Studies, University of Tasmania, Battery Point, Tasmania 7004, Australia 5Pillar of Engineering Systems and Design, Singapore University of Technology and Design, 487372, Singapore Correspondence to: Stefano Galelli ([email protected]) Received: 24 December 2016 – Discussion started: 4 January 2017 Revised: 22 August 2017 – Accepted: 24 August 2017 – Published: 28 September 2017 Abstract. Considerable research effort has recently been ment in value-of-forecast studies involving reservoirs with directed at improving and operationalising ensemble sea- supply objectives. sonal streamflow forecasts. Whilst this creates new opportu- nities for improving the performance of water resources sys- tems, there may also be associated risks. Here, we explore these potential risks by examining the sensitivity of forecast 1 Introduction value (improvement in system performance brought about by adopting forecasts) to changes in the forecast skill for a range Coupled natural-engineered water resources systems provide of hypothetical reservoir designs with contrasting operating a multitude of services to society. A properly functioning sys- objectives. Forecast-informed operations are simulated us- tem can ensure reliable public water supply, support agricul- ing rolling horizon, adaptive control and then benchmarked tural and industrial activity, produce clean hydroelectricity, against optimised control rules to assess performance im- provide amenity, sustain ecosystems and protect communi- provements. Results show that there exists a strong relation- ties against damaging floods. But these benefits are by no ship between forecast skill and value for systems operated means guaranteed; the performance of a given system de- to maintain a target water level. But this relationship breaks pends on the quality of its operating scheme and the intel- down when the reservoir is operated to satisfy a target de- ligence used to support management decisions on the stor- mand for water; good forecast accuracy does not necessarily age, release and transfer of water. Typically, such operating translate into performance improvement. We show that the decisions are governed by control rules based on observable primary cause of this behaviour is the buffering role played system state variables. For example, the operator might select by storage in water supply reservoirs, which renders the fore- from a predefined lookup table the desired volume of water to cast superfluous for long periods of the operation. System release from a reservoir based on the time of year, volume of performance depends primarily on forecast accuracy when water held in storage and current catchment conditions (soil critical decisions are made – namely during severe drought. moisture, snowpack, etc.). The problem with this approach As it is not possible to know in advance if a forecast will is that the decisions it recommends are optimal only under perform well at such moments, we advocate measuring the the narrow range of historical forcing conditions upon which consistency of forecast performance, through bootstrap re- they are trained. This is a major concern given emerging ev- sampling, to indicate potential usefulness in storage opera- idence of sharp trends and abrupt regime shifts in stream- tions. Our results highlight the need for sensitivity assess- flow records and palaeo-reconstructions (Turner and Galelli, 2016a). Flexible, real-time operating schemes that adapt in response to seasonal streamflow forecasts are thus the van- Published by Copernicus Publications on behalf of the European Geosciences Union. 4842 S. W. D. Turner et al.: Complex relationship between forecast skill and value guard of water resources management practice, seen widely 2 Materials and methods as the natural successor to predefined control rules (Rayner et al., 2005; Brown, 2010; Gong et al., 2010; Brown et al., 2.1 Inflow records, climate data and forecasts 2015). A move toward schemes informed by seasonal stream- Our experiments are based on four reservoir inflow records flow forecasts would benefit from a wealth of recent sci- (Table 1), which were selected because they represent a range ence advances, including new ensemble seasonal streamflow of hydrological regimes (perennial, ephemeral, intermittent) forecasting methods, adding to existing ensemble stream- across different regions of Australia. For each inflow record, flow prediction (ESP) and regression methods (e.g. Wang and we study the period 1982–2010 (Fig. 1) for which forecasts Robertson, 2011; Olsson et al., 2016; Pagano et al., 2014; are available. see review by Yuan et al., 2015). Seasonal streamflow fore- The inflow records are derived from streamflow gauges, cast services are becoming available in countries such as storage outflows and lake levels, and were supplied by the the United States, Australia and Sweden. An emerging field Bureau of Meteorology. The gauged data are freely available of research has begun to demonstrate the value of seasonal from http://www.bom.gov.au/waterdata/. Rainfall and evapo- streamflow forecasts when applied to real-world water man- ration forcing data used to generate forecasts are taken from agement problems, such as determining the appropriate water the Australian Water Availability Project (AWAP) gridded release from a reservoir – the focus of the present study. Wa- dataset (Jones et al., 2009; Raupach et al., 2009). ter release decisions can be improved with seasonal forecasts across a variety of reservoir types, including hydropower 2.1.1 Synthetic forecasts: Martingale model of forecast dams (Kim and Palmer, 1997; Faber and Stedinger, 2001; evolution (MMFE) Hamlet et al., 2002; Alemu et al., 2010; Block, 2011), wa- ter supply reservoirs (Anghileri et al., 2016; Zhao and Zhao, Our first experiment is a sensitivity test for forecast value 2014; Li et al., 2014) and reservoir systems operated for mul- as a function of forecast quality. To generate many forecasts tiple competing objectives (Graham and Georgakakos, 2010; of varying quality, we use the Martingale model of forecast Georgakakos et al., 2012). Operators considering whether to evolution (MMFE) (Heath and Jackson, 1994). This model adopt a forecast-informed operating scheme should be en- can be considered superior to one that simply imposes ran- couraged by these outcomes. But they also need to under- dom error on observed values, since it captures the way in stand the associated risks and uncertainties (Goddard et al., which forecast error decreases as the forecast horizon short- 2010). If the new scheme increases the benefits of a system ens and more information becomes available to the forecaster by, say, 20 % in a simulation experiment, then can the opera- (known as the evolution of forecast error) (Zhao et al., 2011). tor assume that 20 % will be guaranteed when the scheme is Here, we vary an “injected error” parameter, which controls implemented in practice? the error of the synthetic forecast. The injected error takes To explore uncertainty in the value of seasonal forecasts values between 0 and 1, where 0 generates a perfect forecast applied to reservoir operations, we conduct two simulation and 1 generates a sufficiently error-laden forecast to ensure experiments using reservoir inflow time series recorded at that our experiments include a wide range of forecast perfor- four contrasting catchments located in Australia. Our first mance. (Note that an error injected of 1 should not be inter- experiment uses synthetically generated forecasts of varying preted as having any physical meaning, such as equivalence skill to test for sensitivity in simulated forecast value across a to climatology.) Because the model uses probabilistic sam- range of reservoirs. Forecast value is calculated using cumu- pling to generate forecasts for a given error, the deviation of lative penalty costs incurred for deviation from a predefined the forecast from the observation will vary in time, although objective over a 30-year simulation. We define two simple, the temporal average of the error will match the error injected contrasting objectives: a “supply objective”, which aims to given enough data points. The code for this model is avail- maintain a target release by allowing storage to vary, and able as open source (Turner and Galelli, 2017). a “level objective”, which aims to maintain a target storage Here, the synthetic forecasts are constructed to overlay the level by varying the release. As we shall see, the contrast in four inflow time series described above. For each catchment, performance between the two operational settings is striking. we generate 1000, 12-month-ahead, monthly resolution syn- Our second experiment aims at explaining this outcome by thetic forecasts. The quality of the forecasts is varied by sam- applying an advanced seasonal streamflow forecast system to pling from a uniform distribution between 0 and 1 to feed a range of fabricated reservoirs