ADHydro: QUASI-3D HIGH PERFORMANCE HYDROLOGICAL MODEL

Fred L. Ogden, Professor, [email protected]; Wencong Lai, Post-doctoral Associate, [email protected]; Robert C. Steinke, Research Scientist, [email protected]

Abstract: In recent decades, computational models have been developed to solve point-scale process models using physics-based or conceptual approaches. The integration of these processes across space-time has been limited by computational power to either high-resolution over small spatial domains, or coarse resolution over large spatial domains. These modeling approaches have lead to improved understanding at both small and large scales, but have required parameterization of important phenomenon, and the corresponding lack of model sensitivity to changes and uncertainties in parameter values. The CI-WATER project, a cooperative agreement between the US National Science Foundation EPSCoR program and the Utah and Wyoming Ph.D.-granting universities, is developing tools to cross the digital divide that impedes application of high performance computing (HPC) to solve hydrological science, engineering, and management problems. We are developing software tools to ease simulations using HPC resources. These tools include web-aware model setup and visualization tools that will interact with dedicated HPC systems or cloud systems, workflows for model setup, and web services for data provisioning. These tools are being developed with generality in mind, supporting a range of HPC-aware models, including the US Army Corps of Engineers Gridded Surface/Subsurface Analysis (GSSHA) model, and an unstructured mesh high-resolution physics-based hydrological model, called ADHydro. The ultimate objective of ADHydro development is perform multi- decadal simulations of large watersheds such as the Upper Colorado River above Lake Powell (288,000 km2) in the contexts of land use, water use, and climate changes. We are cooperating with the US Army Corps of Engineers, Engineering Research and Development Center, Coastal and Hydraulics Laboratory, and the National Center for Atmospheric Research, Research Applications Laboratory in linking with their HPC hydrological and atmospheric models. This presentation will showcase our software tools under development.

INTRODUCTION

Physically-based, spatially distributed hydrologic models have been widely used in hydrologic modeling and watershed management, such as SWAT (Arnold et al., 1993), MIKE SHE (Refsgaard and Storm, 1995), and GSSHA (Downer and Ogden, 2004). Applications of such hydrologic models for large watersheds typically use coarse spatial and temporal resolution. Detailed simulation of larger watersheds with hyperresolution hydrological prediction is a grand challenge because significant computational resources and data are required (Wood et al., 2011).

In recent years, a growing number of distributed hydrological models have been developed in parallel computing environments with the advent of high performance computing (Apostolopoulos and Georgakakos, 1997; Cui et al., 2005; Wang et al., 2011; Vivoni et al., 2011; Hwang et al., 2014). HPC hydrological models coupled with meteorological models are capable to model high-resolution long term simulation in large watersheds in order to evaluate the impact of climate and land use changes. HPC hydrological models can be classified into two categories according to their parallel algorithm. One category of such models, for example ParFlow(Ashby and algout, 1996; Kollet and Maxwell, 2006), PFLOTRAN (Mills et al., 2007), PHGS (Hwang et al., 2014), utilize pre-developed HPC packages or libraries for parallel preconditioner and solver. The other category of models adopt spatial domain decomposition with either sub-basin based (Li et al, 2011; Vivoni et al., 201; Wu et al., 2013) or unit based message passing (Cheng et al., 2005; Liu et al., 2014).

Parallelization of hydrologic models can be implemented using parallel programming standards such as Open Multi-Processing (OpenMP), Message Passing Interface (MPI), grid computing, and other parallel programming toolkit. The PHGS (Hwang et al., 2007) applied OpenMP in the HydroGeoSphere model using parallel matrix solver. The FSDHM model was parallelized using OpenMP by dividing simulation units into hydrologic independent layers (Liu et al., 2014). However, OpenMP only works in shared memory machines. Yalew et al. (2013) parallelized the SWAT watershed model using the distributed grid computing by splitting large scale model into small scale components. As MPI can distribute computing loads and converge results through message transfer and communication between processors, it is the most used technology in parallel hydrologic models for domain decomposition (Li et al, 2011; Vivoni et al., 2011; Wang et al., 2011; Wu et al., 2013). However, using the MPI library for data partitioning and communication is not straightforward. The pWASH123D model (Cheng et al., 2005) utilizes the DBuilder (Hunter and Cheng, 2005) parallel data management toolkit, which hides the MPI scheme for map generating, sending, and receiving between meshes with simple Application Programming Interfaces (API) and embedded ParMETIS partitioner library (Karypis et al., 1997).

The development of the ADHydro model is presented here. The ADHydro model is a high- resolution physics-based distributed hydrological model developed in parallel computing environment. The merits of the model comparing with other HPC hydrologic models include: an innovative method for modeling vadose zone dynamics, a water management module considering reservoirs, diversions and irrigation, a coupled strategy to estimate interception evaporation, and snow processes through the community Noah land surface model with mutiparameterization options (Noah-MP) and the capability to ingest downscaled atmospheric forcing from the Weather Research and Forecasting (WRF) meteorologic model using the CHARM++ parallel programming environment.

GEOSPATIAL DATA AND ATMOSPSHERIC FORCING

The Upper Colorado River Basin above Lake Powell has a watershed area of approximately 288,000 km2, and total length of streams of 467,000 km. It's located in one of the 21 major geographic regions defined by the USGS hydrologic unit code (Seaber et al., 1987) in the US. In the pre-processing step, topographic base map data were acquired from National Elevation Dataset (NED) 1/3 arc-second Digital Elevation Models (DEMs), and USGS National Hydrography Dataset (NHD) (http://viewer.nationalmap.gov/viewer/). Land cover and land use data were obtained through the 30-meter resolution, 16-class land cover classification National Land Cover Database 2011 (NLCD 2011) (http://www.mrlc.gov/). Soil texture types were aggregated from the NRCS county level Soil Survey Geographic Database (SSURGO) and state- wide State Soil Geographic Database (STATSGO) (http://www.nrcs.usda.gov/wps/portal/nrcs/main/soils/survey/geo/).

The TauDEM tools (http://hydrology.usu.edu/taudem/taudem5/) were used to extract channel network from NED DEMs. The resulting channel network and NHD were analyzed and processed by ArcGIS to produce shapefiles and steam network connectivity that includes lakes and reservoirs. The shapefiles were used to generate high resolution unstructured 2D triangular mesh using Triangle (http://www.cs.cmu.edu/~quake/triangle.html). A 1D channel network model with mesh size of 100 meters was also generated. River hydraulic geometry were described in the form of power functions of discharge, which scales with drainage area.

Atmospheric forcing for different scenarios was generated using the Weather Research and Forecasting meteorologic model (Michalakes et al., 2004). The WRF model is a mesoscale numerical weather prediction system design to both atmospheric research and operational forecasting needs. Simulation results from WRF, including precipitation, air temperature, air pressure, wind speed, short and long wave radiation and vapor pressure, were used as atmospheric forcing input for ADHydro. The 4 km resolution WRF outputs were downscaled to the high-resolution meshes.

ADHYDRO QUASI 3-D DISTRIBUTED HYDROLOGIC MODEL

The ADHydro is a high-resolution multi-physics distributed model for hydrological and water resources modeling. Major hydrologic processes are considered, including precipitation and , snowfall and snowmelt in complex terrain, vegetation and evapotranspiration, soil heat flux and freezing, overland flow, channel flow, flow and water management. These hydrologic components are described below.

The ADHydro model uses the explicit finite volume method to solve conservation laws for overland flow and saturated groundwater flow coupled to river flow. The model has a quasi-3D formulation that couples 2D overland flow and 2D saturated groundwater flow using the 1D Talbot-Ogden finite water-content infiltration and redistribution method (Talbot and Ogden, 2008). This eliminates difficulties in solving the highly nonlinear 3D Richards' equation, while the finite volume Talbot-Ogden infiltration method is computationally efficient, mass conservative, and allows simulation of the effect of shallow groundwater tables on runoff generation.

Interception, Evapotranspiration and Snow Melt: Interception, evapotranspiration and snow melt processes are simulated using the Noah-MP model (Niu et al., 2011; Yang et al., 2011). The Noah-MP model considers biophysical processes such as interactive vegetation canopy, multilayer snow pack and soil, overland runoff, and unconfined aquifer for groundwater storage with a dynamic water table. Its major components include 1-layer canopy, 3-layer snow, and 4- layer soil. In Noah-MP, partitioning precipitation into rainfall and snowfall use surface air temperature as a criterion, the canopy water scheme simulates the canopy water interception and evaporation, and the “semitile” subgrid scheme calculates the skin temperature of the canopy and snow/soil surface separately using an interactive energy balance method. Snow and soil layer temperatures are used to asses the energy for melting and freezing for the snow and soil layers. Noah-MP model input includes static input data (e.g. vegetation and soil data, latitude and longitude) and atmospheric forcing data (e.g. precipitation, air temperature, humidity, radiation).

Overland Flow Routing: The shallow overland flow can be simulated using dynamic wave or diffusive wave Shallow Water Equations (SWE). The depth-averaged 2D shallow water equations are derived by integrating the Navier-Stokes equations in the vertical direction under the assumptions of hydrostatic pressure distribution and uniform velocity profiles in the vertical direction. The dynamic wave SWE can be written as (Ying et al., 2009):

(1)

where h is water depth, qs is source/sink term, u and v are flow velocities in x and y directions, respectively, Z = h + zg is water surface elevation, zg is surface ground elevation, and Sfx and Sfy are friction slopes.

Further under the diffusion wave approximation where inertia is not important (Akan and Yen, 1981), the diffusive wave SWE is given as:

(2)

where the diffusion coefficient:

(3)

with s is the maximum slope direction, and n is Manning's roughness coefficient. The diffusive wave approximation neglects the local acceleration term and convective acceleration term in the momentum equations, and it is applicable in situations where Froude number is small.

Channel Flow Routing : The dynamic wave governing equations (Saint-Venant equations) for one-dimensional flows in natural rivers include the continuity equation and momentum equation:

(4)

where A is cross section area of the channel, and Q is volumetric flow rate. The dynamic wave equations can also be simplified under diffusive wave assumption: (5)

Level pool routing method is used for lakes and reservoirs:

(6)

where S is volume of storage in the reservoir, I is inflow, O is outflow, R is rainfall, E is evaporation, Sp is seepage, and Ol is lateral overland flow. Cross-section properties come from empirically-derived scaling relations for the 2-year flow coupled with bankfull cross-section estimators.

Coupling 1D Channel and 2D Overland Flow : A source term based lateral connection between 1D channel and 2D overland flow domain is used. The broad-crested weir discharge is calculated as (Blade et al., 2012):

(7)

where z1 is headwater surface elevation, z2 is tailwater elevation, and zw is weir crest elevation, L is the length of 1D channel element in contact with 2D mesh edge, and K is a constant (generally 0.3 < K < 0.6).

Unsaturated Vadose Zone Flow : The 1D infiltration and redistribution method in the discretized moisture content domain (Ogden et al. 2015b, Talbot and Ogden, 2008), and an optional approximation (Lai et al., 2015) are used to simulate vadose zone flow. The T-O method assumes homogeneous soil, and the water moisture content is discretized into hypothetically interacting bins. The soil moisture characteristic and unsaturated hydraulic conductivity curves are required to discretize the T-O domain. These curves can be described using soil characteristic models such as Brooks-Corey model (1966) or van Genuchten model (1980).

In T-O method, the movement of and groundwater in each bin is simulated, followed by the process of redistribution. The infiltration advancement of surface wetting front in a bin k due to capillary and gravitational forces is given as:

(8) where Zj is position of surface wetting front of bin j, θi is initial water content or the water content of the first bin that is not fully saturated from the groundwater table to the surface, θd is the water content of the right-most bin in the surface wetting front that contains water, K(θi) and K(θd) are the unsaturated hydraulic conductivity of the θi and θd bins respectively, hp is depth of surface ponding, and h(θd) is the capillary pressure of θd bin. While the movement of a groundwater wetting front is given as (Ogden et al. 2015a):

(9)

where the hydrostatic capillary height considering a constant surface influx qin < Ks :

(10)

The redistribution process sorts the bin depths from deepest to shallowest going from left to right for both the surface wetting fronts and the groundwater wetting fronts. This redistribution scheme moves water at the same elevation laterally, and is similar to the game “TetrisTM”, but operating horizontally. After redistribution, the length of both surface wetting front and groundwater front decreases monotonically from high capillary suction to low capillary suction.

Saturated Groundwater Flow: The ADhydro model simulates groundwater flow using a quasi- 3D unsaturated/saturated flow scheme. Flow in the vadose zone is modeled using the 1D infiltration and redistribution method, and flow in the saturated zone is simulated using the 2D Boussinesq equation. The Boussinesq equation for saturated 2D groundwater flow in unconfined aquifer is given by:

(11)

where H is total groundwater hydraulic head, h is groundwater depth, Kx are Ky hydraulic conductivity, R is the vertical recharge rate to the saturated surface, and Sy is the specific yield.

Since the amount of water recharge to saturated zone (W = RΔt) is calculated using the 1D infiltration and redistribution method, to have a more consistent way coupling unsaturated- saturated zone, the saturated flow rate (R') into an element is calculated:

(12) This also allows the saturated flow and unsaturated flow calculated using different time steps. Then the change of groundwater depth can be written as:

(13)

The position of groundwater table is used as boundary condition for 1D unsaturated flow. For example, if the total recharge to saturated groundwater is positive (R' Δt + W > 0), the water table moves upward, the 1D moisture curve should move upward as well using available water from the total recharge.

Coupling 1D Channel and 2D Groundwater Flow : Lateral flow between channel and groundwater is a function of river water surface elevation and groundwater head. The specific flow rate from channel to groundwater can be calculated as (Gunduz and Aral, 2005):

(14)

where Kr is river bottom sediment conductivity, wr is river bed wetted perimeter, Δzb is the river bed thickness, Zr is river water surface elevation, H is groundwater head.

Water Management : A water management model is developed for the Upper Colorado River Basin. Emphasis are placed on the engineered aspects of water management and use, where storage reservoirs, diversions, and irrigation are simulated. Statistical based method and operation rules based optimization method are used. Operational water management rules are explored for the major reservoirs and irrigation districts in the Upper Colorado River Basin from Bureau of Reclamation (http://www.usbr.gov/uc/water/). Typical constrains and rules include maximum and minimum elevations, target elevations for wet and drought seasons, maximum and minimum releases, and contractual, legal, and institutional obligations (Yeh, 1985). The simulation/optimization model determines release decisions for different planning purposes. Interactions between reservoirs and river/aquifer system are also considered.

CHARM++ Parallelization: The ADHydro implementation uses the Charm++ parallel programming system. Charm++ is based on location transparent message passing between migrateable C++ objects. Each object represents an entity in the model such as a mesh element. These objects can be migrated between processors or serialized to disk allowing the Charm++ system to automatically provide capabilities such as load balancing and checkpointing. Objects interact with each other by passing messages that the Charm++ system routes to the correct destination object regardless of its current location. REFERENCES

Akan, A. O., and Yen, B. C. (1981). “Diffusion-wave flood routing in channel networks,” Journal of the Hydraulics Division, 17(6), pp 719-732. Apostolopoulos, T. K., and Georgakakos, K. P. (1997). “Parallel computation for streamflow prediction with distributed hydrologic models,” Journal of , 197(1-4), pp 1-24. Arnold J. G., Allen, P. M., and Bernhardt, G. (1993). “A comprehensive surface-groundwater flow model,” Journal of Hydrology, 142(1-4), pp 47-69. Ashby, S. F., and Falgout, R. D. (1996). “A parallel multigrid preconditioned conjugate gradient algorithm for groundwater flow,” Nuclear Science and Engineering, 124(1), pp 145-159. Blade et al.,(2012). “Integration of 1D and 2D finite volume schemes for computations of water flow in natural channels,” Advances in Water Resources, 42, pp 17-29. Brooks, R. H., and Corey, A. T. (1966). “Properties of porous media affecting fluid flow,” Journal of the Irrigation and Drainage Division, Proceeding of the American Society of Civil Engineers, 92(IR2), pp 61-88. Cheng, J. R. C., Hunter, R. M., Cheng, H., Richards, D. R., and Yeh G. T. (2005). “Parallelization of a watershed model-Phase III: Coupled 1-dimensional channel, 2-dimensional overland, and 3-dimensional subsurface flows,” Computational Methods in Water Resources XVI, Copenhagen, Denmark, CMWR CD-ROM, paper 64, pp 19-22. Cui, Z., Vieux, B. E., Neeman, H., and Moreda, F. (2005). “Parallelisation of a distributed hydrologic model,” Int'l. J. Computer Applications in Technology, 22(1), pp 42-52. Downer, C. W., and Ogden F. L. (2004). “GSSHA: Model to simulate diverse stream flow producing process,” Journal of Hydrological Engineering, 9(3), pp 161-174. Gunduz, O., and Aral, M. M. (2005). “River networks and groundwater flow: a simultaneous solution of a coupled system,” Journal of Hydrology, 301(1-4), pp 216-234. Hunter, R. M., and Cheng, J. R. C. (2005). “Dbuilder: A parallel data management toolkit for scientific applications,” Proceedings of the 2005 International Conference on Parallel and Distributed Processing Techniques and Applications, pp 825-831. Hwang, H.-T., Park, Y.-J., Sudicky, E. A., and Forsyth, P. A. (2014). “A parallel computational framework to solve flow and transport in integrated surface-subsurface hydrologic systems,” Environmental Modelling & Software, 61, pp 39-58. Karypis, G., Schloegel, K., and Kumar, V. (1997). “Parmetis: Parallel graph partitioning and sparse matrix ordering library,” Version 1.0, Department of Computer Science, University of Minnesota. Kollet, S. J., and Maxwell, R. M. (2006). “Integrated surface-groudwater flow modeling: A free- surface overland flow boundary condition in a parallel groundwater flow model,” Advances in Water Resources, 29(7), pp 945-958. Lai, W., F.L. Ogden, R.C. Steinke, and C.A. Talbot, (2015). An efficient and guaranteed stable numerical method for continuous modeling of infiltration and redistribution with a shallow dynamic water table, Water Resour. Res., in press. Li, T., Wang, G., Chen, J., and Wang, H. (2011). “Dynamic parallelization of hydrological model simulation,” Environmental Modelling & Software, 26(12), pp 1736-1746. Liu, J., Zhu, A., Liu, Y., Zhu, T., and Qin, C. (2014). “A layered approach to parallel computing for spatially distributed hydrological modeling,” Environmental Modelling & Software, 51, pp 221-227. Michalakes, J., Dudhia, J., Gill, D., Henderson, T., Klemp, J., Skamarock, W., & Wang, W. (2004). “The weather research and forecast model: software architecture and performance,” In Proceedings of the 11th ECMWF Workshop on the Use of High Performance Computing In Meteorology, 25, pp 29. Mills, R. T., Lu, C., Lichtner, P. C., and Hammond, G. E. (2007). “Simulating subsurface flow and transport on ultrascale computers using PFLOTRAN,” In Journal of Physics: Coference Series, 78(1), IOP Publishing. Niu G. Y., et al. (2011). “The community Noah land surface model with muliparameterization options (Noah-MP): 1. Model description and evaluation with local-scale measurements,” Journal of Geophysical Research, 116, D12109, doi:10.1029/2010JD015139. Ogden, F.L., W. Lai, R.C. Steinke, and J. Zhu, (2015a), Validation of finite water-content vadose zone dynamics method using column experiment with a moving water table and applied surface flux. Water Resour. Res., in press. Ogden, F.L., W. Lai, R.C. Steinke, J. Zhu, C.A. Talbot, and J.L. Wilson, (2015b). “An ordinary differential equation alternative to the 1-D Richards' Equation in the Unsaturated Zone”. Water Resour. Res., in review. Refsgaard, J. C., and Storm, B. (1995). MIKE SHE. In: Singh, V. P. (Ed), Computer Models of Watershed Hydrology. Water Resources Publications, pp 809-846. Seaber, P. R., Kapinos, F. P., and Knapp, G. L. (1987). Hydrologic Unit Map. US Government Printing Office. Talbot, C. A., and Ogden, F. L. (2008). “A method for computing infiltration and redistribution in a discretized moisture content domain,” Water Resources Research, 44(8), W08453, DOI: 10.1029/2008WR006815. Van Genuchten, M. T. (1980). “A closed-from equation for predicting the hydraulic conductivity of unsaturated soils,” Soil Science Society of America Journal, 44(5), pp 892-898. Vivoni, E. R., et al. (2011). “Real-world hydrologic assessment of a fully-distributed hydrological model in a parallel computing environment,” Journal of Hydrology, 409(1-2), pp 483-496. Wang H., Fu, X., Wang, G., Li, T., and Gao, J. (2011). “A common parallel computing framework for modeling hydrological processes of river basins,” Parallel Computing, 37, pp 302-315. Wood et al. (2011). “Hyperresolution global land surface modeling: Meeting a grand challenge for monitoring Earth's terrestrial water,” Water Resources Research, 47, W05301, doi:10.1029/2010WR010090. Wu, Y., Li, T., Sun, L., and Chen, J. (2013). “Parallelization of a hydrological model using the message passing interface,” Environmental Modelling & Software, 43, pp 1240131. Yalew, S, van Griensven, A., Ray, N. Kokoszkiewicz, L, and Betrie, G. D. (2013). “Distributed computation of large scale SWAT models on the Grid,” Environmental Modelling & Software, 41, pp 223-230. Yang Z. L., et al. (2011). “The community Noah land surface model with muliparameterization options (Noah-MP): 2. Evaluation over global river basins,” Journal of Geophysical Research, 116, D12110, doi:10.1029/2010JD01540. Yeh, W. W-G. (1985). “Reservoir management and operations models: A state-of-the-art review.” Water Resources Research, 21(12), 1797-1818. Ying, X., Jorgeson, J., and Wang, S. (2009). “Modeling dam-break flows using finite volume method on unstructured grid,” Engineering Applications of Computational Fluid Mechanics, 3(2), pp 184-194.