Open Water Data Solutions for Accessing the National Water Model Software Introduction Michael A
Total Page:16
File Type:pdf, Size:1020Kb
Open Water 21 Open Water Data Solutions for Accessing the National Water Model Software Introduction Michael A. Souffront Alcantara1*, Shawn Crawley2, Michael J. Stealey3, E. James Nelson4, Daniel P. Ames5, Norm L. Jones6 1,2,4,5,6Brigham Young University 3Renaissance Computing Institute *Corresponding Author: [email protected] ABSTRACT Hydrologic modeling can help us understand how to better respond to extreme hydrologic events. Communicating model results to different groups has been a recurring challenge due to the evolving nature of hydrologic models and the distinct needs of different groups. A new National Water Model (NWM) was released in 2016 by NOAA’s Office of Water Prediction (OWP) that generates forecasts for multiple variables at a national scale. The large amounts of data produced by the NWM presents new challenges in water data management. The Open Water Data Initiative (OWDI) was started in 2014 with the goal of standardizing and facilitating water data integration and sharing. Following the conceptual model described in the OWDI, we have created a cyberinfrastructure to store NWM results, and created two web applications deployed on the HydroShare Tethys Portal to facilitate access by interested community users. A REST API was also developed in accordance with the OWDI requirements to provide water data as a service. This API facilitates programmatic data retrieval and integration of our tools with third-party projects. The developed cyberinfrastructure, web apps, and REST API provide a gateway for NWM results to be effectively communicated to scientists, planners, emergency responders, and the general public by meeting their specific needs. Keywords cyberinfrastructure, hydrologic modeling, data management, data visualization 1. Introduction 2009; Goodall, Horsburgh, Whiteaker, Maidment, & Zaslavsky, 2008; Horsburgh et al., 2009). While the Hydrologic modeling is an important tool that NetCDF format has become a standard for the shar- helps us understand how to better respond to extreme ing and processing of large scientific datasets (Rew & events. Model results can come in many different for- Davis, 1990), water data needs to be effectively com- mats depending on the model and their intended pur- municated not only to the scientific community, but pose. Due to the evolving nature of hydrologic models, also to decision makers, emergency responders, and integrating and communicating results has historically the general public (Table 1). To fulfill this goal, water been a major challenge. More generally, this is a com- data needs to be presented as actionable information mon challenge faced in water data (Beran & Piasecki, that is accessible and understandable for all user lev- Open Water 22 Group Use Level Key Functionality Scientists Research Comprehensive Decision Makers Planning Relevant Engineers/Emergency Responders Action Accessible General Public Awareness Intuitive Table 1. Water data use levels and key functionality needed per user group. els. In the United States, the open water data initia- Chapel Hill, we developed essential cyberinfrastruc- tive (OWDI), started in 2014, seeks to standardize ture for retrieving NWM forecasts and dealing with and facilitate water data integration and sharing. The the vast amount of data being produced by this model. conceptual model developed for the OWDI includes In accordance with the goals of the OWDI, we devel- four key functionalities that are essential for engaging oped a series of web applications using the open source the broader community of water data providers and Tethys Platform (Swain et al., 2016) for accessing and users: Water Data Catalog, Water Data as a Service, visualizing NWM forecasts via the web. These appli- Enriching Water Data, and Community for Water Data cations serve as a gateway to the NWM forecasts that (Blodgett, Read, Lucido, Slawecki, & Young, 2016). scientists can use in their own research, and decision The National Water Model (NWM) is a hydrologic makers and emergency managers can use to provide model that generates forecasts for multiple variables support before, during and after forecasted events. The across the continental US (NOAA, 2016). It was released purpose of this article is to introduce these web apps in 2016 by the National Weather Service (NWS) Office as a starting point for improving the overall national of Water Prediction (OWP) in collaboration with the ability to access, visualize, and analyze NWM forecasts. National Center for Atmospheric Research (NCAR) We present our complete cyberinfrastructure and app and the National Center for Environmental Prediction designs, as well as validation tests and storage and per- (NCEP). The NWM simulates runoff conditions for formance metrics in addition to our deployed apps. the 2.7 million reaches of the National Hydrography Dataset (NHD), which represents a significant 2.0 Cyberinfrastructure increase over the approximate 4000 locations cur- rently forecasted by the NWS. This significant amount As part of the preparation for the NWC summer of new information being produced by the NWM institute we developed and deployed two web appli- presents new challenges in water data management. cations along with the accompanying back-end pro- In an effort to engage the academic community and cesses to expose the volumes of data generated daily facilitate research, the NWM was piloted and tested to support the NWM. This work was sponsored by through a summer institute program sponsored by the the Consortium of Universities for the Advancement NOAA at the new National Water Center (NWC) in of Hydrologic Science Inc. (CUAHSI), which was also Tuscaloosa, Al. The focus of the summer institute is the group responsible for the organization of the NWC to enhance the NWM by exploring products derived Summer Institute. CUAHSI includes more than 100 from the NWM forecasts that could facilitate data US universities with a common goal to advance hydro- communication to a variety of academic, public, and logic science. Our web apps were deployed on the private groups. As part of the activities leading up to HydroShare apps portal (https://apps.hydroshare.org/). the 2016 summer institute, we were tasked to develop HydroShare serves as a “community of water data” an interface that the academic community and poten- that is an online, collaborative resource website for tially other groups could use to access, visualize and storing and sharing water data (Tarboton et al., 2014). add value to NWM forecasts. In collaboration with Tethys is a web framework for developing water the Renaissance Computing Institute (RENCI) at resources web applications. It offers a suite of open Open Water 23 source software selected to address the unique devel- NOAA Operational Model Archive and Distribution opment needs of water resources web apps (http:// System (NOMADS) and through an NCEP FTP server. www.tethysplatform.org/). Water resources web However, these outputs are only persisted for two days, applications often need an interactive interface where so they must be retrieved and exposed on a separate site. users can provide inputs and visualize results, and The NWM produces forecasts for a number of a background server where analyses and calcula- hydrologic parameters that vary depending on the tions can be performed. Tethys provides tools for model’s type and configuration. These variables the creation of an intuitive web interface in the form include streamflow, streamflow velocity, total evapo- of interactive maps, forms, and graphs, while at the transpiration, subsurface runoff, soil saturation, same time providing tools to connect the front- snow depth, and snow water equivalent (Figure 2). end user interface to background server processes. A processed NWM NetCDF file ranges from 72MB The NWM is produced by the OWP in collabora- to 2GB in size depending on model configuration and tion with NCAR and NCEP using NOAA’s Cray XC40 type. The number of files per forecast varies by con- supercomputer. The model has four different configu- figuration depending on frequency, duration, and rations or forecast products, which differ in duration, forecast time step. One NetCDF file includes a sin- time step, and frequency (Figure 1). All four configu- gle time step value for the set of stream reaches, grid rations produce a unique forecast, with the exception cells, or reservoir points modeled. Therefore, a single of the long-range configuration, which is an ensemble forecast is composed of many large NetCDF files. For forecast with four different members lagged by four example, one complete streamflow forecast for the six-hour time intervals for a total of 16 forecasts per short-range configuration (the shortest configura- Figure 1. National Water Model details (adapted from http://water.noaa.gov/about/nwm). day. The analysis and assimilation configuration is pro- tion) would include fifteen ~72MB NetCDF files for duced in near real-time and assimilates observation a total of 1GB uncompressed. The total size of a com- data from USGS gages. It serves as initialization for the plete forecast for the other configurations scales up other three configurations by providing an estimate of quickly. For example, the medium-range forecast for current conditions. In addition, the NWM produces the simulated land results includes 80 files of approx- results for three geospatial types or shapes: channel, imately 2GB each. The two examples above represent land, and reservoir. The channel and reservoir types the total size for a single forecast. To provide a sense are based on the US NHD plus dataset, while the land of how much data is being produced daily, there are type is based on a 1km2 land surface grid. The outputs twenty-four 15-hour short-range forecasts, one medi- of the model are made available as NetCDF files on the um-range forecast, and one 16-member ensemble Open Water 24 Figure 2. NWM variables per model configuration and type (not comprehensive). Each configuration forecasts streamflow (red), while the rest of the variables forecasted varies depending on configuration (short, medium, or long) and type (channel, land, or reservoir). Figure 3.