The South Florida Water Management Model, Version 5.5 Review of the SFWMM Adequacy as a Tool for Addressing Water Resources Issues Final Panel Report October 28, 2005 Panel: Rafael L. Bras, Chair Anthony Donigian Wendy Graham Vijay Singh Jery Stedinger Executive Summary Panel Task On August 1 2005 the South Florida Water Management District convened a panel of experts to perform a review of the South Florida Water Management Model (SFWMM), version 5.5, as described on the Documentation of the South Florida Water management Model, Version 5.5, Final Draft, August 2005. The essence of the Panel’s task was “To conduct an independent and objective review of the adequacy of the SFWMM [South Florida Water Management Model] as a regional modeling tool for addressing water resources issues in South Florida. The review shall rely on the latest documentation of the model as the primary source of information about the model.” The panel interpreted the mandate broadly, seeking to judge the adequacy of the model for its stated objectives and judging whether the written documentation articulates sufficiently well the capabilities of the model. It should be noted that the Panel could not, nor attempted to judge the accuracy of the coding of the model nor did it perform quality control exercises to vouch that it is error free. The Panel judged intended functionality and performance based only on the material provided and hence the accuracy of the model over its whole range of operations cannot be ascertained, except by written and oral assurances of the SFWMD staff and other users. General Findings 1. The SFWMM is an appropriate model for planning and operational planning for the South Florida system. 2. It is clear from the documentation and the workshop presentations that the model performs satisfactorily and is quite robust. 3. The Panel did not find glaringly missing processes or elements of the system. 4. There is some unevenness in the model’s representation of the hydrologic processes. 5. The model is a reasonably complete representation of the district’s operational and management system. 6. The description of the operation and management element of the model suffers from the ‘forest-from-the-trees’ syndrome. Too much detail is provided on specific elements/components, while lacking sufficient overview discussions. 7. The sensitivity and uncertainty analyses that have been reported are inadequate. 8. The District should be more explicit about the precision of calibrated parameters, and the impact of such uncertainty on the values of simulated metrics of systems performance in the SFWMM. 9. There are changes that should be implemented in a completely new version of this model, an effort that the Panel encourages. 2 Major Recommendations 1. The Panel strongly urges the SFWMD to clarify the model objectives with concise statements in the first chapter of the document. 2. A more rigorous discussion and justification of the level of complexity chosen for each process in each region should be included in the written documentation. 3. Chapter 3 on Policy and System Management Components is possibly the most important chapter in the document. The Panel recommends that Chapter 3 include more discussions of how the whole system operates, before discussing details of specific elements and components. 4. The Panel recommends that water budgets be specifically recognized as key hydrologic performance metrics, and should be both shown and discussed. 5. Automatic optimization search techniques should be considered for parameter calibration. This is important for investigating model behavior as related to establishing parameter values. The Panel recommends that parameter precision be quantified. 6. The Panel recommends that concepts of precision be integrated from the initial calibration process on through the sensitivity and uncertainty analyses that are conducted on the model parameters as well as the performance measures. 7. The Panel suspects that the sensitivity and uncertainty analyses that have been conducted are inadequate. However the Panel is certain that the documentation of the calibration, sensitivity analysis and uncertainty analysis efforts are unclear. It is recommended that efforts on appropriate sensitivity and uncertainty analysis be redoubled and documented appropriately. 8. The Panel recommends an additional chapter to summarize the objectives and capabilities of the model. It should state its limitations as well as appropriate uses, possibly illustrated with experiences with the model. Finally it should speak to the future developments of the model, its useful life and plans for a Next Generation Regional Model. Summary of Responses to District Questions A. Clarity and Appropriateness: Are the objectives of the documentation clear? Are the objectives met? 1. This is overall a complete and readable document. The introduction and initial description of the system is a good one. There is some lack of uniformity apparent in the presentation. 2. Nowhere can the reader find a concise statement of model goals, objectives and intended use. Nowhere does the documentation state clearly that the model is intended for planning and operations planning, or describe the meaning of those terms. The stated purposes of the model largely address simulation ability without discussing the nature and use of those simulations, possibly by example. 3 B. Based on the documentation and presentations provided by the District, are the modeling techniques and methodologies used in the SFWMM appropriate for the temporal and spatial scale of the model? 1. All model processes and techniques are defensible, none of them can be found incorrect or unexplainable. That does not imply that there may not be better ways to represent processes or to model the system. The SFWMM is a result of over 20 years of development. As is commonly the case in this type of a model, it reflects “patchiness”; unevenness in process representation; and solutions that are less than ideal or forced to fit an existing structure. That does not imply that anything is wrong or inappropriate but it is a reflection of legacy and changing times and technologies. C. Physical and Hydrologic Processes: Does the SFWMM include all the important physical and hydrological processes necessary to address regional-scale water resource issues in South Florida? 1. All significant processes that describe the regional hydrology of South Florida are represented in the SFWMM. The methodologies used to represent these processes range from quite empirical (e.g., the treatment of the unsaturated zone and evapotranspiration) to more physically-based (e.g., groundwater flow and overland flow in the natural areas). Decisions regarding the level of sophistication required for modeling different hydrologic processes in different regions seem to have been made, based on intuition, experience and data availability, to improve computational efficiency, or to improve model calibration. While the panel respects and accepts the judgment of the modelers at the SFWMD, a more rigorous discussion and justification of the level of complexity chosen for each process in each region should be included in the written documentation. D. Does the SFWMM include all the important structural and operational rules to address regional-scale water resource issues in South Florida? Are the structural features and operational rules addressed adequately? 1. As far as the Panel can ascertain, given the limitations of the review process, the SFWMM tries to represent all the important structural and operational rules of the system. This assessment is largely founded on the testimony of the district staff. Nevertheless, the documentation of the operational rules suffers from the ‘forest-from-the-trees’ syndrome. Too much detail is provided on specific elements/components, while lacking sufficient overview discussions to allow the reader a clearer image of how the whole system operates as an aggregation of its components. Starting with an overview discussion would help clarify how the component information later in the chapter fits into the overall model. 2. There is no clear distinction on where the model representation ends and actual operating rules and regulations begin. The presentations made it clear that the model is not used for actual operations, but the link between the model and field operations/actions is not clear. The Panel recommends 4 that the relationship between the model and field operations be discussed and clarified. E. Calibration and Validation: Is the model calibration process adequate for a predictive model in water resources management? Based on available tools, procedures, and data, is the model validation/verification procedure conducted in an appropriate manner? 1. The SFWMM is a regional scale hydrology model and is primarily used as a planning and management tool to assist with environmental management, water supply and regional flood control. It is not a research model. When judged against this backdrop, the model calibration and verification was probably satisfactory. 2. An unanswered question is: What is the level of model accuracy needed for appropriate model uses? This has direct bearing on model calibration and verification. A discussion of this issue is important for model calibration and verification. F. Overall appropriateness of model comparable to others outside South Florida 1. There is no other existing model that can do what the SFWMM does in South Florida. The value of the SFWMM is that it provides an integrated description of this unique, large, and complicated system. It would be wrong to think of this model or any other future model as a generic hydrologic tool. For the foreseeable, SFWMD needs a customized tool, one that is appropriate for the unique environment it needs to represent. 5 1.0 Introduction On August 1 2005 the South Florida Water Management District convened a panel of experts to perform a review of the South Florida Water Management Model (SFWMM), version 5.5 (as described in Documentation of the South Florida Water Management Model, Version 5.5, August 2005, Final Draft). Relevant elements of the Statement of Work are attached in Appendix E.