Validation of Biophysical Models: Issues and Methodologies. Gianni Bellocchi, Mike Rivington, Marcello Donatelli, Keith Matthews
Total Page：16
File Type：pdf, Size：1020Kb
Validation of biophysical models: issues and methodologies. Gianni Bellocchi, Mike Rivington, Marcello Donatelli, Keith Matthews To cite this version: Gianni Bellocchi, Mike Rivington, Marcello Donatelli, Keith Matthews. Validation of biophysical models: issues and methodologies.. Sustainable agriculture, 2, Springer, 27 p., 2011, Sustainable Agriculture Reviews, 978-94-007-0393-3 978-94-007-0394-0. hal-02810072 HAL Id: hal-02810072 https://hal.inrae.fr/hal-02810072 Submitted on 6 Jun 2020 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Validation of Biophysical Models: Issues and Methodologies Gianni Bellocchi, Mike Rivington, Marcello Donatelli, and Keith Matthews Contents Abstract The potential of mathematical models is widely acknowledged for examining components and 1 Introduction ................................... 578 interactions of natural systems, estimating the changes 2 Issues on Model Validation ...................... 581 and uncertainties on outcomes, and fostering commu- 2.1 Validation Purpose ........................ 581 nication between scientists with different backgrounds 2.2 Interpretation of Phenomena ................ 582 2.3 Model Comparison ........................ 583 and between scientists, managers and the community. 2.4 Model Predictions ......................... 583 For favourable reception of models, a systematic ac- 2.5 Model Complexity ........................ 584 crual of a good knowledge base is crucial for both sci- 2.6 Data Accuracy and Quality ................. 585 ence and decision-making. As the roles of models grow 2.7 Robustness of Model Results ................ 587 2.8 Time Histories ............................ 587 in importance, there is an increase in the need for ap- 2.9 Summary ................................ 587 propriate methods with which to test their quality and 3 Validation of Models ............................ 588 performance. For biophysical models, the heterogene- 3.1 Validation Measures ....................... 588 ity of data and the range of factors influencing useful- 3.2 Disaggregating Statistics ................... 590 ness of their outputs often make it difficult for full anal- 3.3 Statistical Hypothesis Tests ................. 591 ysis and assessment. As a result, modelling studies in 3.4 Validation Criteria ......................... 593 3.5 Combining Multiple Statistics ............... 593 the domain of natural sciences often lack elements of 3.6 Summary ................................ 595 good modelling practice related to model validation, 4Conclusion.................................... 596 that is correspondence of models to its intended pur- pose. Here we review validation issues and methods References ........................................ 597 currently available for assessing the quality of biophys- ical models. The review covers issues of validation purpose, the robustness of model results, data qual- ity, model prediction and model complexity. The im- portance of assessing input data quality and interpreta- tion of phenomena is also addressed. Details are then provided on the range of measures commonly used for validation. Requirements for a methodology for assessment during the entire model-cycle are synthe- sised. Examples are used from a variety of modelling studies which mainly include agronomic modelling, G. Bellocchi () e.g. crop growth and development, climatic modelling, Grassland Ecosystem Research Unit, French National Institute for Agricultural Research, 234 Avenue du Brézet, e.g. climate scenarios, and hydrological modelling, e.g. 63100 Clermont-Ferrand, France soil hydrology, but the principles are essentially appli- e-mail: [email protected], cable to any area. It is shown that conducting detailed [email protected] validation requires multi-faceted knowledge, and poses E. Lichtfouse et al. (eds.), Sustainable Agriculture Volume 2, DOI 10.1007/978-94-007-0394-0_26, 577 c Springer Science+Business Media B.V. - EDP Sciences 2011. Reprinted with permission of EDP Sciences from Bellocchi et al., Agron. Sustain. Dev. 30 (2010) 109–130. DOI: 10.1051/agro:2009001 578 G. Bellocchi et al. substantial scientific and technical challenges. Special REF Relative modelling efficiency emphasis is placed on using combined multiple statis- RMA Reduced major axis tics to expand our horizons in validation whilst also RMdAE Relative median absolute error tailoring the validation requirements to the specific RMSE Root mean square error objectives of the application. ROC Receiver-operator characteristic curve RRMSE Relative root mean square error RMSV Root mean square variation Keywords Accuracy r Modelling r Multiple statistics r SB Simulation bias Validation SDSD Square differences of the standard devia- tion U Theil’s inequality coefficient Abbreviations UB Systematic error proportion of Theil’s inequality coefficient AIC Akaike’s information criterion US Variance proportion of Theil’s inequality BIC Bayesian information criterion coefficient CD Coefficient of determination UC Covariance proportion of Theil’s inequality Cp Mallows’ statistic coefficient CRM Coefficient of residual mass VG Geometric mean variance d Willmott’s index of agreement D Kolmogorov-Smirnov’s statistic E Mean relative error 1Introduction EF Modelling efficiency EF1 Modified modelling efficiency Fa2 Factor of two The mathematical modelling of natural processes has FB Fractional bias undergone a large development during the last decades Ef Fractional gross error and, due to the complexity of the processes involved, LC Lack of correlation this development is expected to pursue for a long time. LCS Lack of positive correlation weighted by The development of quantitative models to support the the standard deviations description of natural and semi-natural systems and LOFIT Lack of statistical fit decision-making in natural resource management is in- MAE Mean absolute error deed considered to be of high priority. This is because MaxE Maximum error models have a multitude of uses for scientists, man- MB Mean bias agers and policy-makers investigating and governing MBE Mean bias error natural processes. A major strength of models is in ex- MdAE Median absolute error ploring interactions and feedback (e.g. Wainwright and MG Geometric mean bias Mulligan, 2004), helping to identify uncertainties and MSE Mean square error areas were we lack knowledge. They are also important NMSE Normalized mean square error supportive tools for the communication of complex is- NU Non-unity slope sues to stakeholders of a non-scientific background. It PI Range-based pattern Index is therefore important to demonstrate that a model has PI-F F-based pattern index been tested using the most appropriate methods in or- PIdoy Range-based pattern index versus day of der to achieve credibility with users of the estimates year the model makes. PITmin Range-based pattern index versus mini- A mathematical model is, by definition, an ap- mum air temperature proximate reconstruction of actual phenomena and P (t) Student’s t-test probability an integration of natural processes into mathemat- r Pearson’s correlation coefficient ical formulae. For agricultural and ecological sys- r2 Least-square regression coefficient of de- tems and resource use (climate, land, vegetation...), termination a multitude of different theories coexist, not only Validation of Biophysical Models: Issues and Methodologies 579 amongst disciplines (plant physiology, hydrology, climatology...), but also within disciplines (Argent, 2004; Beven, 2007; Arnold et al., 2008). Ecological, soil, meteorological and hydrological conditions in the actual systems are indeed the product of multiple con- current processes, where multiple factors interact at different scales, each examined by different disciplines (Parker et al., 2003). This produces an abundance of theories and alternative explanations and, conse- quently, alternative models. One of the challenges is the process of bringing data and models together. It is required that numerical models should be preceded by thorough evaluation before use in practical applica- tions, because the approximations used for the synthe- Fig. 1 Phases of modelling and simulation and the role of vali- sis of a model often lead to discrepancies and devia- dation and verification (after Schlesinger, 1979) tions of the model results from nature. Model evaluation is an essential step in the mod- as system representation (model structure) and pro- elling process because it indicates if the implementa- gram verification play a role besides numerical eval- tion of the calculations involved reproduces the con- uation (structural assessment and program testing, not ceptual model of the system to be simulated (model discussed in this paper) in assessing model accuracy reliability) and the level of accuracy of the model in re- (Donatelli et al., 2002b). Shaeffer (1980) developed a producing the actual system (model usefulness) (Huth methodological approach to evaluate models that con- and Holzworth, 2005). Model evaluation includes any sisted