FAIRMODE Guidance Document on Modelling Quality Objectives and Benchmarking
Total Page:16
File Type:pdf, Size:1020Kb
FAIRMODE Guidance Document on Modelling Quality Objectives and Benchmarking Version 3.2 Janssen, S., Thunis, P. With contributions of: Adani, M., Carnevale, C., Cuvelier, C., Durka, P., Georgieva, E., Guerreiro, C., Malherbe, L., Maiheu, B., Meleux, F., Montero, A., Miranda, A., Olesen, H., Pfäfflin, F., Stocker, J., Sousa Santos, G., Stidworthy, A., Stortini, M., Trimpeneers, E., Viaene, P., Vitali, L., Vincent, K., Wesseling, J. EUR 30264 EN This publication is a Technical report by the Joint Research Centre (JRC), the European Commission’s science and knowledge service. It aims to provide evidence-based scientific support to the European policymaking process. The scientific output expressed does not imply a policy position of the European Commission. Neither the European Commission nor any person acting on behalf of the Commission is responsible for the use that might be made of this publication. For information on the methodology and quality underlying the data used in this publication for which the source is neither Eurostat nor other Commission services, users should contact the referenced source. The designations employed and the presentation of material on the maps do not imply the expression of any opinion whatsoever on the part of the European Union concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Contact information Name: P. Thunis Address: European Commission, Joint Research Center (JRC), via Fermi 2749, 21027 Ispra (VA), Italy Email: [email protected] Tel.: +39 0332 785670 EU Science Hub https://ec.europa.eu/jrc JRC120649 EUR 30264 EN PDF ISBN 978-92-76-19745-4 ISSN 1831-9424 doi:10.2760/30226 Print ISBN 978-92-76-19746-1 ISSN 1018-5593 doi:10.2760/15956 Luxembourg: Publications Office of the European Union, 2020 © European Union, 2020 The reuse policy of the European Commission is implemented by the Commission Decision 2011/833/EU of 12 December 2011 on the reuse of Commission documents (OJ L 330, 14.12.2011, p. 39). Except otherwise noted, the reuse of this document is authorised under the Creative Commons Attribution 4.0 International (CC BY 4.0) licence (https://creativecommons.org/licenses/by/4.0/). This means that reuse is allowed provided appropriate credit is given and any changes are indicated. For any use or reproduction of photos or other material that is not owned by the EU, permission must be sought directly from the copyright holders. All content © European Union, 2020 How to cite this report: Janssen, S., Thunis, P., FAIRMODE Guidance Document on Modelling Quality Objectives and Benchmarking, version 3.2, EUR 30264 EN, Publications Office of the European Union, Luxembourg, 2020, ISBN 978-92-76-19745-4, doi:10.2760/30226, JRC120649 Contents Acknowledgements ................................................................................................ 1 Abstract ............................................................................................................... 2 Version history ...................................................................................................... 3 1 INTRODUCTION ................................................................................................. 4 1.1 Why Modelling Quality Objectives? ................................................................. 4 1.2 What are the purposes of this Document? ....................................................... 4 1.3 Who is the target audience of this Document? ................................................. 4 1.4 What are the main components of this Document? ........................................... 5 2 BENCHMARKING: A WORD OF CAUTION ............................................................... 6 3 DEFINITIONS .................................................................................................... 7 4 MAIN ASSUMPTIONS .......................................................................................... 9 5 MODELLING INDICATORS AND CRITERIA ........................................................... 10 5.1 Statistical indicators ................................................................................... 10 5.2 Modelling quality indicator (MQI) and modelling quality objective (MQO) .......... 11 5.2.1 MQI and MQO for hourly/daily/maximum daily 8-hour mean concentration data 11 5.2.2 MQI and MQO for yearly average model results ..................................... 12 5.2.3 The 90% principle .............................................................................. 12 5.3 Fulfilment of the hourly/daily vs annual MQO ................................................ 13 5.4 Calculation of the associated model uncertainty ............................................. 13 5.5 Comparison to values in the AQD ................................................................. 14 5.6 Modelling performance indicators (MPI) and criteria (MPC) .............................. 14 5.6.1 Temporal MPI and MPC ....................................................................... 15 5.6.2 Spatial MPI and MPC .......................................................................... 16 5.7 Data requirements for application of the MQI and MPI .................................... 16 5.7.1 Measurement data quality ................................................................... 17 5.7.2 Minimum number of stations for application of the MQO ......................... 17 5.7.3 Data assimilation / data fusion ............................................................ 17 5.7.4 Exceptional cases............................................................................... 18 6 REPORTING MODEL PERFORMANCE ................................................................... 20 6.1 Hourly data ............................................................................................... 20 6.1.1 Target Diagram ................................................................................. 20 6.1.2 Summary Report ............................................................................... 21 6.2 Yearly average data ................................................................................... 22 6.2.1 Scatter Diagram ................................................................................ 22 6.2.2 Summary Report ............................................................................... 23 7 OPEN ISSUES .................................................................................................. 25 i 7.1 Performance criteria for high percentile values .............................................. 25 7.2 Application of the procedure to other parameters........................................... 25 8 FORECASTING & EXCEEDANCES INDICATORS ..................................................... 27 8.1 The forecast Modelling Quality Indicator ....................................................... 27 8.2 Threshold exceedance indicators.................................................................. 28 8.3 Diagrams in the DELTA tool ........................................................................ 29 8.3.1 Forecast Target diagram ..................................................................... 29 8.3.2 Forecast Performance Diagram ............................................................ 30 8.3.3 Forecast diagrams using Air Quality Indices .......................................... 32 8.3.4 Diagrams for exceedance indicators ..................................................... 32 8.3.5 Summary report ................................................................................ 34 9 REFERENCES................................................................................................... 37 9.1 Peer reviewed articles ................................................................................ 37 9.2 Reports/ working documents / user manuals ................................................. 37 Annexes ............................................................................................................. 39 ANNEX 1. MEASUREMENT UNCERTAINTY ............................................................ 42 ANNEX 2. OVERVIEW OF EXISTING LITERATURE ................................................. 44 ANNEX 3. RELATED TOOLS ............................................................................... 47 List of abbreviations and definitions ....................................................................... 39 ii Acknowledgements The authors thank the Fairmode community for their contributions as well as for the fruitful discussions that took place around the particular topic of MQO and benchmarking. The authors wish to acknowledge in particular the contributions of: Mario Adani, ENEA, Italy Claudio Carnevale, University Brescia, Italy Cornelis Cuvelier, ex-JRC, Ispra, Italy Pawel Durka, IOS, Poland Emilia Georgieva, National Institute of Meteorology and Hydrology, Bulgaria Cristina Guerreiro, NILU, Norway Laure Malherbe, INERIS, France Bino Maiheu, VITO, Belgium Frederic Meleux, INERIS, France Alexandra Montero, University Aveiro, Portugal Ana Miranda, University Aveiro, Portugal Helge Olesen, Aarhus University, Denmark Florian Pfäfflin, IVU Umwelt GmbH, Germany Jenny Stocker, CERC, UK Gabriela Sousa Santos, NILU, Norway Amy Stidworthy, CERC, UK Michele Stortini, ARPA Emilia Romagna, Italy Elke Trimpeneers, IRCEL, Belgium Peter Viaene, VITO, Belgium Keith Vincent, Ricardo AEA, UK Lina Vitali, ENEA, Italy Joost Wesseling, RIVM