Guidance on Verification of Operational Seasonal Climate Forecasts 2018 edition WEATHER CLIMATE WATER CLIMATE WEATHER WMO-No. 1220 Guidance on Verification of Operational Seasonal Climate Forecasts 2018 edition WMO-No. 1220 EDITORIAL NOTE Typefaces employed in this volume do not signify standard or recommended practices, and are used solely for legibility. The word shall is used to denote practices that are required for data representation to work. The word should denotes recommended practices. METEOTERM, the WMO terminology database, may be consulted at http://public.wmo.int/en/ resources/meteoterm. Readers who copy hyperlinks by selecting them in the text should be aware that additional spaces may appear immediately following http://, https://, ftp://, mailto:, and after slashes (/), dashes (-), periods (.) and unbroken sequences of characters (letters and numbers). These spaces should be removed from the pasted URL. The correct URL is displayed when hovering over the link or when clicking on the link and then copying it from the browser. WMO-No. 1220 © World Meteorological Organization, 2018 The right of publication in print, electronic and any other form and in any language is reserved by WMO. Short extracts from WMO publications may be reproduced without authorization, provided that the complete source is clearly indicated. Editorial correspondence and requests to publish, reproduce or translate this publication in part or in whole should be addressed to: Chairperson, Publications Board World Meteorological Organization (WMO) 7 bis, avenue de la Paix Tel.: +41 (0) 22 730 84 03 P.O. Box 2300 Fax: +41 (0) 22 730 81 17 CH-1211 Geneva 2, Switzerland Email: [email protected] ISBN 978-92-63-11220-0 NOTE The designations employed in WMO publications and the presentation of material in this publication do not imply the expression of any opinion whatsoever on the part of WMO concerning the legal status of any country, territory, city or area, or of its authorities, or concerning the delimitation of its frontiers or boundaries. The mention of specific companies or products does not imply that they are endorsed or recommended by WMO in preference to others of a similar nature which are not mentioned or advertised. ACKNOWLEDGEMENTS Lead author: Dr Simon J. Mason, Senior Research Scientist, International Research Institute for Climate and Society (IRI), United States of America The Guidance on Verification of Operational Seasonal Climate Forecasts has been prepared under the auspices of the WMO Commission for Climatology. Reviewers: Mr Jean-Pierre Céron (Météo France) Mr Akihiko Shimpo (Japan Meteorological Agency) Dr William Wang (Bureau of Meteorology, Australia) Dr Normand Gagnon (Environment and Climate Change Canada) Dr Richard Graham (Met Office, United Kingdom of Great Britain and Northern Ireland) Dr Anthony Barnston (IRI, United States) CONTENTS Page EXECUTIVE SUMMARY . vii 1 . INTRODUCTION . 1 2 . FORECAST AND VERIFICATION DATA . 3 2.1 Defining the target variable ................................................... 3 2.2 Sizes of forecast regions ...................................................... 4 2.3 Gridding of forecasts ......................................................... 5 2.4 Verification using station data ................................................. 5 2.5 Data inconsistencies ......................................................... 6 3 . ATTRIBUTES OF “GOOD” FORECASTS . 8 3.1 Types of forecast “goodness” .................................................. 8 3.2 Probabilistic forecasts and forecast quality ...................................... 9 3.2.1 Attributes of “good” probabilistic forecasts ................................ 10 3.2.1.1 Resolution ................................................... 10 3.2.1.2 Discrimination ............................................... 10 3.2.1.3 Reliability ................................................... 11 3.2.1.4 Sharpness ................................................... 12 3.2.1.5 Skill ........................................................ 12 3.2.2 Attributes of individual probabilistic forecast maps ......................... 13 4 . MEASURING FORECAST QUALITY . 15 4.1 The verification of climatological forecasts ...................................... 15 4.2 Measuring the quality of series of forecasts ...................................... 16 4.2.1 Measuring discrimination ............................................... 16 4.2.1.1 Relative operating characteristics ............................... 16 4.2.1.2 Generalized discrimination .................................... 20 4.2.2 Measuring resolution .................................................. 22 4.2.2.1 Resolution components of multi-attribute scores .................. 22 4.2.2.2 Hit scores as measures of resolution ............................. 23 4.2.3 Measuring reliability ................................................... 25 4.2.4 Measuring multiple attributes ........................................... 27 4.2.5 Detailed diagnostics ................................................... 29 4.3 Measuring the quality of individual forecast maps ................................ 33 4.3.1 Scoring of attributes ................................................... 33 4.3.2 Model diagnostics ..................................................... 35 5 . UNCERTAINTY OF RESULTS . 36 APPENDIX A . WEIGHTED VERSIONS OF THE VERIFICATION SCORES . 37 APPENDIX B . CALCULATION OF THE RECOMMENDED SCORES AND GRAPHS . 42 APPENDIX C . GLOSSARY . 59 REFERENCES . 65 EXECUTIVE SUMMARY The purpose of this publication is to describe and recommend procedures for the verification of operational probabilistic seasonal forecasts, including those from the Regional Climate Outlook Forums (RCOFs), National Meteorological and Hydrological Services and other forecasting centres. The recommendations are meant to complement the WMO Commission for Basic Systems Standardized Verification System for Long-range Forecasts (SVSLRF). SVSLRF defines standards for verifying model outputs from Global Producing Centres (GPCs), and so includes procedures for measuring the quality of ensemble prediction systems. In contrast, the procedures described in this publication are exclusively for verification of probabilistic forecasts, which may be model outputs, expert subjective assessments, or a combination of both. A second difference from SVSLRF is that procedures described in this publication are concerned not only with verification of a history of forecasts, but also of forecasts for one specific target period – for last year’s RCOF forecast, for example. The recommended procedures range in complexity from simple measures for communicating forecast quality to non-specialists, to detailed diagnostic procedures for in-depth analyses of the various strengths and weaknesses of forecasts. Interpretive guides for each of the procedures are included to assist the user in understanding the verification results, and worked examples are included in Appendix B. A glossary of technical terms is also provided in Appendix C. Ideally the multiple attributes of forecast quality should be measured individually, but some commonly used procedures measure more than one attribute at once. These procedures can lead to results that are difficult to interpret, and may lead to misleading conclusions. Alternative procedures that measure individual attributes are suggested in preference throughout this guidance publication because of their simpler interpretation and more informative results. Nevertheless, the occasional need for summary scores is recognized, and some suggestions are presented. While reliability (do the forecast probabilities give an accurate indication of the uncertainty in the outcome?) is unquestionably an important attribute, ultimately the most important attributes are resolution or discrimination. Resolution measures whether the outcome differs given different forecasts, while discrimination measures whether the forecasts differ given different outcomes. As long as there is some resolution or discrimination the forecasts contain potentially useful information, regardless of how poor the reliability is. For detailed diagnostics of forecast quality, reliability diagrams are recommended; these diagrams measure reliability and resolution. SVSLRF recommends constructing the diagrams with points at 10% increments, consistent with typical numbers of ensemble members available for estimating the forecast probabilities. It is recommended in this publication that the diagrams be drawn with points at 5% increments rather than 10% because of the general practice of issuing seasonal forecasts with probabilities rounded to the nearest 5%. A number of suggestions are made for simplifying the communication of reliability diagrams to non-specialists, including: presenting the information in tables rather than graphs; fitting regression lines to the reliability curves and describing their slopes in terms of the change in frequency of occurrence given 10% increments in the forecast probability; and so-called “tendency” diagrams, which provide simple visual indications of unconditional biases. For more technical use, the reliability and resolution components of the ignorance and Brier scores (BS) are suggested. Of these, the components of the ignorance score are perhaps to be preferred because of its asymmetric measurement of probability errors, which is appropriate for categories that do not have a climatological probability of 0.5. Unfortunately, with only