UNIVERSITY OF THE BASQUE COUNTRY UPV/EHU DOCTORAL THESIS Application of machine learning techniques to weather forecasting Author: Supervisors: Pablo Rozas Larraondo Prof. Iñaki Inza Cano Prof. José A. Lozano Dissertation submitted to the Department of Computer Science and Artificial Intelligence of the University of the Basque Country UPV/EHU as partial fulfilment of the requirements for the PhD degree in Computer Science. Donostia, November 6, 2018 (cc)2019 PABLO ROZAS LARRAONDO (cc by 4.0) v “Climate is what we expect, weather is what we get.” Mark Twain iii UNIVERSITY OF THE BASQUE COUNTRY UPV/EHU Abstract Computer Science Faculty Computer Science and Artificial Intelligence Department Application of machine learning techniques to weather forecasting by Pablo Rozas Larraondo Weather forecasting is, even today, an activity mainly performed by humans. Al- though computer simulations play a major role in modelling the state and evolution of the atmosphere, there is a lack of methodologies to automate the interpretation of the information generated by these models. This doctoral thesis explores the use of machine learning methodologies to solve specific problems in meteorology and particularly focuses on the exploration of methodologies to improve the accuracy of numerical weather prediction models. The work presented in this manuscript comprises two different approaches of applying machine learning to weather fore- casting problems. In the first part, classical methodologies, such as multivariate non- parametric regression and binary trees, are used to perform regression on meteoro- logical data. This first part, particularly focuses on forecasting wind, whose circular nature creates interesting challenges for classic machine learning algorithms. The second part of this thesis explores the analysis of weather data as a generic struc- tured prediction problem using deep neural networks. Neural networks, such as convolutional and recurrent networks, provide a method for capturing the spatial and temporal structure inherent in weather prediction models. This part explores the potential for deep convolutional neural networks to solve difficult problems in meteorology, such as modelling precipitation from basic numerical model fields. The research underpinning this thesis serves as an example of how collaboration between the machine learning and meteorology research communities is mutually beneficial and leads to advances in both disciplines. Weather forecasting models and observational data represent unique examples of large (petabytes), structured and high-quality data sets, which the machine learning community demands for developing the next generation of scalable algorithms. v Acknowledgements There are so many people I would like to thank for helping me and supporting me along these years. I will try to keep this list as brief as possible. To both my PhD supervisors Prof. Jose A. Lozano and Prof. Inaki Inza Cano • for their guidance and patience. To the Australian National University and the National Computational Infras- • tructure for their support in carrying out the last part of this research. To my family and friends for being there during these years offering their un- • conditional love. vii Contents Abstract iii Acknowledgements v 1 Introduction and state-of-the-art 1 1.1 Contents and structure ............................ 1 1.2 The origins and evolution of weather forecasting ............. 1 1.3 Numerical weather prediction ........................ 5 1.4 The sources of weather data ......................... 8 1.5 Machine learning ............................... 10 1.6 Weather forecasting: The machine learning approach .......... 17 2 Objectives, hypothesis and methodology 27 2.1 Objectives ................................... 27 2.2 Hypothesis ................................... 27 2.3 Methodology ................................. 28 2.3.1 Use of circular kernel regression for improving wind forecasting 28 2.3.2 Circular regression trees ....................... 30 2.3.3 Convolutional neural networks for precipitation classification . 32 2.3.4 Convolutional encoder-decoders for image to image regression 33 3 Results 35 4 Conclusions and future work 73 4.1 Conclusions .................................. 73 4.2 Future Work .................................. 75 Bibliography 77 ix List of Figures 1.1 Representation of Richardson’s idea of a “forecast factory”, which would employ some sixty-four thousand human computers sitting in tiers around the circumference of a giant globe, solving the equa- tions to forecast the weather. (Source: http://cabinetmagazine.org/ issues/27/foer.php). ............................ 2 1.2 Example of an F80 regular Gaussian grid used by some NWP to rep- resent the Earth’s atmosphere (Source: https://www.ecmwf.int). ... 3 1.3 NOAA CFS output for the global atmospheric precipitable water on March 15, 1993 for a 12-hour accumulation period (Source: https: //www.ncdc.noaa.gov). ............................ 4 1.4 An ensemble of forecasts produces a range of possible scenarios given an initial probability distribution of a forecasted parameter. The dif- ferent ensemble members provide an indication of the possible re- sulting scenarios based on a probability distribution. (Source: https: //www.ecmwf.int). .............................. 5 1.5 The performance of the Ensemble Prediction System (EPS) has im- proved steadily since it became operational in the mid-1990s, as shown by this skill measure for forecasts of the 850 hPa temperature over the northern hemisphere at days 3, 5 and 7. Comparing the skill measure at the three lead times demonstrates that on average the performance has improved by two days per decade. The level of skill reached by a 3-day forecast around 1998/99 (skill measure = 0.5) is reached in 2008-2009 by a 5-day forecast. In other words, today a 5-day forecast is as good as a 3-day forecast 10 years ago. The skill measure used here is the Ranked Probability Skill Score (RPSS), which is 1 for a per- fect forecast and 0 for a forecast no better than climatology. (Source: https://www.ecmwf.int). .......................... 6 1.6 Comparison between the horizontal resolutions used in global models today [30 km] (b) and the equivalent models 10 years ago [87.5 km] (a). (Source: http://www.climatechange2013.org). ........... 7 1.7 The Global Observing System (GOS) consists of a network of syn- optic surface-based observations made at over 11000 land stations, by about 7000 ships and 750 drifting buoys at sea and around 900 upper-air stations, together with reports from aircraft and remotely sensed data from geostationary and polar orbiting satellites. (Source: http://www.wmo.int/pages/prog/www/OSY/GOS.html). ......... 9 1.8 Comparison of a non-linear regression model (blue) and a linear model (black) representing a 2-dimensional data set. Source: Creative Com- mons by M. Giles ............................... 13 1.9 Comparison of the shape of seven common window functions used in kernel regression. Source: Creative Commons by Brian Amberg ... 14 x 1.10 This figure represents a comparison between a 24-hour prediction of daily mean temperature and the observed temperature values at In- dianapolis (USA). Source (Malone, 1955) .................. 18 1.11 Example of a decision tree used to forecast the event of hail based on thresholds for different observed and NWP parameters. Source (McGovern et al., 2017) ............................ 20 1.12 Sample images of atmospheric rivers (jet-streams) correctly classified and extracted from a multi-Terabyte NWP dataset by a deep CNN model. Source (Liu et al., 2016) ....................... 24 1.13 Comparison of the traditional NWP and machine learning approaches to weather forecasting. ............................ 25 2.1 Relationship between GFS and METAR wind speed values from San Sebastian. GFS wind direction is represented using a color scale, with yellow colours showing northerly winds and clue colours represent- ing southerly winds. ............................. 29 2.2 Example of the proposed circular regression tree and a representation of how the space is divided in contiguous regions. ............ 31 2.3 Example of the resulting Class Activation Maps for an ERA-Interim CNN, trained using the observed precipitation at Helsinki-Vantaa air- port, EFHK, (left) and Rome Fiumicino airport, LIRF, (right). Coast- lines have been overlaid as a reference for readers. ............ 33 2.4 Representation of the transformations performed by the encoder-decoder network to the geopotential height field and its transformation into a field representing the total precipitation field for the same region. ... 34 xi I would like to dedicate this work to Kate, Amaia and Tomas, who, one by one, came into my life during this time and made it all possible. 1 Chapter 1 Introduction and state-of-the-art 1.1 Contents and structure This doctoral thesis is conceived with the objective of exploring new avenues for the application of machine learning methodologies to the field of weather forecast- ing. One of the main objectives is to publish the research outcomes in peer-reviewed publications, as a way of measuring its relevance and impact within the meteoro- logical and machine learning communities. At the moment of the defending this thesis, two manuscripts have been published, another manuscript was presented in a workshop held as part of a well known machine learning conference and a third paper has been submitted to a weather forecasting journal. This document aims to provide the