Weather Forecast Improvements at the UK Met Office: Responding to the Big Data Challenge

Vicky Pope

www.metoffice.gov.uk © Crown Copyright, Met Office 2018 | Agenda

What is the Met Office? What is the problem? How to think differently

How are we using technology? What are we doing differently? Why is it relevant to you?

www.metoffice.gov.uk 2 © Crown Copyright, Met Office 2017 | What is the Met Office purpose?

Protection Prosperity Well-being

• £30bn of economic value to the UK (2005 – 2015 London Economics, 2015) • 14:1 return on tax-payer investment (London Economics, 2015) • Additional £1.2bn to be delivered from 2015 £100m capital grant

www.metoffice.gov.uk 3 © Crown Copyright, Met Office 2017 | What is the Met Office?

UK Government Science Part of UK Government - A world-leading science all Government employees institute in Earth science

Commercial Business UN International Community As a trading fund – Part of a ~200 strong able to participate in international community, competitive markets UN treatice

Necessarily complex driven by complex environment

www.metoffice.gov.uk 4 © Crown Copyright, Met Office 2017 Weather Information Value Chain

Monitor Model & Decision Economic & Interpret Produce Forecast making Value Observe Monitor Model and observations Model & & Forecast Observe improvements

November March 2016 Spring 2017 2017 Autumn 2017 2018-19 2016

Increased Bigger UK Global New radar vertical domain model Better use of network resolution resolution observations Extend runs increase to Run UK Precipitation Couple to 5 days 10km ensembles to Biggest type atmosphere 5 days improvement Doppler and ocean Improved New UK in a decade winds forecasts of hourly Increased Better showers nowcasts resolution convection

Improved showers Pflops

• Data • Gridded forecast data • Spot data • Observations • Warnings and Guidance • National Severe Weather Warnings • Guidance • Scripts • Forecaster/Meteorologist briefings • Verbal and web

24/7 support © Crown copyright Met Office Public Weather Service reach

Branded Social and reach economic (trusted benefit for information UK economy provider) Met Office drivers Revenue to offset cost to Drive tax payer and innovation reinvest

Service provider arrangement

Monitor Model & Decision Economic & Interpret Produce Forecast making Value Observe

World Met Service Meteorological providers Organisation Interoperability Services for the Information Produce Economy - INSPIRE

Discovery Catalogue

View Visualisations

Download Data downloaded as is

Transform Reuse for interoperability

Invoke Remote compute

www.metoffice.gov.uk 11 © Crown Copyright, Met Office 2017 | The problem - volume of data

• Doubling every 2 years • Driving transformation every Met Office 5 year cycle • ‘We ain’t seen nothing yet’

www.metoffice.gov.uk 12 © Crown Copyright, Met Office 2017 | How are we using technology?: Becoming the cloud?

2 x Internet gateways

Supercomputer 2 tb/hour – 20 -> 200gb/s Observations Direct connect Cloud

Science STAGING Other

• AWS Direct Connect high performance connection • Split platform (hybrid platform, not hybrid cloud) • ‘Cloudy’ wherever possible

www.metoffice.gov.uk 13 © Crown Copyright, Met Office 2017 | How using technology?: Becoming the cloud?

Enterprise & Science IT Digital IT

http://www.informaticslab.co.uk/

1. Improving workflows for forecasters & scientists Tech: Big data engines, processing, visualisation and Machine Learning Internal partners: Operations, Observations, Science, Technology

2. Faster physical models 3. End-users systems for Tech: Machine Learning decision-making Internal partners: Science 1. Expert users, tools andModels platforms at Exascale 3. Public Weather Service Reach Tech: Chatbots, NLP 2. Models Internal partners: Public Weather Service, at Exascale Digital Reach

www.metoffice.gov.uk 15 © Crown Copyright, Met Office 2017 | What are we doing differently? UK climate projections 2018: Projections over land: updated probability distributions, evolving over time We will update the pdfs from UKCP09. The new pdfs will be on a seasonal resolution but also available for longer term time-averages.

Taking account of: • CMIP5 models • New observations • Improved approach

time | What are we doing differently? UK climate projections 2018: Projections over land: 60 km scale realisations of future climate

We will produce approx. 20 realisations of HadGEM3 using a perturbed parameter approach and focusing on achieving a diversity of climate responses.

×20

time | What are we doing differently? UK climate projections 2018: Projections over land: 60 km scale realisations of future climate

We will produce approx. 20 realisations of HadGEM3 using a perturbed parameter approach and focusing on achieving a diversity of climate responses.

We’ll place the global realisations into the context of the pdfs

time | What’s in it for you? New ways of working

Informatics lab

Vicky Pope