Metview in a Pythonic World EGOWS 2019, KNMI, De Bilt, The Netherlands Iain Russell Development Section, ECMWF Thanks to Linus Magnusson and Martin Janousek © ECMWF November 13, 2019 What is Metview? • Workstation software for researchers and operational analysts – Runs on UNIX, from laptops to supercomputers • Retrieve/manipulate/visualise/examine meteorological data • Batch mode or graphical user interface • Can access MARS, either locally or through the Web API • Open Source under Apache Licence 2.0 • Metview is a co-operation project with INPE (Brazil) EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 2 Built on top of ECMWF software packages Metview Data decoding Regridding Plotting Data Access ecCodes Other (NetCDF, (GRIB, ODB_API Geopoints MIR Magics MARS CDS Files WMS URL BUFR) , CSV) EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 3 Using Metview • Icon-based user interface – interactive investigation of data – icons represent data, settings and processes – icons can be chained together – output from one is input to another • e.g. filter fields from a certain date, then pass that to the Cross Section icon • Powerful Python/Macro scripting language – more serious computations – batch or interactive usage EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 4 Generating Python code from the GUI EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 5 Generating Python code from the GUI EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 6 Data Formats • GRIB, BUFR, NetCDF, ODB, Geopoints, CSV • Plot • Examine • Filter, regrid, masking • Maths, Boolean • Specialised: – Cross section – Thermodynamics – Gradient – Vertical integration – Model to pressure lev – Etc EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 7 Contouring schemes • Plenty of options for complete customisation of palettes EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 8 Contouring schemes • A set of pre-defined palettes is also available – But you still have to supply the mapping between values and colours EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 9 Contouring schemes • Can select from pre-defined styles – the styles come from ecCharts – everything is done for you – or choose “Contour Automatic Setting = ECMWF” – style will be chosen based on meta-data EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 10 Complete ecCharts Layers • The ecCharts icon goes further – retrieves data from MARS and styles it EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 11 Macro language • Powerful, high-level scripting language • Native handling of major data types – e.g. fieldsets (GRIB) • Some nice stuff, e.g. model-obs differences (gridded minus scattered data) EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 12 Goals of a Python interface • Combine the power of Macro and the Python ecosystem – High-level meteorological aware functions and objects – Thin Python bindings on top of the core C++ Metview code – Easy translation from Macro to Python – With help from B-Open EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 13 Goals of a Python interface (2) • Opens up new development environments • E.g. Jupyter notebooks – Great teaching tool – Mentioned quite a lot in ECMWF’s recent workshop “Building reproducible workflows for earth sciences” EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 14 Creating a Python interface • Python provides all of the language features that are available in Macro (e.g. loops, conditionals, functions), so no need to try to ‘extend’ it • Python provides the main ‘primitives’ that Metview uses, e.g. numbers, strings, datetimes, lists, dictionaries • So the only things we need to provide are: – All the Macro functions – Correct handling of those ‘primitive’ types – Correct handling of indexing (Python uses 0-base, Macro uses 1-base) – Specific classes, e.g. Fieldset, Geopoints, Odb – Make them Pythonic, e.g. iterable containers – Overload operators, e.g. diffs = fieldset1 – fieldset2 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 15 Implementation • First: split Metview’s Macro module into a shared library plus a small executable that links to it • Use cffi in Python to load the library – Need a header file with C-style function prototypes for all API calls • Obtain list of all functions – Dynamically create a new Python function to call each one and add to namespace EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 16 Implementation • Add classes that know how to be passed between C++ and Python – Allow object-oriented, e.g. mv.interpolate(precip, 20.1, 19.3) == precip.interpolate(20.1, 19.3) • Add tests ☺ EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 17 Python bindings • Can extract data as numPy arrays from most data types – See Jupyter example “Principal component analysis of ensemble forecast fields” EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 18 Python bindings • Can obtain a pandas dataframe from Geopoints, Table and ODB – See Jupyter example “Difference between gridded field (GRIB) and scattered observations (BUFR)” EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 19 Python bindings Internally uses the Python • Can obtain an xarray dataset from GRIB module cfgrib, developed with – see Jupyter example “Computing ensemble B-Open mean and spread with xarray and plotting the results with Metview” EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 20 Metview now used in Verification • “The availability of Metview's classes and functions in Python stimulated a significant overhaul of a major operational verification software at ECMWF • Metview now takes care of all meteorological data decoding, filtering and post-processing, including geography-aware calculations, extending the applicability of the verification software to a wider range of data formats, grids and parameters • Thanks to utilising Metview the verification application significantly reduced its code base and opened up the path to a more modular and fast-to-develop software architecture.” • Work done with help from B-Open EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 21 Verification project – next steps • Development will continue into 2020 with a focus on enhancing its modularity, building a 'verification toolbox' with an extended scope to cover the wide range of recent and future verification demands in research and operations EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 22 Metview in Diagnostics • “For model evaluation and diagnostics, Metview's Python interface is a clear step forward • Here the combination in Python of using Metview for the calculations on the GRIB files and the Pandas module for making the time-series analysis has proven to be very powerful • One example application is a simple cyclone tracker aimed for advanced diagnostics” EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 23 Metview in Diagnostics (2) • “The Python script uses the Metview module to first retrieve data from the MARS archive • Next, a Python function performs the cyclone tracking by using several Metview functions on the fieldset containing the mean- sea-level-pressure • It returns a new position of the cyclone, which is used together with the distance(), mask() and integrate() functions in Metview to obtain various diagnostic quantities • These are collected in a Pandas array, which can then for example be used for plotting with the matplotlib library” EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 24 Future • Ability to add new Metview modules written in Python, not C++ – Should increase community contributions • Build on this work to create something that benefits from all the hindsight that we have, and utilises more existing Python modules to provide a richer data processing / visualisation experience • Already in prototype phase: a new plotting module written in Python and using the SkinnyWMS EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 25 Examples • See the Gallery for Macro and Python examples EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 26 Examples (2) EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 27 Examples • See the Jupyter Notebooks for more Python EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 28 Tutorials • Plenty of material online including tutorials, but most of it is currently Macro-based (very easy to convert though) EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 29 Metview Availability • Available on ECMWF systems: – Versioned using the ‘module’ system – [module swap metview/new] – metview • On other systems: – Install from RPM (Linux) – Install from conda (Linux and macOS) – Ubuntu package – Build from source – Build from bundle – pip install metview EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 30 For more information… • Email the developers: – [email protected] • Or ask User Support – [email protected] • Visit our web pages: – http://confluence.ecmwf.int/metview • Documentation and tutorials available • Gallery of examples Questions? EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 31.