TheThe challengeschallenges ofof thethe ECMWFECMWF graphicsgraphics packagespackages
Sylvie Lamy-Thépaut Graphics Section Slide 1 ECMWF
UCAR/NCAR, 5 August 2009 1 Outline z Who are we? - ECMWF - The Graphics Section z What are our Missions? - Magics - Metview - Wrep : our new web project z What are our Challenges? - Web applications - New sources of data - High Volume of data Slide 2 - Modern Interactive Desktops
UCAR/NCAR, 5 August 2009 2 What is ECMWF? z European Centre for Medium Range Weather Forecasts z We provide operational medium- and extended-range forecasts and a state-of-the-art super-computing facility for scientific research. z Supported by 31 States z 220 Employees z Founded 33 years ago z Based in Reading, west of London,Slide United 3 Kingdom.
UCAR/NCAR, 5 August 2009 3 Supporting States and Co-operation
Belgium Ireland Portugal Denmark Italy Switzerland Germany Luxembourg Finland Spain The Netherlands Sweden France Norway Turkey Greece Austria United Kingdom Co-operation agreements or working arrangements with: Czech Republic Montenegro ACMAD Croatia Morocco ESA Estonia Romania EUMETSAT Hungary Serbia WMO Iceland Slovakia JRC
Latvia Slovenia Slide 4 CTBTO Lithuania CLRTAP
UCAR/NCAR, 5 August 2009 4 Who are We? ECMWF
Research Operation Department Department
Meteorological Division
Graphics Section
INPE/CPTEC Slide 5 Brazil Meteo-France
UCAR/NCAR, 5 August 2009 5 What are our Missions? z We are here to help researchers and analysts to access, manipulate and visualise a wide variety of meteorological data. z We develop and maintain : - A graphical package with various APIs : Magics - A desktop based application : Metview. z We participate in the new web project - Easy description and production of plots. z To do that, we use - Unix platforms - Mostly C++ language - Perforce for versioning Slide 6 - Eclipse
UCAR/NCAR, 5 August 2009 6 Magics: Our Graphical package z Magics is meteorologically-oriented - GRIB - BUFR - Specific Visualisation
Slide 7
UCAR/NCAR, 5 August 2009 7 Magics
z Magics provides a simple API -Large set of parameters -Small number of FORTRAN callable subroutines
Slide 8
UCAR/NCAR, 5 August 2009 8 Metview: Our meteorological workstation z Working environment for Operational and Research Meteorologists
z Co-operative project: - ECMWF - INPE/CPTEC (Brazil) - Meteo-France
Slide 9
UCAR/NCAR, 5 August 2009 9 Metview - Interactive z Icon-based interface (drag and drop) z Everything in Metview is an Icon z Data, visual definitions, macros
Slide 10
UCAR/NCAR, 5 August 2009 10 Metview - Data Processing z Meteorological Data Access and Processing Package z GRIB, BUFR, MARS, ODB, geopoints, …
Slide 11
UCAR/NCAR, 5 August 2009 11 Metview – Macro Language z Macro language - powerful meteorologically oriented language
9Simple script language 9Extensive list of operators/functions 9Macro programs: interactive or batch mode 9Automatically convert icons to equivalent macro code 9Macro editor – built-in or selected by user Slide 12 9NEdit: enhanced Macro editor
UCAR/NCAR, 5 August 2009 12 Our Challenges z Magics and Metview have now been up and running for the last 15 years! z They both needed some re-engineering to meet the new users requirements - New high resolution models - More satellite observation - More exchange of data - New web interfaces - New ways to export data for later visualisation ( Google-Earth) z The next generation is coming: Magics++ and MetviewSlide 13 4
UCAR/NCAR, 5 August 2009 13 Magics++ z Magics++ is object-oriented : Its object-oriented architecture allows easy integration of new data formats, new outputs, and new visualisations. z Magics++ is meteorologically oriented, but it is not a standalone application… z Magics++ is the visualisation component of a more complex framework. - Desktop applications, WMS …
TheThe newnew designdesign ofof Magics++Magics++ willwill allowallow itit toto bebe usedused inin thethe newnew generationgeneration ofof meteorologicalmeteorological workstations:workstations: DesktopDesktop oror Web-oriented!Web-oriented!Slide 14
UCAR/NCAR, 5 August 2009 14 Magics++
Slide 15
UCAR/NCAR, 5 August 2009 15 Magics++: new data inputs
Grib2Grib2 (gribAPI)(gribAPI) ODAODA
NetCDFNetCDF
Slide 16
UCAR/NCAR, 5 August 2009 16 Magics++: new contouring z Algorithms developed by Hiroshi Akima - documented in the ACM Transactions on Mathematical Software z New contouring has no license restrictions and we have full control of the code z INPE/CPTEC (Brazil) has successfully implemented a C++ version z Algorithms handle gridded and scattered data z Accuracy is configurable by the user, although Magics++ will always choose sensible automatic values by Slide 17 default
UCAR/NCAR, 5 August 2009 17 Magics++: new visualisations and projections
BoxplotBoxplot climagrclimagraamm
Slide 18 Taylor projection PolygonPolygon shading shading Taylor projection
UCAR/NCAR, 5 August 2009 18 Magics++: new outputs z Magics++ produces better publication-quality plots by supporting PNG, EPS and by optimizing PostScript output z Magics++ uses Cairo to generate PNG and PDF z We wrote our own SVG driver to have full control on the output. z We are also thinking in creating our own meta internal format for speeding the web production. Slide 19
UCAR/NCAR, 5 August 2009 19 Magics++: KML/KMZ output z Generates OGC compliant KML 2.2 z Very different to other 2D outputs z Generates time series
Slide 20
UCAR/NCAR, 5 August 2009 20 Magics++: ready for the web z It is a new software: can handle web requirements - Produces wide range of web formats: PNG, PDF, SVG, KML - Generates metadata info regarding the data displayed and legend - Generates simple JavaScript codes to enable map navigation z An XML based interface: MagML - The XML convention makes it easy to use in a web context - A MagML interpreter can be easily embedded in a complex web project allowing the generation of plot on demand - The MagML syntax is similar to the Metview icons convention z A JSON Interface tailored for the needsSlide 21 of our new web project.
UCAR/NCAR, 5 August 2009 21 Magics++: our programming experience z Autotools (configure) based installation enables easier spread of Magics++ - Users are more confident to update - Debian and Fedora community have or plan to package Magics++ z C++ proved again to be a good choice - Already used in Metview for 15 years - Fast, clear structured object-oriented code - Only issue: compiler support z Backwards compatibility - Important in an operational environment Slide 22 - Can limit new developments
UCAR/NCAR, 5 August 2009 22 Metview in the age of web services
Now that there is the ability of powerful web services, where does a meteorological workstation, such as Metview, come in? z The increasing amount of data to be processed does still need processing speed best achieved by an optimized software z While maps allow visualisation on the web a workstation can give more tools to analyse and work with data itself z The tool, once installed, is always available and independent of network and other services z We also need a tool to design the products for the web at the first place!!! Slide 23
UCAR/NCAR, 5 August 2009 23 Metview 4: new development choices z Replace MAGICS with Magics++ - Offers all features of Magics++ z Replacement of Motif with Qt z Continue to use OpenGL API - Efficient - Well established z Use autotools for installation z Offer tools for: - post-processing & visualisation of model analysis and forecasts - observation monitoring - development of web products - model verification Slide 24
UCAR/NCAR, 5 August 2009 24 Metview 4: plans z A new visualisation module to take advantage of all the benefits provided by Magics++ z More functionality to handle NetCDFs z Revisit our macro language z More facilities to handle satellite data z MagML integration z OGC service client (and server?) z Full 64 bit memory support z Better installation tools (Autotools)
Slide 25
UCAR/NCAR, 5 August 2009 25 MV4 : magnification tool to explore data
Slide 26
UCAR/NCAR, 5 August 2009 26 MV4 :examiners to explore large dataset
GrGrib2ib2 /NetCDF /NetCDFExamin Examinorsors
Slide 27 ODAODA examinor examinor
UCAR/NCAR, 5 August 2009 27 The Web era
Re-engineeringRe-engineering the the Web Web system system with with a a view view to to providing providing a a resilient resilient serviceservice with with interactive interactive features features such such as as zooming zooming and and on-demand on-demand productionproduction of of customised customised plots plots for for members members states… states… z Our current web has been available since January 2002 z The number of products on web continues to grow in response to user requests (currently more than half a million single gif images on the web site) z The users expectations of web services are increasing - High availability - more interactivity: zoom, pan, click Slide 28 z We need to plan for emerging standards (e.g. OGC/GIS/INSPIRE)
UCAR/NCAR, 5 August 2009 28 Magics++/Metview: on the server side z Easy description of products - MagML ot JSON (both being Metview like) z Generation of JavaScript for navigation, zooming, panning.. z Generation of metadata for title, legend… z Use of the macro language to perform computations on fields. - Threshold computation for probabilities maps. - Accumulation for rainfall. Slide 29 z Use of HADOOP to store the data
UCAR/NCAR, 5 August 2009 29 Slide 30
UCAR/NCAR, 5 August 2009 30 Magics++ : on the WMS Server side z Easy description of layers z Fast visualisation z High quality graphics z Definition and use of projection needs to be assessed - E.g. definition of polar stereographic projections z How should WMS served maps be updated? - On request, once or periodically z Work after this will aim at catalogue and feature services Slide 31
UCAR/NCAR, 5 August 2009 31 Magics++ : on the WMS Server side
KNMIKNMI
Meteo-FMeteo-Francerance
Slide 32
UCAR/NCAR, 5 August 2009 32 Metview 4 : on the WMS Client side z The Metview architecture makes the integration of new data layers easy. z The new WebClient icon allows the users to access and overlay data from external WMS. z Reliability of an external WMS? z Where to find the rules to overlay?
Slide 33
UCAR/NCAR, 5 August 2009 33 Metview 4
Slide 34
UCAR/NCAR, 5 August 2009 34 OGC standards and web services z ECMWF is committed to investigating the use of OGC standards and (web) services z We hosted jointly with Met Office and Météo France a Workshop in Nov 2008 z ECMWF wants to take an active role in the OGC Meteorology DWG z Any implementations need to be tested with external partners to ensure interoperability
Slide 35
UCAR/NCAR, 5 August 2009 35 2nd workshop on the use of GIS/OGC standards in meteorology z 23 – 25 November 2009 Toulouse France z To review the use of OGC standards in geo-sciences in Europe and worldwide.
More information at http://www.meteo.fr/cic/meetings/gis-ogc/
Slide 36
UCAR/NCAR, 5 August 2009 36 12th Workshop on Meteorological Operational Systems z 2nd –6th November 2009 at ECMWF, Reading, UK z Speakers are invited to report on “new trends in meteorological visualisation applications”
More information at www.ecmwf.int/newsevents/meetings/workshops/2009/MOS_12/ Slide 37
UCAR/NCAR, 5 August 2009 37 Thank you!
Slide 38
UCAR/NCAR, 5 August 2009 38