DOCSLIB.ORG

Configuration Management at ECMWF

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Configuration Management at ECMWF Configuration management at ECMWF Glenn Carver Paul Burton [email protected], [email protected] IS-ENES2 Configuration Management Workshop 2013, Met Office, Exeter, UK 23-24 September 2013 Slide 1 © ECMWF Who are we and what do we do? European Centre We are an independent international organisation funded by 34 States Medium-Range Up to fifteen days ahead. Today our products also include monthly and seasonal forecasts and we collect and store meteorological data. Weather Forecasts We produce global weather forecasts What do we have to achieve this? People About 260 staff, specialists and contractors Equipment State-of-the-art supercomputers and data handling systems Budget £50 million per year Experience 37 years Slide 2 © ECMWF Software at ECMWF IFS : Integrated Forecast System ● Approx 3 million lines of code. ● Includes forecast model, 4D-Var, tangent linear & adjoint models, land- surface and wave models, NEMO ocean model and single column model. ● Mostly Fortran90 with some C. ● New components in development use C++ (e.g. new I/O layer, new control layer) ● Operaonal and research use. Other soUware: ● Libraries and tools. e.g. grib_api, Metview. ● Moving to Open Source (Apache2). ● DisTnct development and management from IFS code. Configuraon management is different for IFS &’other’. Slide 3 © ECMWF Configuration management IFS: ● Perforce for last 10yrs, previously Clearcase. ● Perl wrappers are used to make commands easier for scienTsts, enforce local convenTons & addiTonal sanity checks. Based on previous VCS clearcase workflow. ● Code is separated into directory ‘projects’ which currently are compiled into separate libxx.a before the final link step. ● Parallel build restricted to parallel compilaon of libraries. ● Increasing dependency between projects (because of Fortran modules) enforces compilaon order of libraries. ● Current build system uses a separate dependency analysis step, and recursive makefiles. Old, inflexible, difficult to debug and slow. FCM provides opportunity to update & improve current IFS build system Slide 4 © ECMWF Configuration management (2) Other soUware packages: ● Managed by Forecast Products department. ● Separate Perforce repository. ● Libraries and tools e.g. grib_api, magics++, Metview, etc. ● Mix of languages: C, C++ and increasingly python. ● Moving from Perforce to ‘git’. Part of general desire to move away from Perforce. ● Moving from ‘autotools’ to ‘CMake’. Slide 5 © ECMWF OpenIFS Academic version of IFS. ● Much smaller codebase; forecast mode only. ● Needed to be easier to build for users. ● Useful testbed for technical changes suitable for IFS. ● Early users find ‘fcm make’ easy to use. ● No ‘wrappers’ around fcm as yet (e.g. WRF-style) but might be needed in future. FCM ● Only use ‘fcm make’ facility but other users (e.g. UK NCAS) will use more fcm funcTonality. ● Thanks to Dave & Ma for assistance. ● FCM well received at ECMWF. Slide 6 © ECMWF Benefits & Issues ● Benefits ○ Integrated & reliable dependency analysis ○ Fast & parallel ○ Removes inter-project dependency issues ○ Easy to change compile opTons per file ● Issues ○ Will require C++ support. ○ Beier access to log file - hidden nature not ideal. ○ Can’t see compiler warnings using -v and -vv is too verbose. ○ Temporary nature of final libXX.a for linking causes problems for tools like CrayPAT which inspect the link step. Slide 7 © ECMWF Summary FCM works very well with OpenIFS. Will be used for IFS and other ECMWF codes. Interested in integraon with git. Look forward to collaboraon with Met Office on FCM Interested in parTcipang in IS-ENES2 community evaluaon - believe this will be of benefit to ECMWF. Slide 8 © ECMWF .
Load more

Recommended publications
  • Hsafew2019 S2b.Pdf
    How to work with SSM products From Download to Visualization Apostolos Giannakos Zentralanstalt für Meteorologie und Geodynamik (ZAMG) https://www.zamg.ac.at 1 Topics • Overview • ASCAT SSM NRT Products • ASCAT SSM CDR Products • Read and plot ASCAT SSM NRT Products • Read and plot ASCAT SSM CDR Products • Summary 2 H SAF ASCAT Surface Soil Moisture Products ASCAT SSM Near Real-Time (NRT) products – NRT products for ASCAT on-board Metop-A, Metop-B, Metop-C – Swath orbit geometry – Available 130 minutes after sensing – Various spatial resolutions • 25 km spatial sampling (50 km spatial resolution) • 12.5 km spatial sampling (25-34 km spatial resolution) • 0.5 km spatial sampling (1 km spatial resolution) ASCAT SSM Climate Data Record (CDR) products – ASCAT data merged for all Metop (A, B, C) satellites – Time series format located on an Earth fixed DGG (WARP5 Grid) – 12.5 km spatial sampling (25-34 km spatial resolution) – Re-processed every year (in January) – Extensions computed throughout the year until new release 3 Outlook: Near real-time surface soil moisture products CDOP3 H08 - SSM ASCAT NRT DIS Disaggregated Metop ASCAT NRT SSM at 1 km (pre-operational)* H101 - SSM ASCAT-A NRT O12.5 Metop-A ASCAT NRT SSM orbit 12.5 km sampling (operational) H102 - SSM ASCAT-A NRT O25 Metop-A ASCAT NRT SSM orbit 25 km sampling (operational) H16 - SSM ASCAT-B NT O12.5 Metop-B ASCAT NRT SSM orbit 12.5 km sampling (operational) H103 - SSM ASCAT-B NRT O25 Metop-B ASCAT NRT SSM orbit 25 km sampling (operational) 4 Architecture of ASCAT SSM Data Services
    [Show full text]
  • Portage of AL26T1 Op4 on IBM Regatta P690 Machine
    Portage of AL26T1_op4 on IBM Regatta p690 machine . François Thomas [email protected] Revised and Approved by National Institute of Meteorology Tunisie April, 5th, 2004 Part 7: Data assimilation, Variational computations 1. S ummary This document outlines the work performed at INM from February, 22nd to March, 2nd. The tasks that were performed during this stay are: • system tasks : software upgrade (AIX, LoadLeveler, Parallel Environment, compilers), disk space management, installation of a web server, etc. • compilation and link of AL26T1_OP4 • unsuccessful attempt to compile ODB • implementation of a workload management strategy using LoadLeveler, WLM and vsrac, a tool developped by IBM Montpellier • test and implementation of an asynchronous method for running a forecast with slightly outdated coupling files from Meteo France • start a port of Metview/Magics We will describe all these tasks and give four appendices regarding : • software management under AIX • AIX system backup • users management with AIX • vsrac : task placement tool 2. S ystem tasks 2.1 Sof tware upgrade The first step of the work was to upgrade the system software to the latest levels. The table below summarizes the levels currently installed on the p690 Text (Body) Software name Software name AIX 5.1 ML5 : Maintenance Level 5 Parallel Environment 3.2.0.17 Load Leveler 3.1.0.22 xlf compiler 8.1.1.4 ESSL library 3.3.0.6 The best way to get system software fixes is to use the IBM support web site : http://www- 1.ibm.com/servers/eserver/support/pseries/fixes/. This service is free of charge. See Appendix A for details about the software management under AIX.
    [Show full text]
  • Article, We Describe the Architecture of for Using Data from Numerical Atmospheric Models for Re- the Tool
    Geosci. Model Dev., 5, 55–71, 2012 www.geosci-model-dev.net/5/55/2012/ Geoscientific doi:10.5194/gmd-5-55-2012 Model Development © Author(s) 2012. CC Attribution 3.0 License. A web service based tool to plan atmospheric research flights M. Rautenhaus, G. Bauer, and A. Dornbrack¨ Deutsches Zentrum fur¨ Luft- und Raumfahrt, Institut fur¨ Physik der Atmosphare,¨ Oberpfaffenhofen, Germany Correspondence to: M. Rautenhaus ([email protected]) Received: 10 August 2011 – Published in Geosci. Model Dev. Discuss.: 1 September 2011 Revised: 13 January 2012 – Accepted: 13 January 2012 – Published: 17 January 2012 Abstract. We present a web service based tool for the plan- of atmospheric research flight planning is to explore large ning of atmospheric research flights. The tool provides on- amounts of atmospheric prediction and observation data in line access to horizontal maps and vertical cross-sections of order to extract specific regions of interest. Subsequently, a numerical weather prediction data and in particular allows flight route is designed considering the scientific objectives the interactive design of a flight route in direct relation to of the campaign, the predicted atmospheric situation, and in- the predictions. It thereby fills a crucial gap in the set of strument, aircraft and legal constraints (illustrated in Fig. 1). currently available tools for using data from numerical atmo- Several efforts have been started in recent years to sup- spheric models for research flight planning. A distinct fea- port the planning of flights during aircraft-based field cam- ture of the tool is its lightweight, web service based architec- paigns. Besides the application of general purpose tools for ture, requiring only commodity hardware and a basic Internet the exploration of weather prediction and observation data connection for deployment.
    [Show full text]
  • I/O Analysis of Climate Applications
    I/O analysis of climate applications Arne Beer, MN 6489196, Frank Röder, MN 6526113 Introduction About the paper and our goals In this paper we analyze and present the strengths and weaknesses of different data structures required by some carefully picked climate and weather prediction models. Further we investigate the bare minimum of data required by those models. Another important aspect we will elaborate is the pre- and post- processing of data and the precise moment it occurs. In the following sections we will elucidate some models and the process on how to set them up as well as running sample cases. At the end there will be a section about the overall life cycle of data. Getting started With intent of getting an overview about the richness of climate applications and their land, ice, weather and ocean modulation, we took an in-depth look at some models. The main goal was to find a model with proper documentation and a license which allowed us to use it for our research and benchmarking purposes. Most model investigated by us were too old for our objective. In many cases there were practically no documentation, let alone active support or new releases. The next obvious step was to get an good overview of up to date and easy to handle models which are still supported. For this purpose we put a spreadsheet together from which we could choose the most promising ones. We came across some good looking models, but despite a good first impression, most of them are shipped with broken scripts, bad documentation or a hidden provision, which forbids us to use them.
    [Show full text]
  • Metview's Python Interface Opens New Possibilities
    from Newsletter Number 162 – Winter 2019/20 COMPUTING Metview’s Python interface opens new possibilities 11.4 11.5 10.4 11.7 11.6 12.2 12.6 12.9 12.2 12.0 13.5 13.3 12.7 14.1 13.1 12.8 13.0 13.5 12.0 12.7 11.8 12.7 12.5 12.6 13.4 12.6 14.5 14 13.9 13.3 14.0 13 13.1 12.5 13.2 12 13.5 14.0 13.1 14.1 14.1 11 13.8 14.1 14.6 13.6 14.1 13.6 13.1 13.7 10 13.8 12.8 12.7 www.ecmwf.int/en/about/media-centre/media-resources doi: 10.21957/hv3sp41ir5 Iain Russell, Linus Magnusson, Martin Janousek, Sándor Kertész Metview’s Python interface opens new possibilities This article appeared in the Computing section of ECMWF Newsletter No. 162 – Winter 2019/20, pp. 36–39 Metview’s Python interface opens new possibilities Iain Russell, Linus Magnusson, Martin Janousek, Sándor Kertész Metview is ECMWF’s interactive and batch processing software for accessing, manipulating and visualising meteorological data. Metview is used extensively both at ECMWF and in the Centre’s Member and Co-operating States. A national meteorological service may for example use it to plot fields produced by the ECMWF model and a regional model, calculate some additional fields such as temperature advection, and plot vertical cross sections. The recent addition of a Python interface to Metview has expanded its range of uses and has made it accessible to more potential users.
    [Show full text]
  • Metview in a Pythonic World EGOWS 2019, KNMI, De Bilt, the Netherlands
    Metview in a Pythonic World EGOWS 2019, KNMI, De Bilt, The Netherlands Iain Russell Development Section, ECMWF Thanks to Linus Magnusson and Martin Janousek © ECMWF November 13, 2019 What is Metview? • Workstation software for researchers and operational analysts – Runs on UNIX, from laptops to supercomputers • Retrieve/manipulate/visualise/examine meteorological data • Batch mode or graphical user interface • Can access MARS, either locally or through the Web API • Open Source under Apache Licence 2.0 • Metview is a co-operation project with INPE (Brazil) EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 2 Built on top of ECMWF software packages Metview Data decoding Regridding Plotting Data Access ecCodes Other (NetCDF, (GRIB, ODB_API Geopoints MIR Magics MARS CDS Files WMS URL BUFR) , CSV) EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 3 Using Metview • Icon-based user interface – interactive investigation of data – icons represent data, settings and processes – icons can be chained together – output from one is input to another • e.g. filter fields from a certain date, then pass that to the Cross Section icon • Powerful Python/Macro scripting language – more serious computations – batch or interactive usage EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 4 Generating Python code from the GUI EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 5 Generating Python code from the GUI EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 6 Data Formats • GRIB, BUFR, NetCDF, ODB, Geopoints, CSV • Plot • Examine • Filter, regrid, masking
    [Show full text]
  • Metview's New Python Interface
    Metview’s new Python interface – first results and roadmap for further developments EGOWS 2018, ECMWF Iain Russell Development Section, ECMWF Thanks to Sándor Kertész Fernando Ii Stephan Siemen © ECMWF October 15, 2018 What is Metview? • UNIX, Open Source under Apache Licence 2.0 • Metview is a co-operation project with INPE (Brazil) EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 2 High-level data processing with Metview Macro • We already have the Macro language… Forecast – observation difference: forecast = retrieve (…) # GRIB from MARS obs = retrieve (…) # BUFR from MARS t2m_obs = obsfilter(data: obs, parameter: 012004, output: 'geopoints’) plot(forecast – t2m_obs) EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 3 Why create a Python interface to Metview? • Metview’s Macro language is great; Python has even more language features • Not everyone knows Metview Macro! – Less learning curve for people who already know Python – Better for community-building – Learning Python is a good investment in general! • Enable Metview to work seamlessly within the Python eco-system – Bring Metview’s data processing and interactive data inspection tools into Python sessions ; interact with Python data structures – Use existing solutions where possible (e.g. for multi-dimensional data arrays, data models) • Enable Metview to be a component of the Copernicus Climate Data Store Toolbox EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 4 Current Status • Alpha release (developed with B-Open) – Beta release just around the corner • Available on github and PyPi – https://github.com/ecmwf/metview-python – pip install metview • Python layer only – this still requires the Metview binaries to be installed too EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 5 Macro / Python Macro comparison Python EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 6 Macro / Python Macro comparison Python EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS 7 Data types (1) • All Metview Macro functions can be called from Python, e.g.
    [Show full text]
  • Manuscript Submission Is Archived on Phaidra ( DOI:10
    Flex_extract v7.1 – A software package to retrieve and prepare ECMWF data for use in FLEXPART Anne Philipp1,2, Leopold Haimberger2, and Petra Seibert3 1Aerosol Physics & Environmental Physics, University of Vienna, Vienna, Austria 2Department of Meteorology and Geophysics, University of Vienna, Vienna, Austria 3Institute of Meteorology, University of Natural Resources and Life Sciences, Vienna, Austria Correspondence: Anne Philipp ([email protected]) Abstract. Flex_extract is an open-source software package to efficiently retrieve and prepare meteorological data from the European Centre for Medium-Range Weather Forecasts (ECMWF) as input for the widely-used Lagrangian particle dispersion model FLEXPART and the related trajectory model FLEXTRA. ECMWF provides a variety of data sets which differ in a number of parameters (available fields, spatial and temporal resolution, forecast start times, level types etc.). Therefore, the 5 selection of the right data for a specific application and the settings needed to obtain them are not trivial. Consequently, the data sets which can be retrieved through flex_extract by both authorised member-state users and public users as well as their properties are explained. Flex_extract 7.1 is a substantially revised version with completely restructured code, mainly written in Python3, which is introduced with all its input and output files and an explanation of the four application modes. Software dependencies and the methods for calculating the native vertical velocity η_, the handling of flux data and the preparation of 10 the final FLEXPART input files are documented. Considerations for applications give guidance with respect to the selection of data sets, caveats related to the land-sea mask and orography, etc.
    [Show full text]
  • Meeting the Challenges of the Next Generation of User Interfaces Iain
    Meeting the challenges of the next generation of user interfaces Iain Russell, Sylvie Lamy-Thépaut, Graphics Section, ECMWF Abstract Metview is a meteorological workstation designed to be a complete working environment for both the operational and research meteorologist. Its capabilities include powerful data access, processing and visualisation. It currently uses MAGICS 6 as its plotting library, but is being adapted to use the newest version, Magics++. This raises some interesting questions about the future of Metview’s user interface. Metview and Magics++ are presented in this paper, along with current developments in both. Introduction to Metview Metview is an interactive meteorological application designed to access, manipulate and visualise meteorological data on UNIX workstations. It was developed as part of a cooperation between ECMWF, INPE/CPTEC (Brazilian National Institute for Space Research / Centre for Weather Forecasts and Climate Studies), with assistance from Météo-France. Metview was conceived as having a modular architecture, where modules can be run on difference machines. Modules can be seen as ‘mini applications’ which sit on top of Metview’s kernel, adding both essential and optional functionality. Through this method, Metview can be easily extended by the addition of new modules. Standard software libraries such as OpenGL, Motif and PNG are used within Metview, easing its portability within the UNIX world. Metview has been installed on UNIX systems such as IBM’s AIX and various Linux distri - butions such as SuSE, RedHat and Ubuntu. Using Metview Metview can be run in either of two ways: interactively or in batch mode. The interactive user interface is an icon- driven drag-and-drop style interface.
    [Show full text]
  • A Software Package to Retrieve and Prepare ECMWF Data For
    Flex_extract v7.1.2 – A software package to retrieve and prepare ECMWF data for use in FLEXPART Anne Philipp1,2, Leopold Haimberger1, and Petra Seibert3 1Department of Meteorology and Geophysics, University of Vienna, Vienna, Austria 2Aerosol Physics & Environmental Physics, University of Vienna, Vienna, Austria 3Institute of Meteorology, University of Natural Resources and Life Sciences, Vienna, Austria Correspondence: Anne Philipp ([email protected]) Abstract. Flex_extract is an open-source software package to efficiently retrieve and prepare meteorological data from the European Centre for Medium-Range Weather Forecasts (ECMWF) as input for the widely-used Lagrangian particle dispersion model FLEXPART and the related trajectory model FLEXTRA. ECMWF provides a variety of data sets which differ in a number of parameters (available fields, spatial and temporal resolution, forecast start times, level types etc.). Therefore, the 5 selection of the right data for a specific application and the settings needed to obtain them are not trivial. Consequently, the data sets which can be retrieved through flex_extract by both member-state users and public users as well as their proper- ties are explained. Flex_extract 7.1.2 is a substantially revised version with completely restructured code, mainly written in Python3, which is introduced with all its input and output files and an explanation of the four application modes. Software dependencies and the methods for calculating the native vertical velocity η_, the handling of flux data and the preparation of 10 the final FLEXPART input files are documented. Considerations for applications give guidance with respect to the selection of data sets, caveats related to the land-sea mask and orography, etc.
    [Show full text]
  • Metview Computer User Training Course 2016
    Data Analysis and Visualisation Using Metview Computer User Training Course 2016 Iain Russell, Fernando Ii, Sándor Kertész, Stephan Siemen, Sylvie Lamy-Thépaut Development Section [email protected] Slide 1 © ECMWF September 30, 2016 1 Outline Day 1: Introduction, main features Day 2: Data (1) and processing Day 3: Data (2), time and graphs Day 4: Graphics formats, advanced usage Day 5: Batch jobs, exploring Metview Slide 2 Data Analysis and Visualisation using Metview © ECMWF 2016 2 Metview: meteorological workstation Retrieve/manipulate/visualise meteorological data Working environment for operational and research meteorologists Allows analysts and researchers to easily build products interactively and run them in batch mode Built on core ECMWF technologies: MARS, GRIB_API, Magics, ODB, Emoslib (ecCodes, MIR ) Open Source under Apache Licence 2.0 - Increased interest from research community Metview is a co-operation project withSlide 3INPE (Brazil) Data Analysis and Visualisation using Metview © ECMWF 2016 3 What is Metview? Access Manipulate Examine Data Overlay Plot Can be run interactively or in batch Can be easily installed and runs self-contained standalone - From laptops to supercomputers Slide 4 - No special data servers required (but can be easily connected to MARS or local databases) Data Analysis and Visualisation using Metview © ECMWF 2016 4 Metview history Announced at first EGOWS in June 1990 (Oslo) First prototype in 1991 INPE First operational version in 1993 Metview 1.0 OpenGL graphics introduced in 1998
    [Show full text]
  • The NOAA National Operational Model Archive and Distribution
    THE NOAA OPERATIONAL MODEL ARCHIVE AND DISTRIBUTION SYSTEM (NOMADS) G. K. RUTLEDGE† National Climatic Data Center 151 Patton Ave, Asheville, NC 28801, USA E-mail: [email protected] J. ALPERT National Centers for Environmental Prediction Camp Springs, MD 20736, USA E-mail: [email protected] R. J. STOUFFER Geophysical Fluid Dynamics Laboratory Princeton, NJ 08542, USA E-mail: [email protected] B. LAWRENCE British Atmospheric Data Centre Chilton, Didcot, OX11 0QX, U.K. E-mail: [email protected] Abstract An international collaborative project to address a growing need for remote access to real-time and retrospective high volume numerical weather prediction and global climate model data sets (Atmosphere-Ocean General Circulation Models) is described. This paper describes the framework, the goals, benefits, and collaborators of the NOAA Operational Model Archive and Distribution System (NOMADS) pilot project. The National Climatic Data Center (NCDC) initiated NOMADS, along with the National Centers for Environmental Prediction (NCEP) and the Geophysical Fluid Dynamics Laboratory (GFDL). A description of operational and research data access needs is provided as outlined in the U.S. Weather Research Program (USWRP) Implementation Plan for Research in Quantitative Precipitation Forecasting and Data Assimilation to “redeem practical value of research findings and facilitate their transfer into operations.” † Work partially supported by grant NES-444E (ESDIM) of the National Oceanic and Atmospheric Administration (NOAA). 1 2 1.0 Introduction To address a growing need for real-time and retrospective General Circulation Model (GCM) and Numerical Weather Prediction (NWP) data, the National Climatic Data Center (NCDC) along with the National Centers for Environmental Prediction (NCEP) and the Geophysical Fluid Dynamics Laboratory (GFDL) have initiated the collaborative NOAA Operational Model Archive and Distribution System (NOMADS) (Rutledge, et.
    [Show full text]