Technical Memo 857 ical Memo Technical Memo Technical Memo Technical Memo Techn nical MemoThe ECMWF Technical ScalabilityMemo Technical Memo Technical Memo Tec hnicalProgramme: Memo Technical Progress Memo Technical Memo Technical Memo Te Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Technical Memo Technical Memo Technicaland Memo Plans Technical Memo Technical Memo Technical Memo Technical Memo Technical Technical Memo Technical Memo TechnicalPeter Bauer, Memo Tiago Quintino, Technical Nils Wedi, Memo Technical Memo Technical Memo Technical Memo Technical Antonino Bonanni, Marcin Chrust, Willem Deconinck, Technical Memo Technical Memo TechnicalMichail Diamantakis, Memo Peter Technical Düben, Stephen Memo English, Technical Memo Technical Memo Technical Memo Technical Johannes Flemming, Paddy Gillies, Ioan Hadade, Technical Memo Technical Memo TechnicalJames Hawkes, MikeMemo Hawkins, Technical Olivier Iffrig, Memo Technical Memo Technical Memo Technical Memo Technical Technical Memo Technical MemoChristian Technical Kühnlein, MichaelMemo Lange, Technical Peter Lean, Memo Technical Memo Technical Memo Technical Memo Technical Pedro Maciel, Olivier Marsden, Andreas Müller, Technical Memo Technical MemoSami TechnicalSaarinen, Domokos Memo Sarmany, Technical Michael Sleigh, Memo Technical Memo Technical Memo Technical Memo Technical Simon Smart, Piotr Smolarkiewicz, Daniel Thiemert, Technical Memo Technical MemoGiovanni Technical Tumolo, Christian Memo Weihrauch, Technical Cristiano Zanna Memo Technical Memo Technical Memo Technical Memo Technical Technical Memo Technical MemoFebruary 2020Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Memo Technical Series: ECMWF Technical Memoranda A full list of ECMWF Publications can be found on our website under: http://www.ecmwf.int/en/publications Contact: [email protected] © Copyright 2020 European Centre for Medium-Range Weather Forecasts, Shinfield Park, Reading, RG2 9AX, UK Literary and scientific copyrights belong to ECMWF and are reserved in all countries. This publication is not to be reprinted or translated in whole or in part without the written permission of the Director- General. Appropriate non-commercial use will normally be granted under the condition that reference is made to ECMWF. The information within this publication is given in good faith and considered to be true, but ECMWF accepts no liability for error or omission or for loss or damage arising from its use. ECMWF Scalability Programme Abstract The efficiency of the forecasting system on future high-performance computing and data handling systems is considered one of the key challenges for implementing ECMWF’s ambitious strategy. This was already recognised by ECMWF in 2013, and has led to the foundation of the Scalability Programme. The programme aims to address this challenge as a concerted action between the Centre and its Member States, but also draws in the computational science expertise available throughout Europe. This technical memorandum provides an overview of the status of the programme, highlights achieve- ments from the first five years ranging from observational data pre-processing, data assimilation, forecast model design and output data post-processing, and defines the roadmap for the next five years towards a sustainable system that can operate on the expected range of hardware and software technologies. This point in time is crucial because the programme will have a strong focus on implementation and operational benefit in the next period. Research Report No. 857 1 ECMWF Scalability Programme Executive Summary This paper describes ECMWF’s Scalability Programme that was founded in 2013. The programme de- fines and implements the software infrastructure needed to enable the Centre’s forecast production work- flow to operate on future high-performance computing and data handling technologies that will be more heterogeneous and require different software methodologies. Only through a concerted software devel- opment effort will this technology transfer be possible and allow ECMWF to implement its ambitious strategy. The paper describes both 2014-2019 developments and the 2020-2024 roadmap in detail so that ECMWF and its Member States have available a basis for future coordination and resource planning. While the first phase of the programme already produced operational benefits, it has mostly been a research phase. The key achievements of this first phase are as follows: 1. ECMWF has tested the limits of the existing prediction system on the largest super-computing fa- cilities in the world, and explored the most promising remaining performance optimisation options that can be realised within the existing code infrastructure. For example, mixed precision arith- metics, concurrent execution of model components, overlapping computation and communication, and the use of more efficient CPU1-type processors and interconnects are expected to provide code speed-ups of the order of 3 by 2021. No significant further enhancements will be available unless CPU technology improves significantly. 2. ECMWF has established the novel concept of weather and climate dwarfs and benchmarks in the community. This concept allows, (1) to drive performance optimisations targeted at the entire range of new processor technologies in the most efficient way and, (2) to produce realistic estimates of sustained performance of the full system on large-scale, future super-computers. Already now, throughput speed-ups by a factor of more than 20 have been achieved for the most costly model components on single GPUs2, and a factor of 24 on over 11,500 GPUs, when compared to CPUs. The first ever implementation of a forecast model component on an FPGA3 resulted in a factor of 2.5 improvement of time to solution and at least a factor of 10 in energy to solution compared to CPUs. 3. ECMWF has built its entire performance enhancement strategy on co-design, namely the co-design of numerical methods and algorithms with code implementation, so that, (1) scientific and compu- tational performance can be traded off against one another if needed, and (2) extensions to coupled modelling and assimilation and atmospheric composition will be future proof. This was possible by creating the flexible OOPS4 data assimilation system, two dynamical cores operating on the same grid, and the generic Atlas data structure framework, all of which have reached a very high level of maturity. 4. ECMWF has undertaken the modernisation of various data handling components of its forecasting system. These include (1) the extension of the IFS5 I/O6 server to the wave model and ensemble, (2) the development of a new object-store for model output, supporting novel I/O architectures, and (3) research into novel methods of post-processing leading to critical insight into the success- ful development of the new product generation software. All these developments have already 1Central Processing Unit 2Graphics Processing Unit 3Field Programmable Gate Array 4Object Oriented Prediction System 5Integrated Forecasting System 6Input/Output 2 Research Report No. 857 ECMWF Scalability Programme been successfully deployed into time-critical operations, the combination of which has helped to achieve a 5-fold speed-up in product generation, whilst also increasing system robustness. The new product generation software has already been tested with a novel, non-volatile memory extension technology providing a factor 10 speed-up. 5. ECMWF has developed the Kronos workflow and benchmark generator, that has been used already in the current HPC7 procurement providing, for the first time, a full workflow benchmark including model output and product generation, thus being much closer to the reality of our operational system. 6. ECMWF has founded a European collaboration network and established itself as the leading centre performing cutting-edge research at the interface between computational and weather and climate science. This has been achieved through leadership of and participation in several European re- search projects, providing key contributions to European