E-Infrastructures H2020-EINFRA-2016-2017 EINFRA-11-2016: Support to the next implementation phase of Pan- European High Performance Computing Infrastructure and Services (PRACE) PRACE-5IP PRACE Fifth Implementation Phase Project Grant Agreement Number: EINFRA-730913 D5.4 HPC Infrastructures Workshop #9 Final Version: 1.0 Author(s): Huub Stoffers, SURFsara Date: 26.11.2018 D5.4 HPC Infrastructures Workshop #9 Project and Deliverable Information Sheet PRACE Project Project Ref. №: EINFRA-730913 Project Title: PRACE Fifth Implementation Phase Project Project Web Site: http://www.prace-project.eu Deliverable ID: < D5.4> Deliverable Nature: < Report > Dissemination Level: Contractual Date of Delivery: PU* 30 / 04 / 2019 Actual Date of Delivery: 30 / 11 / 2018 EC Project Officer: Leonardo Flores Añover * - The dissemination levels are indicated as follows: PU – Public, CO – Confidential, only for members of the consortium (including the Commission Services) CL – Classified, as referred to in Commission Decision 2005/444/EC. Document Control Sheet Title: HPC Infrastructures Workshop #9 Document ID: D5.4 Version: <1.0> Status: Final Available at: http://www.prace-project.eu Software Tool: Microsoft Word 2013 File(s): PRACE-5IP-D5.4.docx Written by: Huub Stoffers, SURFsara Authorship Contributors: Javier Bartolomé Rodriguez (BSC), Radek Januszewski (PSNC), Andreas Johansson (NSC, Linköping University), François Robin (CEA), Susanna Salminen (CSC, IT Center for Science Ltd.), Filip Stanek (IT4Innovations, VSB-TUO), Gert Svensson (KTH) Reviewed by: Kadir Diri, UHEM Veronica Teodor, JUELICH Approved by: MB/TB Document Status Sheet Version Date Status Comments 0.1 04/09/2018 Draft Based on latest (09.02.2018) PRACE 5IP template downloaded from BSCW, included several summaries 0.2 26/09/2018 Draft Corrections, still several summaries missing, layout with illustrations and tables, 1st draft of introduction, main findings, and conclusions PRACE-5IP- EINFRA-730913 i 26.11.2018 D5.4 HPC Infrastructures Workshop #9 Version Date Status Comments 0.3 30/10/2018 Draft Processed feedback from original presenters that responded, finalisation of lay-out, addition of last missing summary. 0.4 02/11/2018 Draft Minor corrections, finalisation of main findings and conclusions. Version for WP5 internal review 0.5 12/11/2018 Draft Processed corrections and feedback from Gert Svensson and François Robin, WP5 internal review. Version for PRACE internal (PMO) review 0.6 22/11/2018 Draft Processed corrections and feedback from Veronica Teodor 1.0 26/11/2018 Final Processed corrections and feedback from Kadir Diri. PRACE-5IP- EINFRA-730913 ii 26.11.2018 D5.4 HPC Infrastructures Workshop #9 Document Keywords Keywords: PRACE, HPC, Research Infrastructure, HPC facility, datacentre, Energy efficiency, Cooling, Power management, Power provisioning Disclaimer This deliverable has been prepared by the responsible Work Package of the Project in accordance with the Consortium Agreement and the Grant Agreement n° EINFRA-730913. It solely reflects the opinion of the parties to such agreements on a collective basis in the context of the Project and to the extent foreseen in such agreements. Please note that even though all participants to the Project are members of PRACE AISBL, this deliverable has not been approved by the Council of PRACE AISBL and therefore does not emanate from it nor should it be considered to reflect PRACE AISBL’s individual opinion. Copyright notices 2018 PRACE Consortium Partners. All rights reserved. This document is a project document of the PRACE project. All contents are reserved by default and may not be disclosed to third parties without the written consent of the PRACE partners, except as mandated by the European Commission contract EINFRA-730913 for reviewing and dissemination purposes. All trademarks and other rights on third party products mentioned in this document are acknowledged as own by the respective holders. PRACE-5IP- EINFRA-730913 iii 26.11.2018 D5.4 HPC Infrastructures Workshop #9 Table of Contents Document Control Sheet ................................................................................................................. i Document Status Sheet ................................................................................................................... i Document Keywords .................................................................................................................... iii List of Figures .............................................................................................................................. vii List of Tables ............................................................................................................................... viii References and Applicable Documents .................................................................................... viii List of Acronyms and Abbreviations ........................................................................................... ix List of Project Partner Acronyms .............................................................................................. xii Executive Summary ....................................................................................................................... 1 1 Introduction ............................................................................................................................. 1 2 Programme, content, and speakers ....................................................................................... 2 3 Session I – Site Updates .......................................................................................................... 5 3.1 SuperMUC-NG – LRZ’s next step towards an improved waste-heat recovery – Herbert Huber, LRZ .................................................................................................................. 5 3.1.1 SuperMUC-NG Phase 1 ............................................................................................. 5 3.1.2 High-frequency power usage measurement and energy aware scheduling ............... 6 3.1.3 Quantification of benefits ........................................................................................... 6 3.2 Surprises and challenges with stainless steel welding – Tiziano Belotti, CSCS ......... 7 3.3 The ENI Green Datacentre in its early years – Luca Bortot, ENI .............................. 9 3.3.1 A datacentre designed to be a prototype being in production ................................... 9 3.3.2 Example: The discovery of a fan “going against the intended current” ................. 10 3.3.3 Monitoring and data collection: “never trust a single source” ............................... 11 4 Session II – Standards and Regulations .............................................................................. 12 4.1 Tips, Tricks, and Trends in Liquid Cooling: The State of the Practice – Mike Patterson, Intel .......................................................................................................................... 12 4.1.1 Liquid versus air cooling, a balanced view .............................................................. 12 4.1.2 Intel activity pertaining to liquid cooling ................................................................. 12 4.1.3 ASHRAE Updates ..................................................................................................... 13 4.1.4 Immersion cooling .................................................................................................... 13 4.1.5 System Design Guidance topics ............................................................................... 13 4.2 Open Standards for Datacentre Availability – Lex Coors, Interxion ....................... 13 PRACE-5IP- EINFRA-730913 iv 26.11.2018 D5.4 HPC Infrastructures Workshop #9 4.3 Update from the Energy Efficient HPC Working Group (EE HPC WG) – Natalie Bates, LLNL .............................................................................................................................. 14 5 Session III - Energy to Solution and TCO .......................................................................... 15 5.1 Presentations of the results from the three pilot systems of the PRACE-3IP Pre- Commercial Procurement ........................................................................................................ 15 5.1.1 CINES, prototype “Frioul” – Jerome Chapelle & Eric Boyer ................................ 15 5.1.1.1 PCP goals ......................................................................................................... 15 5.1.1.2 “Frioul” ............................................................................................................ 16 5.1.2 CINECA, prototype “D.A.V.I.D.E.” – Carlo Cavazzoni ......................................... 17 5.1.3 JSC, prototype “JUMAX” – Dirk Pleiter ................................................................. 19 5.2 TCO model for HPC procurement assessment – Eric Boyer, GENCI ..................... 23 6 Session IV – Bologna Science Park ...................................................................................... 24 6.1 Bologna Science Park: an infrastructure for HPC – Gaetano Moron, INFN/CNAF 24 7 Session V – Heat re-use ......................................................................................................... 25 7.1 Making HPC Climate Positive – Eric Rylander, Stockholm Data Parks at Stockholm Exergi ........................................................................................................................................