- Spring 2018 · Vol. 16 · No. 1 Innovatives Supercomputing in Deutschland Understanding How the Universe Formed IllustisTNG provides largest-ever galaxy formation simulation (p. 35 and p. 59) New HLRS Sociopolitical Advisory Board GCS Architecture Transition Multidisciplinary group to counsel on JSC and LRZ announce new machines how HPC can address a broader range (p. 8 and p. 10) of society’s challenges (p. 15) Imprint Publishers Prof. Dr. D. Kranzlmüller Prof. Dr. Dr. Th. Lippert Prof. Dr.-Ing. Dr. h.c. Dr. h.c. Director, Leibniz Supercomputing Director, Jülich Supercomputing Hon.-Prof. M. M. Resch Centre Centre Director, High-Performance Computing Center Stuttgart © HLRS 2018 InSiDE is published two times a year by the Gauss Center for Supercomputing (HLRS, JSC, LRZ) Editor-in-Chief Michael Resch, HLRS [email protected] Editor Eric Gedenk, GCS, HLRS [email protected] [email protected] Production Manager F. Rainer Klank, HLRS [email protected] Design Zimmermann Visuelle Kommunikation www.zimmermann-online.info Printing DCC Document Competence Center Siegmar Kästl e.K., Ostfildern If you would like to receive InSiDE regu- larly, please send an e-mail with your postal address to F. Rainer Klank: [email protected] Visit InSiDE online at: http://inside.hlrs.de Editorial Welcome to this new issue of InSiDE, the publi- and longest multiphysics earthquake simulation cation about innovative supercomputing in Ger- ever performed. The team used SuperMUC to re- many published by the Gauss Centre for Super- create the devastating 2004 Sumatra-Andaman computing (GCS). Over the last year, GCS has earthquake. A research consortium led by cosmo- been taking the next steps in hardware upgrades. logists from the Heidelberg Institute of Theoretical Jülich Supercomputing Centre will put new sys- Studies used HLRS to break another simulation tems in operation by mid-2018. The next genera- record—their own. The team’s IllustrisTNG simula- tion of LRZ’s SuperMUC system will replace the tion is the largest simulation of galaxy formation, current system in autumn this year. With these surpassing the scope and detail they achieved in two installations, Germany is strengthening its their record-breaking 2015 Illustris simulation. In position as Europe’s HPC leader. 2019 will see addition to massive simulation among the stars, a further step toward realizing Germany’s HPC University of Cologne researchers used JSC’s strategy when HLRS will start to install its new JUQUEEN to study extreme inner-Earth pro- system to replace Hazel Hen. cesses. The team combined theory and experi- ment to gain greater understanding of the material These new systems serve as the first phase of and geological processes happening deep below the German smart scale strategy, announced last the surface of the Earth. year at ISC 2017. The project SiVeGCS, which is supported by the federal government as well as GCS also has a clear focus on its own research the state governments of Baden-Württemberg, as well as training activities. This is reflected in Bavaria, and North Rhein-Westphalia, provides a number of reports on European and German the necessary funds for German leading-edge projects. GCS is grateful for the funding received high-performance computing for the next 8 years. by the European Commission, the Federal Minis- Funding will not only cover the initial investments try of Science (BMBF) and the German Research and operational costs of these machines, but will Foundation (DFG). It is only by providing such also cover funding for additional scientists and a robust framework that HPC can have a true researchers to get the absolute best from these impact on national research and industry. This systems. issue also presents some of GCS’ training activi- ties that are unique in Europe. About 2,000 sci- The German approach for HPC is entirely appli- entists and industry experts go through our train- cation driven. It is the users that determine which ing programs every year, making this the largest way we are headed. To that end, this issue pres- HPC training program in Europe. ents a variety of applications that were run on GCS systems. Researchers from the Technical Univer- Prof. Dieter Kranzlmüller sity of Munich won the best paper award at last Prof. Thomas Lippert year’s SC17 conference for presenting the largest Prof. Michael Resch 1 Editorial 24 Navigating A Post-Moore’s-Law World 26 Fenix Consortium to build a Federated News Data and Computing Infrastructure for the Human Brain Project and other Events communities 29 GPU Hackathon Dresden 2018 6 End of an Era at Jülich Supercomput- ing Centre as JUQEEN End-of-Life 31 The Promise and Perils of Simulation approaches in the Social Sciences 8 Jülich Supercomputing Centre 33 HLRS Strengthens Collaborations starts deployment of JUWELS in Asia 10 SuperMUC-NG: 35 Cosmologists Create Largest In the Service of Science Simulation of Galaxy Formation, Break Their Own Record 13 JURECA Booster starts Operation 38 Researchers Create Largest, Longest 15 Sociopolitical Advisory Board to Help Multiphysics Earthquake Simulation to Guide HLRS Vision Date 17 Human Brain Project Successfully 42 Researchers Compute Their Way Enters the Next Phase to the Centre of the Earth 20 Leibniz Supercomputing Centre Named as A Bavarian Big Data Competence Centre Applications 22 Designing for Energy Efficiency 48 Optimization and HPC-Applications of in HPC Centers the Flow Solver FLOWer inSiDE | Spring 2018 54 Automated Code Generation for 98 SMITH: Smart Medical Information Maximizing Performance of Detailed Technology for Healthcare Chemistry Calculations in OpenFOAM 101 Mikelangelo Bringing HPC into 59 Simulating the Emergence the Cloud of Cosmic Structures 103 Integrated Services for the European 66 Data Assimilation with the Integrated Open Science Cloud Terrestrial Systems Modeling Plat- form, TerrSysMP-PDAF 104 MYX: MUST Correctness Checking for YML and XMP Programs 72 Rapidly Rotating Convection in Geophysical Systems 107 MoeWE Project Opens Supercomputing-Akademie 76 Cavitation Erosion in Injection Systems 80 Secondary Circulations at an Isolated Semi-Arid Forest Centers Systems Projects Trainings 86 ORPHEUS–Fire Safety in the Under- 110 Leibniz Supercomputing Centre (LRZ) ground 112 Höchstleistungsrechenzentrum 89 EU H2020 CoE Performance Stuttgart (HLRS) Optimization and Productivity (POP) Successfully Finished 114 Jülich Supercomputing Centre (JSC) 95 TELEMAC - A hydrodynamic solver 116 Timetable, High Performance Com- for HPC? puting, Courses and Tutorials 117 Visit InSiDE Online for Details inSiDE | Spring 2018 Vol. 16 No. 1 News /Events News /Events In this section you will find general activities of GCS in the fields of HPC and education. End of an Era at Jülich Supercomputing Centre as JUQEEN End-of-Life approaches In spring 2018, the Blue Gene/Q system The successor system, JUGENE [1], a se- JUQUEEN at Jülich Supercomputing Centre cond-generation Blue Gene/P system, was (JSC) is being decommissioned and disman- installed in 2007 with 16 racks, 65,536 com- tled. This event marks the end of a 13-year era pute cores and a peak performance of 222.8 at JSC in which the leadership-class system in TFLOP/s. In November 2007, JUGENE entered Jülich was based on IBM’s Blue Gene techno- the Top500 list on the second spot and was logy. The Blue Gene architectures were defined the fastest open-science supercomputer by a unique combination of energy-efficient worldwide. The system was formally inaugu- embedded processor technology, a well-ba- rated in February 2008. To accommodate the lanced node and network design, advanced increasing demand from national and Euro- packaging for industry-leading density, and pean users, JUGENE was later upgraded in an uncompromised focus on the highest sca- 2009 to 72 racks and 294,912 compute cores, lability throughout the hardware and software making it the third-fastest system worldwide. layers. With a peak performance of more than one PFLOP/s, JUGENE was the first European JSC operated and, in cooperation with vendor petaflop system. IBM, shaped all three generations of the Blue Gene product line for the benefit of its user In 2012 and 2013, the current Blue Gene/Q communities. system JUQUEEN [2] was deployed in two phases. JUQUEEN now consists of 28 racks In 2005, the first Blue Gene/L system in Ger- with 458,752 compute cores, 1.8 million hard- many was installed in Jülich. The system, ware threads and a peak performance of 5.9 named JUBL (Jülich Blue Gene/L) consisted PFLOP/s. In November 2012, JUQEEN was the of one rack with 1,024 compute nodes and 5th fastest supercomputer in the world and provided a peak performance of 5.7 TFLOP/s. the fastest European supercomputer. Together JUBL immediately attracted a large number of with its user base, JSC tackled the challenge of users and, despite its specialized architecture, exploiting the massive parallelism of this archi- showed promise as an enabler of breakthrough tecture. The continuous growth of the High-Q science in a large number of different science Club [3], a list of applications capable of effi- disciplines. Consequently, already shortly after ciently using the whole JUQUEEN system, its commissioning date, JUBL was extended to showed the success of these efforts. Currently an 8-rack system with 16,348 CPUs and a peak 31 applications from numerous domains have 6 performance of 45.6 TFLOP/s. In June 2006, reached High-Q Club status. JUBL was the fastest system in Germany and eighth-fastest system worldwide according to The road of Blue Gene systems’ architectural the Top500. evolution in Jülich is paved by numerous inSiDE | Spring 2018 high-grade scientific discoveries and pub- lications, many of which were widely reco- gnized for their breakthrough character. The list of major research fields using the systems included elementary particle physics, engi- neering, earth systems modeling, chemistry, condensed matter, plasma physics, and astro- physics.