Front cover IBM POWER8 High-Performance Computing Guide IBM Power System S822LC (8335-GTB) Edition Dino Quintero Joseph Apuzzo John Dunham Mauricio Faria de Oliveira Markus Hilger Desnes Augusto Nunes Rosario Wainer dos Santos Moschetta Alexander Pozdneev Redbooks International Technical Support Organization IBM POWER8 High-Performance Computing Guide: IBM Power System S822LC (8335-GTB) Edition May 2017 SG24-8371-00 Note: Before using this information and the product it supports, read the information in “Notices” on page ix. First Edition (May 2017) This edition applies to: IBM Platform LSF Standard 10.1.0.1 IBM XL Fortran v15.1.4 and v15.1.5 compilers IBM XLC/C++ v13.1.2 and v13.1.5 compilers IBM PE Developer Edition version 2.3 Red Hat Enterprise Linux (RHEL) 7.2 and 7.3 in little-endian mode © Copyright International Business Machines Corporation 2017. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . ix Trademarks . .x Preface . xi Authors. xi Now you can become a published author, too! . xiii Comments welcome. xiv Stay connected to IBM Redbooks . xiv Chapter 1. IBM Power System S822LC for HPC server overview . 1 1.1 IBM Power System S822LC for HPC server. 2 1.1.1 IBM POWER8 processor . 3 1.1.2 NVLink . 4 1.2 HPC system hardware components . 5 1.2.1 Login nodes . 6 1.2.2 Management nodes . 6 1.2.3 Compute nodes. 7 1.2.4 Compute racks . 7 1.2.5 High-performance interconnect. 8 1.2.6 Management and operating system . 8 1.2.7 Parallel file system . 10 1.3 HPC system software components . 11 1.3.1 System software . 12 1.3.2 Application development software . 16 1.3.3 Application software . 18 1.4 HPC system solution. 19 1.4.1 Compute nodes. 19 1.4.2 Management node . 19 1.4.3 Login node . 19 1.4.4 Combining the management and the login node . 19 1.4.5 Parallel file system . 20 1.4.6 High-performance interconnect switch . 20 Part 1. Developers guide . 21 Chapter 2. Compilation, execution, and application development. 23 2.1 Compiler options . 24 2.1.1 IBM XL compiler options . 24 2.1.2 GCC compiler options. 27 2.2 Porting applications to IBM Power Systems . 29 2.3 IBM Engineering and Scientific Subroutine Library. 34 2.3.1 ESSL Compilation in Fortran, XL C/C++, and GCC/G++ . 35 2.3.2 ESSL example . 38 2.4 Parallel ESSL . 40 2.4.1 Program development. 41 2.4.2 Using GPUs with Parallel ESSL . 43 2.4.3 Compilation . 47 2.5 Using POWER8 vectorization . 49 2.5.1 AltiVec operations with GNU GCC . 50 © Copyright IBM Corp. 2017. All rights reserved. iii 2.5.2 AltiVec operations with IBM XL. 51 2.6 Development models . 54 2.6.1 OpenMP programs with the IBM Parallel Environment. 54 2.6.2 CUDA C programs with the NVIDIA CUDA Toolkit. 57 2.6.3 OpenACC . 61 2.6.4 IBM XL C/C++ and Fortran offloading. 64 2.6.5 MPI programs with IBM Parallel Environment v2.3. 70 2.6.6 Hybrid MPI and CUDA programs with IBM Parallel Environment. 75 2.6.7 OpenSHMEM programs with the IBM Parallel Environment. 79 2.6.8 Parallel Active Messaging Interface programs . 80 2.6.9 MPI programs with IBM Spectrum MPI. 81 2.6.10 Migrating from IBM PE Runtime Edition to IBM Spectrum MPI . 82 2.6.11 Using Spectrum MPI . 83 Chapter 3. Running parallel software, performance enhancement, and scalability testing . 89 3.1 Controlling the running of multithreaded applications . 90 3.1.1 Running OpenMP applications . 90 3.1.2 Setting and retrieving process affinity at run time . 93 3.1.3 Controlling NUMA policy for processes and shared memory . 93 3.2 Performance enhancements and scalability tests. 94 3.2.1 ESSL execution in multiple CPUs and GPUs . 94 3.2.2 OpenACC execution and scalability . 101 3.2.3 XL Offload execution and scalability . 101 3.3 Using IBM Parallel Environment v2.3 . 104 3.3.1 Running applications. 104 3.3.2 Managing application . 110 3.3.3 Running OpenSHMEM programs . 111 3.4 Using the IBM Spectrum LSF . 112 3.4.1 Submit jobs . 112 3.4.2 Manage jobs . 117 3.5.