Title 44pt sentence case Scientific Computing on ARM Part 1 Chris Adeniyi-Jones Affiliations 24pt sentence case Principal Engineer, Software and Large-Scale Systems ARM Research 20pt sentence case Linaro Connect - Budapest Thursday 9th March 2017 © ARM 2017 Title 40pt sentence case Agenda Bullets 24pt sentence case § How this got started … bullets 20pt sentence case § Ecosystem for HPC § Compilers § Libraries § Debuggers § Profilers § Scalable Vector Extension § Final Thoughts § Hands-on Sessions 2 © ARM 2017 Title 40pt sentence case Small beginnings: Mont-Blanc Project - 2011 Develop an energy-efficient HPC prototype Port and optimize HPC Research into technologies using low-power commodity embedded applications on the required for the next-generation Bullets 24pt sentence case technology. prototype HPC system bullets 20pt sentence case Prototype Compute evaluation Acceleration Performance Interconnect Prediction Memory Design Study 3 © ARM 2017 Title 40pt sentence case Mont-Blanc HPC Software Ecosystem Bullets 24pt sentence case bullets 20pt sentence case 4 © ARM 2017 Title 40pt sentence case Why “Commodity”? Bullets 24pt sentence case bullets 20pt sentence case Cost Availability Performance Energy efficiency? Supercomputing with Commodity CPUs: Are Mobile SoCs Ready for HPC? Nikola Rajovic, Paul M. Carpenter, Isaac Gelado, Nikola Puzovic, Alex Ramirez, Mateo Valero 5 © ARM 2017 ARM RESEARCH Title 40pt sentence case Serious ARM HPC deployments starting in 2017 Bullets 24pt sentence case § Two big announcements in 2017 about ARM in HPC in Europe bullets 20pt sentence case 6 © ARM 2017 Title 40pt sentence case Fujitsu HPC CPU Bullets 24pt sentence case bullets 20pt sentence case slides from Fujitsu at ISC’16 7 © ARM 2017 Text 54pt sentence case Ecosystem for HPC 8 © ARM 2017 Title 40pt sentence case Ecosystem for HPC Bullets 24pt sentence case List of software components needed: bullets 20pt sentence case § Linux OS availability § Compilers § Libraries § Debuggers § Profilers § Job schedulers Mix of open source and commercial products and applications… 9 © ARM 2017 Released Title 40pt sentence case Planned ARM HPC ecosystem roadmap Concept AppliedMicro X-Gene 1 & 2 Qualcomm Centriq AppliedMicro X-Gene 3 Hardware AMD Seattle Phytium Mars Fujitsu – Post K (SVE) Bullets 24pt sentence case Cavium ThunderX Cavium ThunderX2 bullets 20pt sentence case OpenHPC 1.2 ARM Optimized Routines Open-Source ARM Optimized Routines – vector versions software Altair PBS Pro GCC (gcc/g++/gfortran) LLVM - clang LLVM – Flang ARM C/C++ Compiler – ahead of LLVM trunk ARM Fortran Compiler ARM HPC ARM Performance Libraries tools ARM Code Advisor (Beta) ARM Code Advisor (Full release) ARM Instruction Emulator Allinea DDT and MAP Rogue Wave TotalView ISV software NAG Library & Compiler ISV software PathScale ENZO 2016 2017 Future 10 © ARM 2017 Title 40pt sentence case Open source in the ARM HPC ecosystem Bullets 24pt sentence case § Over the past 12 months many more packages and applications have been bullets 20pt sentence case successfully ported to ARM HPC 11 © ARM 2017 Title 40pt sentence case Linux / FreeBSD w/ AARCH64 support Bullets 24pt sentence case bullets 20pt sentence case 12.04LTS & 14.04LTS ß Also 14.10 & 15.04 releases Debian 8 adds AARCH64 – April 2015 released Fedora 22 released – May 2015 Red Hat Enterprise Linux Server for ARM CentOS Linux 7 for AArch64 Fedora 23 released – Nov 2015 7.2 BETA – Sept, 2015 GA – August 2015 OpenSUSE 13.2 – Nov 2014 SUSE Launches Partner Program to Bring SUSE Linux Enterprise 12 to 64-bit ARM July 2015 @ ISC ß Engaged with FreeBSD foundation / Semi-half & Cavium to get FreeBSD on ARMv8 FreeBSD Beta version demo’d by Semihalf – Nov. 2015 12 © ARM 2017 Title 40pt sentence case HPC filesystems Bullets 24pt sentence case bullets 20pt sentence case Software Status LUSTRE Ported HDFS Ported CEPH Ported BeeGFS Ported 13 © ARM 2017 Title 40pt sentence case Workload and cluster managers Bullets 24pt sentence case Software Status bullets 20pt sentence case IBM LSF Ported HP CMU Ported SLURM Ported Adaptive Computing (Moab) Future Altair PBS Works Ported OpenLava (LSF port) Ported 14 © ARM 2017 Text 54pt sentence case Compilers 15 © ARM 2017 Title 40pt sentence case Open source and commercial compilers § GCC § PathScale Bullets 24pt sentence case § C, C++, Fortran § C, C++ Fortran bullets 20pt sentence case § OpenMP 4.0 § OpenACC § OpenMP 4.0 § LLVM § NAG § C, C++ § Fortran § OpenMP 3.1, (4.0 coming soon) § OpenMP 3.1 § Fortran coming Q1 2017 § ARM C/C++ Compiler § LLVM based § Includes SVE 16 © ARM 2017 Title 40pt sentence case ARM C/C++ Compiler Text 30pt sentence case Commercially supported C/C++ compiler for Linux user-space HPC applications Bullets 24pt sentence case bullets 20pt sentence case LLVM-based § ARM-on-ARM compiler § For application development (not bare-metal/embedded) Regularly pulls from upstream LLVM, adding: § SVE support in the assembler, disassembler, intrinsics and autovectorizer § Compiler Insights to support ARM Code Advisor OpenMP § Uses latest open source (now ARM-optimized) LLVM OpenMP runtime § Changes pushed back to the community 17 © ARM 2017 Title 40pt sentence case ARM C/C++ Compiler – usage Bullets 24pt sentence case § To c o m p i l e C c o d e : bullets 20pt sentence case % armclang -O3 file.c –o file § To c o m p i l e C + + c o d e : % armclang++ -O3 file.cpp –o file 18 © ARM 2017 Title 40pt sentence case Common armclang options Flag Description Bullets 24pt sentence case bullets 20pt sentence case --help Describes the most common options supported by ARM C/C++ Compiler --vsn Displays version information and license details --version -O<level> Specifies the level of optimization to use when compiling source files. The default is -O0 -c Performs the compilation step, but does not perform the link step. Produces an ELF object .o file. Run armclang again, passing in the object files to link -o <file> Specifies the name of the output file -fopenmp Use OpenMP -S Outputs assembly code, rather than object code. Produces a text .s file containing annotated assembly code 19 © ARM 2017 Title 40pt sentence case LLVM OpenMP development Bullets 24pt sentence case § We have been contributing 5000 bullets 20pt sentence case ARM-related upstream patches 4500 to LLVM’s ‘libomp’ 4000 3500 § Example shown here is the Lulesh 3000 benchmark at size=80 running on 2500 1-96 cores 2000 Zones per second § GCC has slightly better serial 1500 performance 1000 500 § libomp demonstrates superior 0 scaling 0 16 32 48 64 80 96 Number of threads § Even when used with GCC! GCC + GOMP GCC + OMP LLVM + OMP 20 © ARM 2017 Title 40pt sentence case Parallelism to enable optimal HPC performance § OpenMP Bullets 24pt sentence case § We are adding enhancements to the LLVM OpenMP implementation to get better bullets 20pt sentence case AArch64 performance § ARM is active member of the OpenMP Standards Committee § OpenACC § PathScale and PGI are strong supporters of OpenACC § supported for ARM within ENZO § Auto-vectorization § ARM actively works on vectorization in GCC and LLVM, and encourages work with vectorization support in the compiler community. § PathScale’s compiler has vectorization support built in § MPI parallelism “just works” § Better Infiniband driver support is coming from Mellanox 21 © ARM 2017 Text 54pt sentence case Libraries 22 © ARM 2017 Title 40pt sentence case – now on ARM Bullets 24pt sentence case OpenHPC is a community effort to provide a Functional Components include Areas bullets 20pt sentence case common, verified set of open source packages for Base OS RHEL/CentOS 7.1, SLES 12 HPC deployments Administrative Conman, Ganglia, Lmod, LosF, ORCM, Nagios, pdsh, Tools prun ARM’s participation: Provisioning Warewulf § Silver member of OpenHPC Resource Mgmt. SLURM, Munge. Altair PBS Pro* I/O Services Lustre client (community version) § ARM is on the OpenHPC Technical Steering Numerical/Scientifi Boost, GSL, FFTW, Metis, PETSc, Trilinos, Hypre, Committee in order to drive ARM build support c Libraries SuperLU, Mumps I/O Libraries HDF5 (pHDF5), NetCDF (including C++ and Fortran Status: 1.2.0 release out now interfaces), Adios Compiler Families GNU (gcc, g++, gfortran) § All packages built on ARMv8 for CentOS and SUSE MPI Families OpenMPI, MVAPICH2 Development Autotools (autoconf, automake, libtool), Valgrind,R, § ARM-based machines are being used for building Tools SciPy/NumPy and also in the OpenHPC build infrastructure Performance PAPI, Intel IMB, mpiP, pdtoolkit TAU Tools 23 © ARM 2017 Title 40pt sentence case Open source library AArch64 inbuilt tuning work Bullets 24pt sentence case ARM actively working with the community for increased support and performance bullets 20pt sentence case § OpenBLAS § ARMv8 kernels included § BLIS § BLIS developers have close relationship with ARM § BLIS supports various ARM processors by default (e.g. ARM Cortex-A53, Cortex-A57 CPUs) § Also currently conducting ARM big.LITTLE development § ATLAS § Work ongoing with ARM Research team § Cortex-A57/A53 patches went into ATLAS § FFTW § Just works. NEON options built into v3.3.5 24 © ARM 2017 Title 40pt sentence case Commercial library support Bullets 24pt sentence case Product availability for 64-bit ARMv8-A bullets 20pt sentence case § ARM Performance Libraries § See following slides § NAG Library § Largest commercially available collection of numerical and statistical algorithms § > 1800 functions § Scales well, takes advantage of ARM Performance Libraries § Tested on Juno (ARM) and ThunderX (Cavium ) § PathScale