MPICH: 3.0 and Beyond

Pavan Balaji Computer Scienst Group Lead, Programming Models and Runme systems Argonne Naonal Laboratory MPICH: Goals and Philosophy

§ MPICH connues to aim to be the preferred MPI implementaons on the top machines in the world § Our philosophy is to create an “MPICH Ecosystem”

TAU PETSc MPE

Intel ADLB HPCToolkit MPI IBM Tianhe MPI MPI MPICH DDT Cray Math MPI MVAPICH Works Microso Totalview MPI ANSYS

Pavan Balaji, Argonne Naonal Laboratory MPICH on the Top Machines

§ 7/10 systems use MPICH

1. Titan (Cray XK7) exclusively 2. Sequoia (IBM BG/Q) § #6 One of the top 10 systems 3. K Computer (Fujitsu) uses MPICH together with other 4. Mira (IBM BG/Q) MPI implementaons 5. JUQUEEN (IBM BG/Q) 6. SuperMUC (IBM InfiniBand) § #3 We are working with Fujitsu 7. Stampede (Dell InfiniBand) and U. Tokyo to help them 8. Tianhe-1A (NUDT Proprietary) support MPICH 3.0 on the K Computer (and its successor) 9. Fermi (IBM BG/Q) § #10 IBM has been working with 10. DARPA Trial Subset (IBM PERCS) us to get the PERCS plaorm to use MPICH (the system was just a lile too early)

Pavan Balaji, Argonne Naonal Laboratory MPICH-3.0 (and MPI-3)

§ MPICH-3.0 is the new MPICH2 :-) – Released mpich-3.0rc1 this morning! – Primary focus of this release is to support MPI-3 – Other features are also included (such as support for nave atomics with ARM-v7) § A large number of MPI-3 features included – Non-blocking collecves – Improved MPI one-sided communicaon (RMA) – New Tools Interface – Shared memory communicaon support – (please see the MPI-3 BoF on Thursday for more details)

Pavan Balaji, Argonne Naonal Laboratory MPICH Past Releases

1.2.x series 1.4.x series MPI 2.2 support Improved FT v1.0.8 v1.2 Hydra introduced v1.4 ARMCI support

v1.2.1 v1.4.1

May ‘08 Nov ’08 May ‘09 Nov ‘09 May ‘10 Nov ‘10 May ‘11 Nov ‘11 May ‘12 Nov ‘12

v1.1.1 v1.3.1 v1.3.2

v1.1 v1.3 v1.5 1.1.x series 1.3.x series 1.5.x series MPI 2.1 support Hydra default Blue Gene/Q Blue Gene/P Asynchronous progress Intel MIC Nemesis Checkpoinng Revamped Build System Hierarchical Collecves FTB Some MPI-3 features

Pavan Balaji, Argonne Naonal Laboratory MPICH 3.0 and Future Plans 3.2.x series Memory FT 3.0.x series Dynamic Execuon Full MPI-3 support Environments Support for ARM-v7 MPI-3.1 support (?) v3.0 nave atomics v3.2 (preview)

v1.3.1 v1.3.2 v1.3.3

Nov ‘12 May ’12 Nov ‘13 May ‘13 Nov ‘14 May ‘14

v3.1 (preview) 3.1.x series Full Process FT Support for GPUs Acve Messages Topology Funconality Support for Portals-4

Pavan Balaji, Argonne Naonal Laboratory MPICH Fault Tolerance

§ Fault Query Model – Errors propagated upstream – Global objects remain valid on faults – Ability to create new communicators/ groups and connue execuon § Fault Propagaon Model – Asynchronous Noficaon of faults to

processes (global noficaon as well as Node 0 Node 1 P1 “subscribe” model) P0 MPI P2 MPI Library MPI § Memory Fault Tolerance Library Library – Trapping memory errors and nofying Hydra Hydra proxy SIGUSR1 SIGUSR proxy applicaon (e.g., global memory) 1 FP SIGCHLD FP List List P2 P2 mpiexec

Pavan Balaji, Argonne Naonal Laboratory MPICH with GPUs

Main Main Rank = 1 Memory Rank = 0 Memory

GPU GPU Memory Memory Network CPU CPU

NVRAM NVRAM

Unreliable Unreliable Memory Memory

§ Treang GPUs as first-class cizens § MPI currently allows data to be moved from main memory to main memory § We want to extend this to allow data movement from/to any memory

Pavan Balaji, Argonne Naonal Laboratory Dynamic Execution Environments with MPICH

§ Ability to dynamically create and manage tasks § Tasks as fine-grained threads § Support within MPICH to efficiently support such models § Support above MPICH for cleaner integraon of dynamic execuon environments (Charm++, FG-MPI)

Pavan Balaji, Argonne Naonal Laboratory Support for High-level Libraries/Languages

§ A large effort to provide improved support for high-level libraries and languages using MPI-3 features § We are currently focusing on three high-level libraries/ languages – CoArray Fortran (CAF-2.0) in collaboraon with John Mellor-Crummey – Chapel in collaboraon with Brad Chamberlain – Charm++ in collaboraon with Laxmikant Kale § Other high-level libraries have expressed interest as well – X10 – OpenSHMEM

Pavan Balaji, Argonne Naonal Laboratory Other Features Planned for MPICH

§ Acve Messages § Topology Funconality § Structural Changes § Support for Portals 4 § Support for PMI-2

Pavan Balaji, Argonne Naonal Laboratory Thank You!

Web: hp://www.mpich.org More informaon on MPICH: hp://www.lmgy.com/?q=mpich