Latest Advances in MVAPICH2 MPI Library for NVIDIA GPU Clusters with InfiniBand
Presented at GTC ’15 Presented by
Dhabaleswar K. (DK) Panda Khaled Hamidouche The Ohio State University The Ohio State University E-mail: [email protected] E-mail: [email protected] h p://www.cse.ohio-state.edu/~panda h p://www.cse.ohio-state.edu/~hamidouc High-End Compu ng (HEC): Petaflop to Petaflop GTC ’15
100-150 PFlops in 2016-17
1 EFlops in 2020-2024?
3 Trends for Commodity Compu ng Clusters in the Top 500 List (h p://www.top500.org) GTC ’15
500 100 Percentage of Clusters 86% 400 Number of Clusters 80 300 60 200 40 100 20
Number of ClustersNumber of 0 0 PercentageClusters of
Timeline
4 Drivers of Modern HPC Cluster Architectures GTC ’15
Mul -core High Performance Interconnects - InfiniBand Accelerators / Coprocessors Processors <1usec latency, >100Gbps Bandwidth high compute density, high performance/wa >1 TFlop DP on a chip Mul -core processors are ubiquitous InfiniBand very popular in HPC clusters
Accelerators/Coprocessors becoming common in high-end systems Pushing the envelope for Exascale compu ng
5 Tianhe – 2 Titan Stampede Tianhe – 1A Large-scale InfiniBand Installa ons GTC ’15
225 IB Clusters (45%) in the November 2014 Top500 list (h p://www.top500.org) Installa ons in the Top 50 (21 systems):
519,640 cores (Stampede) at TACC (7th) 73,584 cores (Spirit) at USA/Air Force (31st) 72,800 cores Cray CS-Storm in US (10th) 77,184 cores (Curie thin nodes) at France/CEA (33rd) 160,768 cores (Pleiades) at NASA/Ames (11th) 65,320-cores, iDataPlex DX360M4 at Germany/Max-Planck (34th) 72,000 cores (HPC2) in Italy (12th) 120, 640 cores (Nebulae) at China/NSCS (35th) 147,456 cores (Super MUC) in Germany (14th) 72,288 cores (Yellowstone) at NCAR (36th) 76,032 cores (Tsubame 2.5) at Japan/GSIC (15th) 70,560 cores (Helios) at Japan/IFERC (38th) 194,616 cores (Cascade) at PNNL (18th) 38,880 cores (Cartesisu) at Netherlands/SURFsara (45th) 110,400 cores (Pangea) at France/Total (20th) 23,040 cores (QB-2) at LONI/US (46th) 37,120 cores T-Pla orm A-Class at Russia/MSU (22nd) 138,368 cores (Tera-100) at France/CEA (47th)
50,544 cores Occigen at France/GENCI-CINES (26th) and many more! 6 Parallel Programming Models Overview GTC ’15
P1 P2 P3 P1 P2 P3 P1 P2 P3
Logical shared memory Shared Memory Memory Memory Memory Memory Memory Memory
Shared Memory Model Distributed Memory Model Partitioned Global Address Space (PGAS) SHMEM, DSM MPI (Message Passing Interface) Global Arrays, UPC, Chapel, X10, CAF, … Programming models provide abstract machine models Models can be mapped on different types of systems e.g. Distributed Shared Memory (DSM), MPI within a node, etc.
7 MVAPICH2 So ware GTC ’15
• High Performance open-source MPI Library for InfiniBand, 10Gig/iWARP and RDMA over Converged Enhanced Ethernet (RoCE) - MVAPICH (MPI-1) ,MVAPICH2 (MPI-2.2 and MPI-3.0), Available since 2002 - MVAPICH2-X (MPI + PGAS), Available since 2012 - Support for NVIDIA GPUs, Available since 2011 - Used by more than 2,350 organiza ons (HPC Centers, Industry and Universi es) in 75 countries - More than 241,000 downloads from OSU site directly - Empowering many TOP500 clusters (Nov ‘14 ranking) • 7th ranked 519,640-core cluster (Stampede) at TACC •11th ranked 160,768-core cluster (Pleiades) at NASA •15th ranked 76,032-core cluster (Tsubame 2.5) at Tokyo Ins tute of Technology and many others - Available with so ware stacks of many IB, HSE and server vendors including Linux Distros (RedHat and SuSE) - h p://mvapich.cse.ohio-state.edu
8 Outline GTC ’15
• Communica on on InfiniBand Clusters with GPUs • MVAPICH2-GPU: CUDA-Aware MPI • MVAPICH2-GPU with GPUDirect-RDMA (MVAPICH2-GDR) • Two-sided Communica on • One-sided Communica on • MPI Datatype Processing • Op miza ons (mul -GPU, device selec on, topology detec on) • MPI and OpenACC • Conclusions
9 Op mizing MPI Data Movement on GPU Clusters GTC ’15
• Connected as PCIe devices – Flexibility but Complexity Memory buffers
Node 0 Node 1 1. Intra-GPU QPI 2. Intra-Socket GPU-GPU CPU CPU CPU 3. Inter-Socket GPU-GPU 4. Inter-Node GPU-GPU 5. Intra-Socket GPU-Host
PCIe 6. Inter-Socket GPU-Host 7. Inter-Node GPU-Host 8. Inter-Node GPU-GPU with IB IB GPU GPU GPU GPU adapter on remote socket
and more . . .
• For each path different schemes: Shared_mem, IPC, GPUDirect RDMA, pipeline … • Cri cal for run mes to op mize data movement while hiding the complexity 10 InfiniBand + GPU Systems (Past) GTC ’15 • Many systems today want to use systems that have both GPUs and high-speed networks such as InfiniBand • Problem: Lack of a common memory registra on mechanism - Each device has to pin the host memory it will use - Many opera ng systems do not allow mul ple devices to register the same memory pages • Previous solu on: - Use different buffers for each device and copy data
11 Sample Code - Without MPI integra on GTC ’15
Naïve implementation with standard MPI and CUDA
At Sender: CPU cudaMemcpy(sbuf, sdev, . . .); MPI_Send(sbuf, size, . . .); PCIe At Receiver: MPI_Recv(rbuf, size, . . .); GPU NIC cudaMemcpy(rdev, rbuf, . . .);
Switch
High Productivity and Poor Performance 12 Sample Code – User Op mized Code GTC ’15
Pipelining at user level with non-blocking MPI and CUDA interfaces Code at Sender side (and repeated at Receiver side)
At Sender: CPU for (j = 0; j < pipeline_len; j++) cudaMemcpyAsync(sbuf + j * blk, sdev + j * blksz,. . .); for (j = 0; j < pipeline_len; j++) { PCIe while (result != cudaSucess) { result = cudaStreamQuery(…); if(j > 0) MPI_Test(…); GPU NIC } MPI_Isend(sbuf + j * block_sz, blksz . . .); } MPI_Waitall(); Switch
User-level copying may not match with internal MPI design High Performance and Poor Productivity 13 Can this be done within MPI Library? GTC ’15
• Support GPU to GPU communica on through standard MPI interfaces - e.g. enable MPI_Send, MPI_Recv from/to GPU memory • Provide high performance without exposing low level details to the programmer - Pipelined data transfer which automa cally provides op miza ons inside MPI library without user tuning • A new Design incorporated in MVAPICH2 to support this func onality
14 Outline GTC ’15
• Communica on on InfiniBand Clusters with GPUs • MVAPICH2-GPU: CUDA-Aware MPI • MVAPICH2-GPU with GPUDirect-RDMA (MVAPICH2-GDR) • Two-sided Communica on • One-sided Communica on • MPI Datatype Processing • Op miza ons (mul -GPU, device selec on, topology detec on) MPI and OpenACC • Conclusions
15 GPU-Direct GTC ’15
• Collabora on between Mellanox and NVIDIA to converge on one memory registra on technique • Both devices register a common host buffer • GPU copies data to this buffer, and the network adapter can directly read from this buffer (or vice-versa)
• Note that GPU-Direct does not allow you to bypass host memory 16 Sample Code – MVAPICH2-GPU (CUDA-Aware MPI) GTC ’15
• MVAPICH2-GPU: standard MPI interfaces used • Takes advantage of Unified Virtual Addressing (>= CUDA 4.0) • Overlaps data movement from GPU • with RDMA transfers
At Sender: inside MVAPICH2 MPI_Send(s_device, size, …);
At Receiver: MPI_Recv(r_device, size, …);
• High Performance and High Productivity 17
OSU-GTC 2015 Outline GTC ’15
• Communica on on InfiniBand Clusters with GPUs • MVAPICH2-GPU: CUDA-Aware MPI • MVAPICH2-GPU with GPUDirect-RDMA (MVAPICH2-GDR) • Two-sided Communica on • One-sided Communica on • MPI Datatype Processing • Op miza ons (mul -GPU, device selec on, topology detec on) • MPI and OpenACC • Conclusions
18 GPU-Direct RDMA (GDR) with CUDA GTC ’15
• Network adapter can directly read/ System write data from/to GPU device Memory memory CPU • Avoids copies through the host • Fastest possible communication between GPU and IB HCA Chip • Allows for better asynchronous set GPU communication InfiniBand • OFED with GDR support is under development by Mellanox and GPU NVIDIA Memor y
19 GPU-Direct RDMA (GDR) with CUDA GTC ’15
• MVAPICH2 design extended to use GPUDirect RDMA • Hybrid design using GPU-Direct RDMA - GPUDirect RDMA and Host-based pipelining - Alleviates P2P bandwidth bo lenecks on SandyBridge and IvyBridge System CPU Memory - Support for communica on using mul -rail - Support for Mellanox Connect-IB and ConnectX VPI adapters - Support for RoCE with Mellanox ConnectX VPI adapters GPU IB Chipset Adapter
GPU Memory
SNB E5-2670 IVB E5-2680V2 P2P write: 5.2 GB/s P2P write: 6.4 GB/s P2P read: < 1.0 GB/s P2P read: 3.5 GB/s 20 Using MVAPICH2-GPUDirect Version GTC ’15
• MVAPICH2-2.1a with GDR support can be downloaded from h ps:// mvapich.cse.ohio-state.edu/download/#mv2gdr/ • System so ware requirements - Mellanox OFED 2.1 or later - NVIDIA Driver 331.20 or later - NVIDIA CUDA Toolkit 6.0 or later - Plugin for GPUDirect RDMA - h p://www.mellanox.com/page/products_dyn?product_family=116 - Strongly Recommended : use the new GDRCOPY module from NVIDIA • h ps://github.com/NVIDIA/gdrcopy • Has op mized designs for point-to-point communica on using GDR • Contact MVAPICH help list with any ques ons related to the package - [email protected]
21 Outline GTC ’15
• Communica on on InfiniBand Clusters with GPUs • MVAPICH2-GPU: CUDA-Aware MPI • MVAPICH2-GPU with GPUDirect-RDMA (MVAPICH2-GDR) • Two-sided Communica on • One-sided Communica on • MPI Datatype Processing • More Op miza ons • MPI and OpenACC • Conclusions
22 Enhanced MPI Design with GPUDirect RDMA GTC ’15
• Current MPI design using GPUDirect RDMA uses Sender Receiver Rendezvous protocol rndz_start • Has higher latency for small messages rndz_reply
• Can eager protocol be supported to improve data performance for small messages? fin • Two schemes proposed and used • Loopback (using network adapter to copy data) Rendezvous Protocol • Fastcopy/GDRCOPY (using CPU to copy data) Sender Receiver
send
R. Shi, S. Potluri, K. Hamidouche M. Li, J. Perkins D. Rosse and D. K. Panda, Designing Efficient Small Message Transfer Mechanism for Inter-node MPI Communica on on InfiniBand GPU Clusters IEEE Interna onal Conference on High Performance Compu ng (HiPC'2014). Eager Protocol 23 Performance of MVAPICH2 with GPU-Direct-RDMA: Latency GTC ’15 GPU-GPU Internode MPI Latency
Small Message Latency 30 MV2-GDR2.1a 25 MV2-GDR2.0b MV2 w/o GDR 20
15 Intel Ivy Bridge (E5-2680 8.6X v2) node with 20 cores 10 NVIDIA Tesla K40c GPU, Latency(us) 2.5X Mellanox Connect-IB Dual- 5 FDR HCA CUDA 6.5, Mellanox OFED 2.1 with GPU-Direct-RDMA 0 2.37 usec 0 2 8 32 128 512 2K
Message Size (bytes) 24 Performance of MVAPICH2 with GPU-Direct-RDMA: Bandwidth GTC ’15 GPU-GPU Internode MPI Uni-Directional Bandwidth
Small Message Bandwidth 3000 MV2-GDR2.1a 2500 MV2-GDR2.0b 10x MV2 w/o GDR 2000 2.2X 1500 Intel Ivy Bridge (E5-2680 s) v2) node with 20 cores NVIDIA Tesla K40c GPU, 1000 Mellanox Connect-IB Dual- FDR HCA Bandwidth (MB/ Bandwidth 500 CUDA 6.5, Mellanox OFED 2.1 with GPU-Direct-RDMA
0 1 4 16 64 256 1K 4K
Message Size (bytes) 25
OSU-GTC 2015 Performance of MVAPICH2 with GPU-Direct-RDMA: Bi- GTC ’15 Bandwidth GPU-GPU Internode MPI Bi-directional Bandwidth
Small Message Bi-Bandwidth 4000 MV2-GDR2.1a 3500 MV2-GDR2.0b 3000 MV2 w/o GDR 2x 2500
2000 Intel Ivy Bridge (E5-2680 v2) node with 20 cores 1500 NVIDIA Tesla K40c GPU, Mellanox Connect-IB Dual- FDR HCA Bi-Bandwidth (MB/s) Bi-Bandwidth 1000 10x CUDA 6.5, Mellanox OFED 500 2.1 with GPU-Direct-RDMA
0 1 4 16 64 256 1K 4K
Message Size (bytes) 26 Applica on-Level Evalua on (HOOMD-blue) GTC ’15
Weak Scalability with 2K Particles/ Strong Scalability with 256K GPU Particles 2000 4000
1500 3000
1000 2000
Average TPS Average 1000 TPS Average 500
0 0 4 8 16 32 64 4 8 16 32 64 Number of GPU Nodes Number of GPU Nodes
• Platform: Wilkes (Intel Ivy Bridge + NVIDIA Tesla K20c + Mellanox Connect-IB)
• GDRCOPY enabled: MV2_USE_CUDA=1 MV2_IBA_HCA=mlx5_0 MV2_IBA_EAGER_THRESHOLD=32768 MV2_VBUF_TOTAL_SIZE=32768 MV2_USE_GPUDIRECT_LOOPBACK_LIMIT=32768 MV2_USE_GPUDIRECT_GDRCOPY=1 MV2_USE_GPUDIRECT_GDRCOPY_LIMIT=16384 27 Applica on Level Evalua on GTC ’15
LBM-CUDA AWP-ODC 150 9.4% 90 MPI MPI-GPU 80 MPI MPI-GPU 70 7.9% 11.1% 100 60 11.8% 50 12.0% 40 50 30 13.7% 20 Step Time (S) (S) Time Step 10 0 0 256*256*256 512*256*256 512*512*256 512*512*512 1 GPU/Proc per Node 2 GPUs/Procs per Node Total Execution Time (s) Time Execution Total Domain Size X*Y*Z Configuration • LBM-CUDA (Courtesy: Carlos Rosale, TACC) • Lattice Boltzmann Method for multiphase flows with large density ratios • One process/GPU per node, 16 nodes • AWP-ODC (Courtesy: Yifeng Cui, SDSC) • Seismic modeling code, Gordon Bell Prize finalist at SC 2010 • 128x256x512 data grid per process, 8 nodes • Oakley cluster at OSC: two hex-core Intel Westmere processors, two NVIDIA Tesla M2070, one Mellanox IB QDR 28 Outline GTC ’15
• Communica on on InfiniBand Clusters with GPUs • MVAPICH2-GPU: CUDA-Aware MPI • MVAPICH2-GPU with GPUDirect-RDMA (MVAPICH2-GDR) • Two-sided Communica on • One-sided Communica on • MPI Datatype Processing • Op miza ons (mul -GPU, device selec on, topology detec on) • MPI and OpenACC • Conclusions
29 Performance of MVAPICH2 with GPU-Direct-RDMA: MPI-3 RMA GTC ’15
GPU-GPU Internode MPI Put latency (RMA put operation Device to Device )
MPI-3 RMA provides flexible synchronization and completion primitives
Small Message Latency 35 MV2-GDR2.1a 30 MV2-GDR2.0b 25 7X 20 Intel Ivy Bridge (E5-2680 15 v2) node with 20 cores NVIDIA Tesla K40c GPU, Mellanox Connect-IB Dual-
Latency(us) 10 2.88 us FDR HCA 5 CUDA 6.5, Mellanox OFED 2.1 with GPU-Direct-RDMA 0 0 2 8 32 128 512 2K 8K Message Size (bytes) 30
OSU-GTC 2015 Communica on Kernel Evalua on with MPI3-RMA: 3Dstencil GTC ’15
3D Stencil with 16 GPU nodes
Send/Recv One-sided 90 80 70 60 50 Based on 40 42% MVAPICH2-2.0b 30 Intel Sandy Bridge 20 (E5-2670) node with 16 Latency (usec) Latency cores 10 NVIDIA Tesla K40c GPU, 0 Mellanox Connect-IB 1 4 16 64 256 1K 4K Dual-FDR HCA CUDA 5.5, Mellanox OFED 2.1 with Message Size (Bytes) GPUDirect-RDMA Plugin
31 Outline GTC ’15
• Communica on on InfiniBand Clusters with GPUs • MVAPICH2-GPU: CUDA-Aware MPI • MVAPICH2-GPU with GPUDirect-RDMA (MVAPICH2-GDR) • Two-sided Communica on • One-sided Communica on • MPI Datatype Processing • Op miza ons (mul -GPU, device selec on, topology detec on) • MPI and OpenACC • Conclusions
32 Non-con guous Data Exchange GTC ’15 Halo data exchange
• Multi-dimensional data • Row based organization • Contiguous on one dimension • Non-contiguous on other dimensions • Halo data exchange • Duplicate the boundary • Exchange the boundary in each iteration
33 MPI Datatype support in MVAPICH2 GTC ’15 • Datatypes support in MPI - Operate on customized datatypes to improve produc vity - Enable MPI library to op mize non-con guous data At Sender: MPI_Type_vector (n_blocks, n_elements, stride, old_type, &new_type); MPI_Type_commit(&new_type); … MPI_Send(s_buf, size, new_type, dest, tag, MPI_COMM_WORLD); • Inside MVAPICH2 - Use datatype specific CUDA Kernels to pack data in chunks - Efficiently move data between nodes using RDMA - Optimizes datatypes from GPU memories - Transparent to the user
H. Wang, S. Potluri, D. Bureddy, C. Rosales and D. K. Panda, GPU-aware MPI on RDMA-Enabled Clusters: Design, 34 Implementation and Evaluation, IEEE Transactions on Parallel and Distributed Systems. MPI Datatype Processing GTC ’15
• Comprehensive support • Targeted kernels for regular datatypes - vector, subarray, indexed_block • Generic kernels for all other irregular datatypes • Separate non-blocking stream for kernels launched by MPI library • Avoids stream conflicts with applica on kernels • Flexible set of parameters for users to tune kernels • Vector • MV2_CUDA_KERNEL_VECTOR_TIDBLK_SIZE • MV2_CUDA_KERNEL_VECTOR_YSIZE • Subarray • MV2_CUDA_KERNEL_SUBARR_TIDBLK_SIZE • MV2_CUDA_KERNEL_SUBARR_XDIM • MV2_CUDA_KERNEL_SUBARR_YDIM • MV2_CUDA_KERNEL_SUBARR_ZDIM R. Shi, X. Lu, S. Potluri, K. Hamidouche, J. Zhang and D. K. Panda, • Indexed_block HAND: A Hybrid Approach to Accelerate Non-con guous Data • MV2_CUDA_KERNEL_IDXBLK_XDIM Movement using MPI Datatypes on GPU Clusters Interna onal Conference on Parallel Processing (ICPP'14). 35 Performance of Stencil3D (3D subarray) GTC ’15
Stencil3D communica on kernel on 2 GPUs with DT TR Enhanced various X, Y, Z dimensions using MPI_Isend/Irecv 2.5
• DT: Direct Transfer, TR: Targeted Kernel 2 1.5 • Op mized design gains up to 15%, 15% and 1
22% compared to TR, and more than 86% latency (ms) 0.5 0 compared to DT on X, Y and Z respec vely 1 2 4 8 16 32 64 128 256 size of DimX, [x,16,16]
2.5 2.5
2 2 1.5 1.5 1 86% 1 latency (ms) latency (ms) 0.5 0.5 0 0 1 2 4 8 16 32 64 128 256 1 2 4 8 16 32 64 128 256 size of DimY, [16,y,16] size of DimZ, [16,16,z] 36 Outline GTC ’15
• Communica on on InfiniBand Clusters with GPUs • MVAPICH2-GPU: CUDA-Aware MPI • MVAPICH2-GPU with GPUDirect-RDMA (MVAPICH2-GDR) • Two-sided Communica on • One-sided Communica on • MPI Datatype Processing • Op miza ons (mul -GPU, device selec on, topology detec on) • MPI and OpenACC • Conclusions
37 Mul -GPU Configura ons (K80 case study) GTC ’15 Process 0 Process 1
• Mul -GPU node architectures are becoming common Memory • Un l CUDA 3.2 - Communica on between processes staged through the host - Shared Memory (pipelined) - Network Loopback [asynchronous) CPU • CUDA 4.0 and later - Inter-Process Communica on (IPC) - Host bypass - Handled by a DMA Engine
I/O Hub - Low latency and Asynchronous - Requires crea on, exchange and mapping of memory handles - Overhead GPU GPU 0 1 38 HCA Intra-Node Communica on Enhancement GTC ’15
800 70000 700 60000 MV2-2.1a 600 MV2-2.1a Enhanced 50000 500 Enhanced 4.7X 40000 400 7.8X 30000 300 200 20000 Latency(us) 100 10000 Bandwidth(MB/s) Bandwidth(MB/s) 0 0 0 8 128 2K 32K 512K 1 16 256 4K 64K 1M Message Size (bytes) Message Size (bytes)
• Two Processes sharing the same K80 GPUs • Proposed designs achieve 4.7X improvement in latency • 7.8X improvement is delivered for Bandwidth • Will be available with next releases of MVAPICH2-GDR 39 Device Selec on GTC ’15
• MVAPICH2 1.8, 1.9 and other MPI libraries require device selec on before MPI_Init • This restric on has been removed with MVAPICH2 2.0 and later • OSU micro-benchmarks s ll select device before MPI_Init for backward compa bility • Uses node-level rank informa on exposed by launchers • We provide a script which exports this node rank informa on to be used by the benchmark - Can be modified to work with different MPI libraries without modifying the benchmarks themselves - Sample script provided with OMB : “get_local_rank” export LOCAL_RANK=$MV2_COMM_WORLD_LOCAL_RANK
40 Topology Detec on GTC ’15
• Automa c detec on of GPUs and IB placement: • Avoid the inter-sockets QPI bo lenecks • Selec on of the nearest HCA and binding the process to the associate CPU set • Dynamic threshold selec on between GDR and host-based transfers
41 Outline GTC ’15
• Communica on on InfiniBand Clusters with GPUs • MVAPICH2-GPU: CUDA-Aware MPI • MVAPICH2-GPU with GPUDirect-RDMA (MVAPICH2-GDR) • Two-sided Communica on • One-sided Communica on • MPI Datatype Processing • Op miza ons (mul -GPU, device selec on, topology detec on) • MPI and OpenACC • Conclusions
42 OpenACC GTC ’15
• OpenACC is gaining popularity • Several sessions during GTC • A set of compiler direc ves (#pragma) • Offload specific loops or parallelizable sec ons in code onto accelerators #pragma acc region { for(i = 0; i < size; i++) { A[i] = B[i] + C[i]; } } • Rou nes to allocate/free memory on accelerators buffer = acc_malloc(MYBUFSIZE); acc_free(buffer); • Supported for C, C++ and Fortran • Huge list of modifiers – copy, copyout, private, independent, etc..
43 Using MVAPICH2 with OpenACC 1.0 GTC ’15
• acc_malloc to allocate device memory - No changes to MPI calls - MVAPICH2 detects the device pointer and op mizes data movement - Delivers the same performance as with CUDA
A = acc_malloc(sizeof(int) * N);
…...
#pragma acc parallel loop deviceptr(A) . . . //compute for loop
MPI_Send (A, N, MPI_INT, 0, 1, MPI_COMM_WORLD);
…… acc_free(A); 44 Using MVAPICH2 with OpenACC 2.0 GTC ’15
• acc_deviceptr to get device pointer (in OpenACC 2.0) - Enables MPI communica on from memory allocated by compiler when it is available in OpenACC 2.0 implementa ons - MVAPICH2 will detect the device pointer and op mize communica on - Delivers the same performance as with CUDA A = malloc(sizeof(int) * N);
…...
#pragma acc data copyin(A) . . . {
#pragma acc parallel loop . . . //compute for loop
MPI_Send(acc_deviceptr(A), N, MPI_INT, 0, 1, MPI_COMM_WORLD);
}
…… free(A); 45 CUDA and OpenACC Extensions in OMB GTC ’15
• OSU Micro-benchmarks (OMB) are widely used to compare performance of different MPI stacks and networks • Enhancements to measure performance of MPI communica on from GPU memory - Point-to-point: Latency, Bandwidth and Bi-direc onal Bandwidth - Collec ves: Latency test for all collec ves • Support for CUDA and OpenACC • Flexible selec on of data movement between CPU(H) and GPU(D): D->D, D->H and H->D • Available from h p://mvapich.cse.ohio-state.edu/benchmarks • Available in an integrated manner with MVAPICH2 stack
46 Outline GTC ’15
• Communica on on InfiniBand Clusters with GPUs • MVAPICH2-GPU: CUDA-Aware MPI • MVAPICH2-GPU with GPUDirect-RDMA (MVAPICH2-GDR) • Two-sided Communica on • One-sided Communica on • MPI Datatype Processing • Op miza ons (mul -GPU, device selec on, topology detec on) • MPI and OpenACC • Conclusions
47 Conclusions GTC ’15
• MVAPICH2 op mizes MPI communica on on InfiniBand clusters with GPUs • Close partnership with NVIDIA during the last four years • Provides op mized designs for point-to-point two-sided, one-sided communica ons, and datatype processing • Takes advantage of CUDA features like IPC, GPUDirect RDMA and GDRCOPY • Delivers - High performance - High produc vity With support for latest NVIDIA GPUs and InfiniBand Adapters 48 Acknowledgments GTC ’15
Dr. Davide Rosse
Filippo Spiga and Stuart Rankin, HPCS, University of Cambridge (Wilkes Cluster)
49 Funding Acknowledgments GTC ’15
Funding Support by
Equipment Support by
50 Personnel Acknowledgments GTC ’15
Current Students Current Senior Research Associates – A. Awan (Ph.D.) – K. Hamidouche – M. Li (Ph.D.) – H. Subramoni – A. Bhat (M.S.) – X. Lu – M. Rahman (Ph.D.) – S. Chakraborthy (Ph.D.) – D. Shankar (Ph.D.) – C.-H. Chu (Ph.D.) Current Post-Doc Current Programmer – A. Venkatesh (Ph.D.) – N. Islam (Ph.D.) – J. Lin – J. Perkins – J. Zhang (Ph.D.) Past Students – W. Huang (Ph.D.) – P. Balaji (Ph.D.) Current Research Specialist – W. Jiang (M.S.) – M. Arnold – D. Buntinas (Ph.D.) – J. Jose (Ph.D.) – S. Bhagvat (M.S.) – S. Kini (M.S.) – M. Luo (Ph.D.) – L. Chai (Ph.D.) – G. Santhanaraman (Ph.D.) – M. Koop (Ph.D.) – A. Mamidala (Ph.D.) – B. Chandrasekharan (M.S.) – A. Singh (Ph.D.) – R. Kumar (M.S.) – G. Marsh (M.S.) – N. Dandapanthula (M.S.) – J. Sridhar (M.S.) – S. Krishnamoorthy (M.S.) – V. Meshram (M.S.) – V. Dhanraj (M.S.) – S. Sur (Ph.D.) – K. Kandalla (Ph.D.) – S. Naravula (Ph.D.) – T. Gangadharappa (M.S.) – H. Subramoni (Ph.D.) – P. Lai (M.S.) – R. Noronha (Ph.D.) – K. Gopalakrishnan (M.S.) – K. Vaidyanathan (Ph.D.) – J. Liu (Ph.D.) – X. Ouyang (Ph.D.) – A. Vishnu (Ph.D.) – S. Pai (M.S.) – E. Mancini – J. Wu (Ph.D.) – S. Potluri (Ph.D.) – S. Marcarelli – W. Yu (Ph.D.) Past Post-Docs – J. Vienne – R. Rajachandrasekar (Ph.D.) – H. Wang – X. Besseron Past Programmers Past Research Scientist – H.-W. Jin – D. Bureddy – M. Luo – S. Sur 51 Thanks! Ques ons? GTC ’15 Contact [email protected] [email protected]
http://mvapich.cse.ohio-state.edu http://nowlab.cse.ohio-state.edu
52