Porting Scalable Parallel CFD Application Hifun on NVIDIA GPU

HiFUN on GPU Krishnababu et. al. Porting Scalable Parallel CFD Application HiFUN on NVIDIA GPU

D. V. Krishnababu, N. Munikrishna, Nikhil Vijay Shende 1 N. Balakrishnan 2 Thejaswi Rao 3

1. S & I Engineering Solutions Pvt. Ltd., Bangalore, India 2. Aerospace Engineering, Indian Institute of Science, Banglore, India 3. NVIDIA Graphics Pvt. Ltd., Banglore, India

GPU Technology Conference Silicon Valley March 26–29, 2018 1 / 18 Introduction http://www.sandi.co.in

HiFUN on GPU Krishnababu The HiFUN Software et. al. High Resolution Flow Solver on Unstructured Meshes. A Computational Fluid Dynamics (CFD) Flow Solver. Primary product of the company SandI. Robust, fast, accurate and efﬁcient tool.

About SandI A technology company. Incubated from Indian Institute of Science, Bangalore. Promotes high end CFD technologies with uncompromising quality standards.

2 / 18 Introduction http://www.sandi.co.in

About SandI A technology company. Incubated from Indian Institute of Science, Bangalore. Promotes high end CFD technologies with uncompromising quality standards.

2 / 18 Features of HiFUN http://www.sandi.co.in/home/products

HiFUN on GPU General Krishnababu et. al.

3 / 18 Features of HiFUN http://www.sandi.co.in/home/products

HiFUN on GPU Well Validated Krishnababu et. al.

AIAA DPW SPICES

AIAA HiLiftPW 4 / 18 Features of HiFUN http://www.sandi.co.in/home/products

HiFUN on GPU Super Scalable Workload: 165 Million Volumes Krishnababu et. al.

Simulation CPU Cores Time (Hours/Days) 256 30/1.25 RANS 10000 1 256 108/4.5 URANS 10000 3 256 525/22 DES 10000 15 5 / 18 SandI–NVIDIA Collaboration

HiFUN on GPU HiFUN on NVIDIA Pascal, Volta GPU Way - Krishnababu et. al. Ahead NVLink With IBM Power CPU

2018 - GTC 2018

GTCx Mumbai 2016 - HiFUN in GPU Apps Catalogue GTC 2016: Poster Presentation

2015 - NVIDIA Innovation Award

2014 - Joint Development Initiative Kicks Off

6 / 18 HiFUN on NVIDIA GPU

HiFUN on GPU Krishnababu et. al. Hybrid Supercomputers Consist of CPU and NVIDIA GPU. Less power to achieve same FLOPS. Less cooling & space.

GPU Thousands of computing cores sharing same RAM. Higher memory bandwidth. High data transfer overheads with CPU.

7 / 18 HiFUN on NVIDIA GPU

HiFUN on GPU Krishnababu et. al. Hybrid Supercomputers Consist of CPU and NVIDIA GPU. Less power to achieve same FLOPS. Less cooling & space.

GPU Thousands of computing cores sharing same RAM. Higher memory bandwidth. High data transfer overheads with CPU.

7 / 18 HiFUN on NVIDIA GPU

HiFUN on GPU Krishnababu et. al. Parallelization Model on GPU Shared memory. Many FLOPS per byte of data from CPU to GPU. Re–look at parallelization of CFD algorithms.

Parallelization Challenges General purpose algorithms. Implicit: Global data dependence. Complex multi–layered unstructured data structure.

8 / 18 HiFUN on NVIDIA GPU

HiFUN on GPU Krishnababu et. al. Parallelization Model on GPU Shared memory. Many FLOPS per byte of data from CPU to GPU. Re–look at parallelization of CFD algorithms.

Parallelization Challenges General purpose algorithms. Implicit: Global data dependence. Complex multi–layered unstructured data structure.

8 / 18 HiFUN on NVIDIA GPU

HiFUN on GPU Constraints Krishnababu et. al. No compromise on distributed memory scalability. Source code maintainability should not suffer. Software portability should not suffer.

Parallel Strategy Accelerate single node performance via ofﬂoad model. Hybrid: MPI and OpenACC directives.

Ofﬂoad Model Computationally intensive part is ofﬂoaded to GPU. Optimal data communication between CPU & GPU.

9 / 18 HiFUN on NVIDIA GPU

HiFUN on GPU Constraints Krishnababu et. al. No compromise on distributed memory scalability. Source code maintainability should not suffer. Software portability should not suffer.

Parallel Strategy Accelerate single node performance via ofﬂoad model. Hybrid: MPI and OpenACC directives.

Ofﬂoad Model Computationally intensive part is ofﬂoaded to GPU. Optimal data communication between CPU & GPU.

9 / 18 HiFUN on NVIDIA GPU

HiFUN on GPU Constraints Krishnababu et. al. No compromise on distributed memory scalability. Source code maintainability should not suffer. Software portability should not suffer.

Parallel Strategy Accelerate single node performance via ofﬂoad model. Hybrid: MPI and OpenACC directives.

Ofﬂoad Model Computationally intensive part is ofﬂoaded to GPU. Optimal data communication between CPU & GPU.

9 / 18 HiFUN on NVIDIA GPU

HiFUN on GPU Krishnababu et. al.

Onera M6 NASA CRM Trap Wing

Conﬁgurations & Workloads (Million) Onera M6 Wing: 1.1, 9.3, 12.12, 15.4 NASA CRM: 6.2, 26.5, 30 NASA Trap Wing: 20, 66

Simulation Type Steady RANS Simulations

10 / 18 HiFUN on NVIDIA GPU

HiFUN on GPU Krishnababu et. al.

Onera M6 NASA CRM Trap Wing

Conﬁgurations & Workloads (Million) Onera M6 Wing: 1.1, 9.3, 12.12, 15.4 NASA CRM: 6.2, 26.5, 30 NASA Trap Wing: 20, 66

Simulation Type Steady RANS Simulations

10 / 18 HiFUN on NVIDIA GPU

HiFUN on GPU Computing Platform: NVIDIA PSG Krishnababu et. al. Node conﬁguration Two Hexa–deca core Intel(R) Xeon(R) Haswell processors. Eight NVIDIA Tesla K–80 GPUs. GPU Memory = 12 GB. Total CPU Memory per node = 256 GB. Inﬁniband interconnect

Software PGI Compiler 16.7 OPENMPI 1.10.2 OpenACC 2.0

11 / 18 HiFUN on NVIDIA GPU

Software PGI Compiler 16.7 OPENMPI 1.10.2 OpenACC 2.0

11 / 18 HiFUN on NVIDIA GPU Parallel Performance Parameters

HiFUN on GPU Krishnababu Ideal Speed–up et. al. Ratio of number of nodes used for a given run to reference number of nodes.

Actual Speed–up Ratio of time/iteration using reference number of nodes to time/iteration using number of nodes for given run.

Accelerator Speed–up Ratio of time per iteration obtained using given no. of CPUs to time per iteration obtained using same no. of CPUs working in tandem with GPUs.

12 / 18 HiFUN on NVIDIA GPU Single Node Performance

HiFUN on GPU Krishnababu et. al.

Accelerator Speed–up on 2 GPU Observations Increase in grid size increases GPU utilization and accelerator speed–up. Important to load GPU completely.

13 / 18 HiFUN on NVIDIA GPU Single Node Performance

HiFUN on GPU Krishnababu et. al.

Varying GPUs % Increase

Observations Increase in no. of GPUs increase accelerator speed–up. Use of 4 GPUs per node is optimal.

14 / 18 HiFUN on NVIDIA GPU Single Node Performance

HiFUN on GPU Krishnababu et. al.

Time to RANS Solution (Hours) Observations Time to solution on 1 million grid ∼ 15 minutes. Time to solution on 30 million grid ∼ half a day. Single node serves as a desktop supercomputer. 15 / 18 HiFUN on NVIDIA GPU Multi–node Performance

HiFUN on GPU Krishnababu et. al.

Parallel Speed–up: 66 Million Workload Observations Near linear speed–up using 2 GPUs per node. Drop in speed–up for larger no. nodes and/or higher GPUs due to lower GPU utilization.

16 / 18 HiFUN on NVIDIA GPU Multi–node Performance

HiFUN on GPU Krishnababu et. al.

Normalized Time Per Iteration: 66 Million Workload Observations Drop in time/iter with increase in no. of nodes and/or GPUs. Time to solution with 8 nodes ∼ 4 hours.

17 / 18 HiFUN on NVIDIA GPU

HiFUN on GPU Krishnababu et. al. Concluding Remarks Ofﬂoad model to port HiFUN on GPU. GPU based computing node is powerful enough to serve as desktop supercomputer. HiFUN is ideally suited to solve grand challenge problems on GPU based hybrid supercomputers. OpenACC directives based ofﬂoad model is an attractive option for porting legacy CFD codes on GPU.

18 / 18 HiFUN on NVIDIA GPU

18 / 18