CUDA NEW FEATURES and BEYOND Stephen Jones, GTC 2019 a QUICK LOOK BACK This Time Last Year

CUDA NEW FEATURES AND BEYOND Stephen Jones, GTC 2019 A QUICK LOOK BACK This Time Last Year...

3x3 ReLU convolution

5x5 input ReLU concat convolution

max 1x1 ReLU pool convolution

DGX-2 + Unified Memory Asynchronous Task Graphs

S9241 – All You Need To Know About Programming NVIDIA’s DGX-2, Wednesday March 20, 1-2PM

2 ACCELERATED COMPUTING IS FULL-STACK OPTIMIZATION 2X More Performance With Software Optimizations Alone HPC Applications Speedup 24x

20x CUDA 10 CUBLAS 10 16x CUFFT 10

12x 2X CUDA 8 on same CUBLAS 8 hardware 8x CUFFT 8

0x 1 2 3 4 5 2x Broadwell vs 4xP100 2x Broadwell vs 4xV100

3 HPC Apps: AMBER, Chroma, GROMACS, GTC, LAMMPS, MILC, NAMD, QE, RTM, SPECFEM3D, VASP TESLA UNIVERSAL ACCELERATION PLATFORM Single Platform To Drive Utilization and Productivity

CUSTOMER USECASES Speech Translate Recommender Healthcare Manufacturing Finance Molecular Weather Seismic Simulations Forecasting Mapping CONSUMER INTERNET INDUSTRIAL APPLICATIONS SUPERCOMPUTING

APPS & Amber +550 FRAMEWORKS Applications NAMD

MACHINE LEARNING | RAPIDS DEEP LEARNING SUPERCOMPUTING NVIDIA SDK & LIBRARIES cuDF cuML cuGRAPH cuDNN cuBLAS CUTLASS NCCL TensorRT CuBLAS CuFFT OpenACC

CUDA

TESLA GPUs & SYSTEMS

TESLA GPU VIRTUAL GPU NVIDIA DGX FAMILY NVIDIA HGX SYSTEM OEM CLOUD 4 TECHNOLOGY PLATFORM DEVELOPMENT TOOLKIT

5 NEW TURING GPU GREATEST LEAP SINCE 2006 CUDA GPU

6 TESLA T4 WORLD’S MOST ADVANCED SCALE-OUT GPU

320 Turing Tensor Cores 2,560 CUDA Cores 65 FP16 TFLOPS | 130 INT8 TOPS | 260 INT4 TOPS 16GB | 320GB/s 70 W

Deep Learning Training & Inference HPC Workloads Video Transcode Remote Graphics

7 TURING SM

TU102 INT32 64 FP32 64 Tensor Cores 8 RT Core 1 Register File 256 KB L1 and shmem 96 KB Max threads 1024 Compute Capability 75*

*Volta (cc70) code runs on Turing without JIT or recompile! 8 RT CORE POTENTIAL FOR ACCELERATION OF NUMERICAL ALGORITHMS

Geometry-Heavy Compute Applications Unstructured Algorithms

Nearest Neighbor Search Credit: Fortmann-Roe Neutron Transport Credit: CERT, Texas A&M

R-Trees, Decision Trees RF Wave Propagation Credit: Wikimedia Credit: COMSOL Seismic Shear Wave Tracing Credit: SERC, Carleton College Radiaton Transport Credit: Greg Stewart / SLAC 9 LOCATING NEIGHBORS WITHIN A RANGE Intersect Rays With Bounding Box Around Points Of Interest

For any arbitrary set of points

For a point P, find neighbors within a shape enclosed in a Bounding Box

Ray-based solution P

1. Attach a box of width R to each point

2. Shoot one ray from P in arbitrary direction, t_max = 2*R

3. Neighbors boxes will have either entry/exit intersection but never both.

4. Refine result points to any shape within the box in SM.

10 RAY TRACED NEAREST NEIGHBOUR SEARCH Using RT-Cores Through OptiX RTX

11 NEW TURING TENSOR CORE

MULTI-PRECISION FOR AI INFERENCE & SCALE-OUT TRAINING 65 TFLOPS FP16 | 130 TeraOPS INT8 | 260 TeraOPS INT4

12 TURING TENSOR CORE New 8-Bit & Sub-Byte Warp Matrix Functions In CUDA

WMMA 16x16x16 8-bit integer WMMA operations = + D A B C ▪ Turing (sm_75) only 16x16 16x16 16x16 16x16 WMMA 32x8x16 ▪ Signed & unsigned 8-bit input = + ▪ 32-bit integer accumulator

D A B C ▪ Match input/output dimensions 32x8 32x16 16x8 32x8 with half WMMA 8x32x16 = + ▪ 2048 ops per cycle, per SM D A B C 8x32 8x16 16x32 8x32

13 EXPERIMENTAL WARP MATRIX FUNCTIONS Turing Enables Experimental Sub-Byte Tensor Core Operations

Experimental Sub-Byte Operations namespace experimental { 4-bit signed & unsigned input namespace precision { 1-bit input with custom matrix operations struct u4; // 4-bit unsigned struct s4; // 4-bit signed 32-bit accumulator output struct b1; // 1-bit } Access via special namespace enum bmmaBitOp { bmmaBitOpXOR = 1 }; nvcuda::wmma::experimental enum bmmaAccumulateOp { bmmaAccumulateOpPOPC = 1 }; }

Enables researchers to experiment with ultra low precision

14 BINARY TENSOR CORES Example: Binarized Neural Networks

1-bit

Concept

▪ Train neural networks on lower-precision data: faster compute, lower memory size

▪ Reduce data to positive / negative sign value – can fit in single bit (1 = +ve, 0 = -ve)

▪ 1-bit weight & activation calculations based only on sign of data

Ref: Binarized Neural Networks: Training Neural Networks with Weights and Activations Constrained to +1 or −1, M. Coubariaux, I. Hubara, D. Soudry, R. El-Yaniv, Y Bengio, 2016 https://arxiv.org/pdf/1602.02830.pdf 15 BINARY TENSOR CORE OPERATION

128-bit population Bitwise 32-bit Integer Output 1-Bit Input Signal count added to XOR Operation Per Point accumulator

Other Row/Column Results

Accumulated Bitwise 32-bit Integer XOR + Count

Previous Accumulation

16 NEW TURING WARP MATRIX FUNCTIONS

Input Precision Output Supported Sizes Max Ops/Clock/SM

half * half or float 1024 16 x 16 x 16 char 32 x 8 x 16 integer (int32) 8 x 32 x 16 2048

unsigned char Native Types Native

precision::u4 (4-bit unsigned) 8 x 8 x 32 4096 precision::s4 (4-bit signed) integer (int32)

Experimental precision::b1 (1-bit) 8 x 8 x 128 16384

* Also available on Volta sm_70. Note: WMMA requires recompilation for Turing sm_75 for peak performance 17 CUTLASS 1.3 GEMM kernels targeting Volta Tensor Cores natively with mma.sync

New in CUDA 10.1 & CUTLASS 1.3: mma.sync PTX assembly instruction enables maximum efficiency of Volta Tensor Cores operation 18 INDEPENDENT THREAD SCHEDULING Communicating Algorithms

Pascal: Lock-Free Algorithms Volta/Turing: Starvation Free Algorithms

Threads cannot wait for messages Threads may wait for messages 19 INDEPENDENT THREAD SCHEDULING Enable Fast Mutexes For Concurrent Data Structures, Replace Complex Lock-Free Algorithms

Multi-threading (CPU)

Acceleration (RTX 2070)

Ref: High Radix Concurrent C++, Olivier Giroux, CppCon 2018 - https://www.youtube.com/watch?v=75LcDvlEIYw

See Also: https://devblogs.nvidia.com/cuda-turing-new-gpu-compute-possibilities/ 20 WARP IMPLEMENTATIONS

Pre-Volta Program Counter (PC) and Stack (S) 32 thread warp

Volta/Turing

PC,S PC,S PC,S PC,S

PC,S PC,S PC,S PC,S PC,S PC,S PC,S PC,S PC,S PC,S PC,S PC,S PC,S PC,S PC,S PC,S PC,S PC,S PC,S PC,S PC,S PC,S PC,S PC,S PC,S PC,S PC,S Convergence PC,S Optimizer

32 thread warp with independent scheduling

21 SYNCHRONIZING WARP FUNCTIONS

Pre-Volta Volta & Turing