Heterogeneous Parallel Computing CUDA C and OpenCL Chris Lamb nvidia.com/cuda Graphics Processing Unit Crysis © 2006 Crytek / Electronic Arts Hellgate: London © 2005-2006 Flagship Studios, Inc. Licensed by NAMCO BANDAI Games America, Inc. Full Spectrum Warrior: Ten Hammers © 2006 Pandemic Studios, LLC. All rights reserved. © 2006 THQ Inc. All rights reserved. © NVIDIA Corporation 2008 “GPU Computing” makes you think of this… © NVIDIA Corporation 2008 But with CUDA you also get… 146X 36X 19X 17X 100X 149X 47X 20X 24X 30X © NVIDIA Corporation 2008 GPUs Are Fast 1995 1999 2002 2003 2004 2005 2006-2007 NV1 GeForce 256 GeForce4 GeForce FX GeForce 6 GeForce 7 GeForce 8 1M Trans 22M Trans 63M Trans 130M Trans 222M Trans 302M Trans 754M Trans 2008 1000 GT200 GeForce GTX 200 GFLOPS 1.4 Billion 750 Transistors 500 G80 G70 250 Quad Core Xeon 3 GHz (96 GFLOPS) 0 Jan Jun Apr May Nov June 2003 2004 2005 2006 2008 © NVIDIA Corporation 2008 GT200 – Consumer Supercomputer GT200 theoretical peak performance: 1 TFLOPS single precision 87 GFLOPS double precision Raw memory bandwidth: ~142 GB/sec Linpack (DGEMM) 95% of peak: sustained 82.4 GFLOPS 8 node (8U!) cluster achieves Linpack Rmax 1.25 TFLOPS Video game market subsidizes R&D > 100 million units shipped Economies of scale at work: GTX295 costs <$500! All NVIDIA cards since 2006 support CUDA © NVIDIA Corporation 2008 Building a 100TF datacenter CPU 1U 4 CPU cores 4 GPUs: 960 cores Tesla 1U Server System 0.07 Teraflop 4 Teraflops $ 2000 $ 8000 400 W 700 W 1429 CPU servers 25 CPU servers 25 Tesla systems $ 3.1 M $ 0.31 M 10x lower cost 571 KW 27 KW 21x lower power © NVIDIA Corporation 2008 “Fast” = High Throughput How to design an architecture for throughput? Throughput vs Latency Maximize parallelism Graphics is a throughput problem So is scientific computing and many other important and emerging problems! © NVIDIA Corporation 2008 The Trend Era of faster CPUs is over There won’t be 10GHz chips “Moore’s Law” => more transistors => wider units It’s got to be parallel to be fast Not just NVIDIA party line – everyone’s saying this GPUs are already at where CPU are going CPU today = 8 cores GT200 = 240 cores © NVIDIA Corporation 2008 NVIDIA GPU Architecture GT200 Memory & I/O Acceleration Communication Fabric Fixed Function 240 scalar cores On-chip memory © NVIDIA Corporation 2008 Texture units Streaming Multiprocessor (SM) 8 scalar cores (SP) per SM 16K 32-bit registers (64KB) usual ops: float, int, branch, … SM special ops: exp, sin, cos, sqrt, mul24, saturate, … synchronization support Inst. Cache Const. Cache Shared double precision unit IEEE 754 64-bit floating point MT Issue fused multiply-add full-speed denorm. operands and results SP SP SP SP Direct load/store to memory high bandwidth (~142 GB/sec) SP SP SP SP Low-latency on-chip memory 16KB available per SM SFU SFU shared amongst threads of a block SM DP supports thread communication Memory © NVIDIA Corporation 2008 Key Architectural Ideas SM SIMT (Single Instruction Multiple Thread) execution Inst. Cache threads run in groups of 32 called warps Const. Cache threads in a warp share instruction unit (IU) MT Issue 1 instruction x 32 threads issued in 4 clocks SP SP HW automatically handles divergence SP SP SP SP Hardware multithreading SP SP HW resource allocation & thread scheduling SFU SFU HW relies on threads to hide latency DP any warp not waiting for something can run context switching is (basically) free Memory © NVIDIA Corporation 2008 How do you write fast code? Task parallelism is short lived… More cores, more memory latency Data parallel is the future Express a problem as data parallel.... Maps automatically to a scalable architecture CUDA architecture provides an insight into a data parallel future © NVIDIA Corporation 2008 CUDA – NVIDIA’s Parallel Computing Architecture PTX ISA and hardware compute engine Includes a C-compiler & support for OpenCL and DX11 Compute Architected to natively support all computational interfaces ATI’s Compute (standard languages and “Solution” APIs) © NVIDIA Corporation 2008 OpenCL and C for CUDA Entry point for developers who prefer high-level C Entry point for expert developers who want a powerful device management API Shared back-end compiler, optimization technology and ISA Runs on the exact same hardware © NVIDIA Corporation 2008 C for CUDA Hierarchy of concurrent threads Parallel kernel run by many threads Thread t all threads execute the same piece of code threads can take different paths through code Threads are grouped into thread blocks Block b threads in the same block can cooperate t0 t1 … tB Threads/blocks have unique IDs © NVIDIA Corporation 2008 Memory hierarchy Thread has registers Thread t Block has shared memory Fast access by threads (fast like registers/cache) Device has off-chip global memory Block b Same DRAM used to store images for graphics t0 t1 … tB memcpy over PCIE to/from host memory © NVIDIA Corporation 2008 Example: Vector Addition Kernel Device Code // Compute vector sum C = A+B // Each thread performs one pair-wise addition __global__ void vecAdd(float* A, float* B, float* C) { int i = threadIdx.x + blockDim.x * blockIdx.x; C[i] = A[i] + B[i]; } int main() { // Run N/256 blocks of 256 threads each vecAdd<<< N/256, 256>>>(d_A, d_B, d_C); } © NVIDIA Corporation 2008 Example: Vector Addition Kernel // Compute vector sum C = A+B // Each thread performs one pair-wise addition __global__ void vecAdd(float* A, float* B, float* C) { int i = threadIdx.x + blockDim.x * blockIdx.x; C[i] = A[i] + B[i]; } Host Code int main() { // Run N/256 blocks of 256 threads each vecAdd<<< N/256, 256>>>(d_A, d_B, d_C); } © NVIDIA Corporation 2008 Example: Host code for vecAdd // allocate and initialize host (CPU) memory float *h_A = …, *h_B = …; // allocate device (GPU) memory float *d_A, *d_B, *d_C; cudaMalloc( (void**) &d_A, N * sizeof(float)); cudaMalloc( (void**) &d_B, N * sizeof(float)); cudaMalloc( (void**) &d_C, N * sizeof(float)); // copy host memory to device cudaMemcpy( d_A, h_A, N * sizeof(float), cudaMemcpyHostToDevice) ); cudaMemcpy( d_B, h_B, N * sizeof(float), cudaMemcpyHostToDevice) ); // execute the kernel on N/256 blocks of 256 threads each vecAdd<<<N/256, 256>>>(d_A, d_B, d_C); © NVIDIA Corporation 2008 Why is this different from a CPU? Different goals produce different designs GPU assumes work load is highly parallel CPU must be good at everything, parallel or not CPU: minimize latency experienced by 1 thread big on-chip caches sophisticated control logic GPU: maximize throughput of all threads # threads in flight limited by resources => lots of resources (registers, bandwidth, etc.) multithreading can hide latency => skip the big caches amortize cost of control logic via SIMT © NVIDIA Corporation 2008 OpenCL CUDA C and OpenCL Programming Styles C for CUDA C with parallel keywords C runtime that abstracts driver API Memory managed by C runtime Generates PTX OpenCL Hardware API - similar to OpenGL Programmer has complete access to hardware device Memory managed by programmer Generates PTX © NVIDIA Corporation 2008 Kernel Execution Total number of work-items = Gx x Gy Size of each work-group = Sx x Sy Global ID can be computed from work-group ID and © NVIDIAlocal Corporation 2008 ID OpenCL Memory Model (Section 3.3) Shared memory model Relaxed consistency Multiple distinct address spaces Address spaces can be collapsed depending on the device’s memory Private Private Private Private subsystem Memory Memory Memory Memory Work Item Work Item Work Item 1 Work Item 1 Address spaces M M Private - private to a work-item Compute Unit 1 Compute Unit N Local - local to a work-group Global - accessible by all work- Local Memory Local Memory items in all work-groups Constant - read only global space Global / Constant Memory Data Cache Implementations map this Compute Device hierarchy To available physical memories Global Memory Compute Device Memory © NVIDIA Corporation 2008 VecAdd: Create Memory Objects cl_mem memobjs[3]; // allocate input buffer memory objects memobjs[0] = clCreateBuffer(context, CL_MEM_READ_ONLY | // flags CL_MEM_COPY_HOST_PTR, sizeof(cl_float)*n, // size srcA, // host pointer NULL); // error code memobjs[1] = clCreateBuffer(context, CL_MEM_READ_ONLY | CL_MEM_COPY_HOST_PTR, sizeof(cl_float)*n, srcB, NULL); // allocate input buffer memory object memobjs[2] = clCreateBuffer(context, CL_MEM_WRITE_ONLY, sizeof(cl_float)*n, NULL, NULL); Creating buffer objects: Section 5.2.1 © NVIDIA Corporation 2008 VecAdd: Program and Kernel // create the program cl_program program = clCreateProgramWithSource( context, 1, // string count &program_source, // program strings NULL, // string lengths NULL); // error code // build the program cl_int err = clBuildProgram(program, 0, // num devices in device list NULL, // device list NULL, // options NULL, // notifier callback function ptr NULL); // user data // create the kernel cl_kernel kernel = clCreateKernel(program, “vec_add”, NULL); Creating program objects: Section 5.4.1 Building program executables: Section 5.4.2 © NVIDIA Corporation 2008 Creating kernel objects: Section 5.5.1 VecAdd: Set Kernel Arguments // set “a” vector argument err = clSetKernelArg(kernel, 0, // argument index (void *)&memobjs[0], // argument data sizeof(cl_mem)); // argument data size // set “b” vector argument err |= clSetKernelArg(kernel, 1, (void *)&memobjs[1], sizeof(cl_mem)); // set “c” vector argument err |= clSetKernelArg(kernel, 2, (void *)&memobjs[2], sizeof(cl_mem)); Setting kernel arguments: Section 5.5.2 Executing Kernels: Section 6.1 Reading,