Parallel Programming Languages: a New Renaissance Or a Return to the Dark Ages?

Parallel programming languages: A new renaissance or a return to the dark ages?

Simon McIntosh-Smith University of Bristol Microelectronics Research Group [email protected]

Simon McIntosh-Smith Dr Jose Nunez-Yanez Head of Group

Prof David May Dr Kerstin Eder

Prof Dhiraj Pradhan Dr Simon Hollis

Energy Aware COmputing (EACO): •Multi-core and many-core computer architectures •High Performance Computing (GPUs, OpenCL) •Network on Chip (NoC) •Reconfigurable architectures •Design verification (formal and simulation-based) •Formal specification and analysis •Silicon process variation •Fault tolerant design (hardware and software)

Quad-core ARM Cortex A9 CPU Quad-core SGX543MP4+ Imagination GPU

548,352 processor cores delivering 8.16 PetaFLOPS (8.16x1015)

http://www.nytimes.com/2006/06/14/technology/14search.html

Report: Google Uses About 900,000 Servers (Aug 1st 2011) http://www.datacenterknowledge.com/archives/2011/08/01/ report-google-uses-about-900000-servers/

OpenMP OpenCL Erlang Unified Parallel C Fortress XC Go Cilk HMPP CHARM++ CUDA Co-Array Fortran Chapel Linda X10 MPI Pthreads C++ AMP

Partitioned Global Address Space GPU languages: (PGAS): •OpenCL •Fortress •CUDA •X10 •HMPP •Chapel •Co-array Fortran Object oriented: •Unified Parallel C •C++ AMP •CHARM++ CSP: XC Multi-threaded: Message passing: MPI •Cilk •Go Shared memory: OpenMP

• OpenCL, DirectCompute, C++ AMP, …

• NVIDIA’s ‘de facto’ standard CUDA

A modern system includes: CPU CPU – One or more CPUs – One or more GPUs – DSP processors – …other? GPU GMCH

DRAM ICH

OpenCLOpenCL letslets programmersprogrammers writewrite aa singlesingle portableportable programprogram thatthat usesuses ALLALLresourcesresources inin thethe

heterogeneousheterogeneous© Simon McIntosh-Smith platformplatform 17 GMCH = graphics memory control hub, ICH = Input/output control hub Industry Standards for Programming Heterogeneous Platforms

CPUs GPUs Multiple cores driving Emerging Increasingly general performance increases purpose data-parallel Intersection computing

Multi- Graphics processor Heterogeneous APIs and programming Computing Shading e.g. OpenMP Languages

OpenCLOpenCL – – Open Open Computing Computing Language Language Open,Open, royalty-free royalty-free standard standard for for portable, portable, parallel parallel programming programming of of heterogeneousheterogeneous parallel parallel computing computing© Simon McIntosh-Smith CPUs, CPUs, GPUs, GPUs, and and other other processors processors18 The origins of OpenCL Ericsson Nokia AMD Merged, IBM needed commonality across Sony products ATI IMG Tech GPU vendor - Freescale wants to steal mkt Nvidia Wrote a share from CPU Khronos TI rough draft Compute straw man group formed + many CPU vendor - API more Intel wants to steal mkt share from GPU

was tired of recoding for Apple many core, GPUs. Pushed vendors to standardize. Dec 2008 © Simon McIntosh-Smith 19 Third party names are the property of their owners. System level architecture of OpenCL

OpenCL programs

OpenCL dynamic libraries

Vendor-supplied drivers

Could also be multi-core CPUs

• One Host + one or more Compute Devices • Each Compute Device is composed of one or more Compute Units • Each Compute Unit is further divided into one or more Processing Elements © Simon McIntosh-Smith 21 The BIG idea behind OpenCL • Replace loops with functions (a kernel) executing at each point in a problem domain (index space). • E.g., process a 1024 x 1024 image with one kernel invocation per pixel or 1024 x 1024 = 1,048,576 kernel executions Traditional loops Data Parallel OpenCL void kernel void trad_mul(const int n, dp_mul(global const float *a, const float *a, global const float *b, const float *b, global float *c) float *c) { { int id = get_global_id(0); int i; for (i=0; i

1024 Synchronization between work-items possible only within work-groups: barriers and memory fences

1024 Cannot synchronize outside of a work-group

• Private Memory Private Private Private Private Memory Memory Memory Memory • Per Work-Item • Local Memory Work-Item Work-Item Work-Item Work-Item • Shared within a Work-Group Local Memory • Global / Constant Local Memory Work-Group Memories Work-Group • Visible to all Work-Groups Global Memory & Constant Memory • Host Memory Compute Device • On the CPU Host Memory

Host • Memory management is explicit You must move data from host Æ global Æ local and back

© Simon McIntosh-Smith 24 OpenCL C Language • Derived from ISO C99 • A “supersubset” (Tim Mattson, Intel) • No standard C99 headers, function pointers, recursion, variable length arrays, or bit fields • Additions to the language for parallelism • Work-items and work-groups • Vector types • Synchronization • Address space qualifiers, e.g. global, local • Optimized image access • Built-in functions

Contexts are used to contain and manage the state of the “world”

Contexts are defined & manipulated by the host and contain: • Devices • Kernels - OpenCL functions • Program objects - kernel sources and executables • Memory objects

Command-queue - coordinates execution of kernels • Kernel execution commands • Memory commands - transfer or mapping of memory object data • Synchronization commands - constrains the order of commands

CPU GPU

Context

ProgramsPrograms Kernels Memory Objects Command Queues

dp_mul __kernel void Buffers Images dp_mul dp_mul( argarg [0][0] CPU program InIn OutOut of of global const float *a, arg[0]valuevalue value binary global const float *b, OrderOrder OrderOrder global float *c) arg [1] dp_mul arg [1] Queue Queue { arg[1]valuevalue value Queu Queu int id = get_global_id(0); GPU program binary c[id] = a[id] * b[id]; argarg [2][2] e e } arg[2]valuevalue value ComputeGPU Device

• The “hello world” program of data parallel programming is a program to add two vectors

C[i] = A[i] + B[i] for i=0 to N-1

• For the OpenCL solution, there are two parts • Kernel code • Host code

__kernel void vec_add (__global const float *a, __global const float *b, __global float *c) { int gid = get_global_id(0); c[gid] = a[gid] + b[gid]; }

// create the OpenCL context on a GPU device // build the program cl_context = clCreateContextFromType(0, err = clBuildProgram(program, 0, NULL, NULL, NULL, CL_DEVICE_TYPE_GPU, NULL, NULL, NULL); NULL);

// get the list of GPU devices associated with context // create the kernel clGetContextInfo(context, CL_CONTEXT_DEVICES, 0, kernel = clCreateKernel(program, “vec_add”, NULL); NULL, &cb); devices = malloc(cb); // set the args values clGetContextInfo(context, CL_CONTEXT_DEVICES, cb, devices, NULL); err = clSetKernelArg(kernel, 0, (void *) &memobjs[0], sizeof(cl_mem)); // create a command-queue err |= clSetKernelArg(kernel, 1, (void *)&memobjs[1], cmd_queue = clCreateCommandQueue(context, devices[0], sizeof(cl_mem)); 0, NULL); err |= clSetKernelArg(kernel, 2, (void *)&memobjs[2], sizeof(cl_mem)); // allocate the buffer memory objects // set work-item dimensions memobjs[0] = clCreateBuffer(context, CL_MEM_READ_ONLY | global_work_size[0] = n; CL_MEM_COPY_HOST_PTR, sizeof(cl_float)*n, srcA, NULL);} memobjs[1] = clCreateBuffer(context,CL_MEM_READ_ONLY | // execute kernel CL_MEM_COPY_HOST_PTR, sizeof(cl_float)*n, srcB, err = clEnqueueNDRangeKernel(cmd_queue, kernel, 1, NULL); NULL, global_work_size, NULL, 0, NULL, NULL); memobjs[2] = clCreateBuffer(context,CL_MEM_WRITE_ONLY, sizeof(cl_float)*n, NULL, // read output array NULL); err = clEnqueueReadBuffer(context, memobjs[2], CL_TRUE, 0, n*sizeof(cl_float), dst, 0, NULL, NULL); // create the program program = clCreateProgramWithSource(context, 1, &program_source, NULL, NULL);

// create the OpenCL context on a GPU device // build theBuild program the program cl_context = clCreateContextFromType(0, err = clBuildProgram(program,BuildBuild thethe programprogram 0, NULL, NULL, NULL, CL_DEVICE_TYPE_GPU, NULL, NULL, NULL); NULL);

// get the list of GPU devices associated with context // create the kernel clGetContextInfo(context, CL_CONTEXT_DEVICES, 0, kernel = clCreateKernel(program, “vec_add”, NULL); Define platform and queuesNULL, &cb); devicesDefine = malloc(cb); platform and queues // set the args values clGetContextInfo(context, CL_CONTEXT_DEVICES, cb, devices, NULL); err Create= clSetKernelArg(kernel, and setup 0, (void kernel *) &memobjs[0], Create and setupsizeof(cl_mem)); kernel // create a command-queue err |= clSetKernelArg(kernel, 1, (void *)&memobjs[1], cmd_queue = clCreateCommandQueue(context, devices[0], sizeof(cl_mem)); 0, NULL); err |= clSetKernelArg(kernel, 2, (void *)&memobjs[2], sizeof(cl_mem)); // allocate the buffer memory objects // set work-item dimensions memobjs[0] = clCreateBuffer(context, CL_MEM_READ_ONLY | global_work_size[0] = n; CL_MEM_COPY_HOST_PTR, sizeof(cl_float)*n, srcA, NULL);} memobjs[1] = clCreateBuffer(context,CL_MEM_READ_ONLY | // executeExecute kernel the kernel CL_MEM_COPY_HOST_PTR,Define Memory sizeof(cl_float)*n, objects srcB, err = clEnqueueNDRangeKernel(cmd_queue,Execute the kernel kernel, 1, NULL);Define Memory objects NULL, global_work_size, NULL, 0, NULL, NULL); memobjs[2] = clCreateBuffer(context,CL_MEM_WRITE_ONLY, sizeof(cl_float)*n, NULL, // read output array NULL); err = clEnqueueReadBuffer(context, memobjs[2], CL_TRUE, ReadRead0, n*sizeof(cl_float), resultsresults backback dst, 0, to toNULL, thethe NULL); hosthost // create the program program =Create clCreateProgramWithSource(context, the program 1, &program_source,CreateCreate NULL, thethe NULL); programprogram It’s complicated, but most of this is “boilerplate” and not as bad as it looks.

#include const std::string hwstring("Hello World\n"); using namespace cl; int main(void) { char * outH = new char[hwstring.length()+1]; Buffer outCL(CL_MEM_WRITE_ONLY, hwstring.length()+1);

Program program(loadProgram("helloWorld.cl")); program.build(); std::function hello = make_kernel(program, "hello"); hello( EnqueueArgs(NDRange(hwstring.length()+1)), outCL); enqueueReadBuffer(outCL, CL_TRUE, 0, hwstring.length()+1, outH); cout << outH << endl; } C++ API published on Khronos website http://www.khronos.org/registry/cl/

Collaborators: Richard Sessions, Amaurys Avila Ibarra

1,120 seconds per simulation 162 seconds per simulation

Less than 2 days to screen a library of 1 million© Simondrug McIntosh-Smith candidates on 1000 GPUs 36 Relative energy efficiency

0.034 kWh per simulation

0.011 kWh per simulation

0.011 kWh = 0.16 pence per simulation 1 million simulations Æ £1,600 on

• Parallel languages are going through a renaissance

• Not just for the niche high end any more

• No silver bullets, lots of “wheel reinventing”

• In HPC, GPUs being adopted quickly at the high-end

• In embedded computing, OpenCL gaining ground

• Everything going heterogeneous many-core…