Opencl and SYCL 23 May 2019

OpenCL and SYCL 23 May 2019

Andrew Richards Standards-based stack for AI and HPC

SYCL-ecosystem

Eigen TensorFlow CUDA-to-SYCL SYCL-DNN SYCL-BLAS

OpenCL: C standard for OpenCL-ecosystem SYCL accelerator programming. + many Intel’s new open- Widely used in machine OpenCV clBlast Halide ComputeCpp triSYCL hipSYCL others source SYCL vision

SYCL: C++ standard for accelerator programming OpenCL suitable for CUDA developers Processor-vendor-provided OpenCL Open-source: Pocl, Intel Neo, GPUopen and big deep learning frameworks

Combined standards give a AI processors huge ecosystem of deep learning and vision software GPUs FPGAs DSPs CPUs

2 © 2019 Codeplay Software Ltd Why OpenCL? • OpenCL is supported on a huge range of processors • OpenCL has a large computer vision and HPC ecosystem • SPIR-V in OpenCL enables new accelerated languages • SPIR-V is a standardized compiler IR that enables e.g. AI graph compilers • OpenCL can support lots of special-purpose accelerators • And we’re working to enable support for even more accelerator processors • Gives access to a huge and growing ecosystem of software • OpenCL is not performance-portable • But it enables performance-portable technologies on top

3 © 2019 Codeplay Software Ltd The SYCL Vision • Bringing acceleration to mainstream C++ developers • Make it easy to port from existing single-source programming models • Supports a large ecosystem of open-source AI software • Support the latest AI operators • Enable C++ performance-portability techniques on top of OpenCL

4 © 2019 Codeplay Software Ltd Growing SYCL ecosystem • http://sycl.tech • Growing open-source ecosystem • SYCL-BLAS, SYCL-DNN, SYCL-ML • TensorFlow, Eigen • Renesas, Imagination Technologies, ARM, Intel, Xilinx, Red Hat

Hand-port CUDA code to SYCL Use a CUDA-to-SYCL refactoring tool Create a combined C++, CUDA, SYCL • Guides provided on Codeplay website • Currently in-development by Institute for source file Software (the Cevelop developers) • Used in Eigen and TensorFlow

6 © 2019 Codeplay Software Ltd ReSYCLator At the Institute for Software we have been developing plug-ins for Eclipse CDT for almost a decade. Some of this work has been contributed back to Eclipse (e.g. the majority of refactorings and the underlying infrastructure for writing refactorings), others like Linticator are available commercially, but the majority is open source (e.g. Metriculator) or free to use (e.g. CUTE). Over time, as more and more plug-ins were created, it became tedious to discover and install them one by one. So we decided to bundle them all into one single download, including the latest release of Eclipse CDT.

7 © 2019 Codeplay Software Ltd CUDA to SYCL SYCL version CUDA version #include #include #include #include #include // function to add the elements of two arrays // Kernel function to add the elements of two arrays void add(cl::sycl::nd_item<1> item, int n, float *x, float *y) __global__ void add(int n, float *x, float *y) { { int index = item.get_local(0); int index = threadIdx.x; int stride = item.get_local_range(0); for (int i = index; i < n; i += stride) int stride = blockDim.x; y[i] y[= ix[i]] = x[+ iy[i];] + y[i]; forfor ((intintii= =index; index; i x(N); cl::sycl::buffer y(N); cudaMallocManaged(&x, N*sizeof(float)); cudaMallocManaged(&y, N*sizeof(float)); gpu_queue.submit([&](cl::sycl::handler &cgh) { auto aX = x.get_access(cgh); auto aY = y.get_access(cgh); add <<<1, 256 >>>(N, x, y); cgh.parallel_for( cl::sycl::nd_range<1>(cl::sycl::range<1>(256), cl::sycl::range<1>(256)), [=](cl::sycl::nd_item<1> it) { cudaDeviceSynchronize(); add(it, N, aX.get_pointer(), aY.get_pointer()); }); });

#include #include #include #ifdef __SYCL__ // function to add the elements of two arrays cl::sycl::buffer x(N); #ifdef __SYCL__ cl::sycl::buffer y(N); void add(cl::sycl::nd_item<1> item, int n, float *x, float *y) #else { cudaMallocManaged(&x, N*sizeof(float)); int index = item.get_local(0); cudaMallocManaged(&y, N*sizeof(float)); #includeint stride = item.get_local_range(0); #endif #include#else #include__global__add(int n, float *x, float *y) { #ifdef __SYCL__ #includeint index gpu_queue.submit([&](cl::sycl::handler &cgh) { int stride = blockDim.x; auto aX = x.get_access(cgh); #endifparallelize ([&] () { auto aY = y.get_access(cgh); forfor ((intint ii == index;index; ii << n;n; ii +=+= stride)stride) cgh.parallel_for( y[y[i]i]y[ y[=i i]x[x[i]] =i= ] x[ x[+i i]y[i];y[] +i+ ]; y[y[ii];]; cl::sycl::nd_range<1>(cl::sycl::range<1>(256), } } }} cl::sycl::range<1>(256)), [=](cl::sycl::nd_item<1> it) { add(it, N, aX.get_pointer(), aY.get_pointer()); }); }); #else add <<<1, 256 >>>(N, x, y); #endif

parallelize.h

#include #include #include ✓ Separation of device-specific details into a header file // function to add the elements of two arrays #ifdef __SYCL__ void add(cl::sycl::nd_item<1> item, int n, float *x, float *y) { int index = item.get_local(0); int stride = item.get_local_range(0); ✓ Core AI code in source files without device-specific code #else __global__ void add(int n, float *x, float *y) { int index = threadIdx.x; int stride = blockDim.x; #endif ✓ “Separation of Concerns” }

#ifdef __SYCL__ cl::sycl::buffer x(N); cl::sycl::buffer y(N); #else ✓ Standard C++ software engineering approach cudaMallocManaged(&x, N*sizeof(float)); cudaMallocManaged(&y, N*sizeof(float)); #endif

#ifdef __SYCL__ gpu_queue.submit([&](cl::sycl::handler &cgh) { #include auto aX = x.get_access(cgh); auto aY = y.get_access(cgh); #include cgh.parallel_for( cl::sycl::nd_range<1>(cl::sycl::range<1>(256), #include cl::sycl::range<1>(256)), [=](cl::sycl::nd_item<1> it) { add(it, N, aX.get_pointer(), aY.get_pointer()); }); }); #include #else add <<<1, 256 >>>(N, x, y); #endif parallelize ([&] () { for (int i = index; i < n; i += stride) y[i] = x[i] + y[i]; }

450,000

400,000

350,000 higher is better) is higher

- 300,000

250,000

200,000

150,000

100,000

50,000

Performance (Mbytes/sec (Mbytes/sec Performance Babelstream hipSYCL benchmarks 0 Add Copy Mul Triad Add Copy Mul Triad Add Copy Mul Triad Add Copy Mul Triad furyx ivybridge knl s9150 CUDA HIP KOKKOS OpenACC OpenCL RAJA Reference Reference OpenMP SYCL Source: http://uob-hpc.github.io/BabelStream

11 © 2019 Codeplay Software Ltd SYCL-BLAS • Open-source BLAS library written using SYCL (like cuBLAS) • Similar C++ style to NVIDIA’s new CUTLASS tech https://github.com/NVIDIA/cutlass • Collaboration with BLAS experts at Universitat Jaume I, Castellion • Supports: Intel GPU, PowerVR, R-Car, AMD GPU, ARM Mali, Intel CPU, +others • Supports kernel fusion for high performance on complex algorithms • Reduces memory bandwidth • Adapts to be optimal on different GPUs and AI processors • Use C++ templates to let the same source code adapt to get best performance • Adapts the algorithm to different memory sizes, types, vector widths • Adapts the algorithm to different tensor shapes and sizes

https://github.com/codeplaysoftware/sycl-blas

with fusion 10 • SCAL, AXPY, NRM2, DOT, SCAL2OP, IAMAX, IAMIN, SCAL3OP, AXPY3OP, ROT, SWAP, COPY, ASUM, ICAMAX, ICAMIN GFLOPS/s 5 0 • BLAS2: Matrix * vector asum axpy dot iamax iamin nrm2 scal • With and without transpose on the matrix clblast - cpu syclblas - cpu syclblas - gpu • GEMV, GER, TRMV, SYMV, SYR, SYR • BLAS3: Matrix multiplies • GEMM (Matrix multiply): With and without transpose on both arguments • Batch GEMM

• All operations can be templated by base datatype

13 © 2019 Codeplay Software Ltd SYCL-BLAS Empirical Optimization • Some processors have hand-optimized libraries integrated into SYCL-BLAS for specific operations • Can tune the algorithms per-processor: • With and without on-chip local memory, work-group size, double buffering, avoid bank conflicts in on-chip memory, cache-line-size , top-level-tile-size, block-level-tile-size

• We continuously run a huge number of benchmarks with different optimization parameters for different platforms and different matrix sizes & shapes

• _axpy(ex, size, alpha, gpu_vX, strd, gpu_vY, strd); These operations are all _asum(ex, size, gpu_vY, strd, gpu_vR); fused on device, so _dot(ex, size, gpu_vX, strd, gpu_vY, strd, gpu_vS); intermediate data is _nrm2(ex, size, gpu_vY, strd, gpu_vT); _iamax(ex, size, gpu_vY, strd, gpu_vImax); stored on-chip to reduce _iamin(ex, size, gpu_vY, strd, gpu_vImin); off-chip memory _rot(ex, size, gpu_vX, strd, gpu_vY, strd, _cos, _sin); bandwidth _dot(ex, size, gpu_vX, strd, gpu_vY, strd, gpu_vU); _swap(ex, size, gpu_vX, strd, gpu_vY, strd);

15 © 2019 Codeplay Software Ltd SYCL-DNN • Open-source SYCL convolution library • Replacement for cuDNN • Adapts the algorithm according to the GPU (or DSP, FPGA…) • Different accelerators benefit from different algorithms for different sizes & shape of convolution • Algorithms: direct, direct-tiled, im2col, Winograd, GEMM (for 1x1) • Supports both training and inference • 2D convolutions, max&average pooling, 2D depth-wise convolutions, Relu, Tanh https://github.com/codeplaysoftware/SYCL-DNN

SYCL-DNN runs 5 algorithms across a range of common convolution sizes and shapes on each processor it supports and “learns” the best algorithm for each size and shape

SYCL-DNN runs a range of different tile sizes and shapes on each algorithm and on each processor to “learn” the best tile sizes & shapes for each convolution

17 © 2019 Codeplay Software Ltd Eigen • Eigen is a C++ linear algebra library • Open-source • Developed by INRIA and Google • Used in TensorFlow • Supports kernel-fusion to reduce bandwidth • Supports: CPU vectorization, CUDA and SYCL

Accelerated Operations in TensorFlow 500

400

300

200 supported

100

Number of operations ofoperations Number 0 CUDA SYCL XLA Accelerated Control Device-specific

OpenCL 1.2 & SYCL 1.2.1 SYCL 2019 • Enables AI & HPC • Support for a larger range software on large range of of AI processors processor types • More developer-friendly • Open conformance tests features

Grow the ecosystem: ISO C++ more open-source, • Bring heterogeneous fill the gaps in the acceleration to the full standards ISO C++ standard • MISRA C++ for • HPC, Deep learning, automotive safe AI Computer vision