DOCSLIB.ORG

From CUDA to Opencl: Towards a Performance-Portable Solution for Multi-Platform GPU Programming

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
From CUDA to Opencl: Towards a Performance-Portable Solution for Multi-Platform GPU Programming From CUDA to OpenCL: Towards a Performance-portable Solution for Multi-platform GPU Programming✩,✩✩ Peng Dua, Rick Webera, Piotr Luszczeka, Stanimire Tomova, Gregory Petersona, Jack Dongarraa,b aUniversity of Tennessee Knoxville bUniversity of Manchester Abstract In this work, we evaluate OpenCL as a programming tool for developing performance-portable applications for GPGPU. While the Khronos group developed OpenCL with programming portability in mind, performance is not necessarily portable. OpenCL has required performance-impacting initializations that do not exist in other languages such as CUDA. Understanding these implications allows us to provide a single library with decent performance on a variety of platforms. We choose triangular solver (TRSM) and matrix multiplication (GEMM) as representative level 3 BLAS routines to implement in OpenCL. We profile TRSM to get the time distribution of the OpenCL runtime system. We then provide tuned GEMM kernels for both the NVIDIA Tesla C2050 and ATI Radeon 5870, the latest GPUs offered by both companies. We explore the benefits of using the texture cache, the performance ramifications of copying data into images, discrepancies in the OpenCL and CUDA compilers’ optimizations, and other issues that affect the performance. Experimental results show that nearly 50% of peak performance can be obtained in GEMM on both GPUs in OpenCL. We also show that the performance of these kernels is not highly portable. Finally, we propose using auto-tuning to better explore these kernels’ parameter space using search heuristics. Keywords: hardware accelerators, portability, auto-tuning 1. Introduction of NVIDIA’s CUDA [7, 8] (Compute Unified Device Archi- tecture), ushered a new era of improved performance for many People associated Graphics Processing Units (GPUs) with applications as programming GPUs became simpler: archaic fast image rendering until the turn of the century. This is when terms such as texels, fragments, and pixels were superseded the science community turned their attention to the hardware with threads, vector processing, data caches and shared mem- predominantly discussed in the computer gaming circles. One ory. of the first attempts of non-graphical computations on a GPU Further changes occuring in ATI’s and NVIDIA’s offerings was a matrix-matrix multiply [1]. In 2001, low-end graphics made GPU acceleration even more pertinent to the scientific cards had no floating-point support; floating-point color buffers community. ATI’s FireStream and NVIDIA’s Fermi architec- did not arrive until 2003 [2]. For the gaming industry, sup- ture added support for Fused Multiply-Add (FMA): a more ac- port for floating-point meant more realistic game-play; rich ad- curate version of the former MAD (Multiply-Add) instruction. vanced lighting effects no longer suffered from banding effects With only a single rounding step, this new instructions bring common in older generations of hardware that only allowed a GPUs even closer to compliance with the IEEE 754 standard for single byte per color channel. For the scientific community, floating-point arithmetic. Additionally, reworking of the cache the addition of floating point meant that overflow associated hierarchy helped with some of the performance issues of the with fixed-point arithmetic was no longer a problem. Many re- past. Finally, Error Correction Codes (ECC) are now used to search publications thoroughly analyzed the new breed of GPU protect the GPU device’s memory as its capacity grows to the hardware using languages borrowed from the graphics com- point of being vulnerable to errors induced by nature, such as munity [3, 4, 5]. It goes without saying that these computa- cosmic radiation. tional advances would not be possible if it weren’t for the pro- In our project called Matrix Algebra on GPU and Multicore grammable shaders that broke the rigidity of the fixed graphics Architectures [9] (MAGMA), we mainly focus on dense matrix pipeline. LU factorization with partial pivoting on a GPU was routines for numerical linear algebra, similar to those available one of the first common computational kernels that ran faster in LAPACK [10]. While CUDA is only available for NVIDIA than an optimized CPU implementation [6]. The introduction GPUs, there are other existing frameworks that allow platform- independent programming for GPUs: ✩This work was supported by the SCALE-IT fellowship through grant num- 1. DirectCompute from Microsoft, ber OR11907-001. 2. OpenGL Shading Language (GLSL), and ✩✩This work was supported by NVIDIA, Microsoft, the U.S. National Sci- ence Foundation, and the U.S. Department of Energy. 3. OpenCL. Preprint submitted to Parallel Computing August 31, 2010 DirectCompute allows access to graphics cards from mul- is harder, even impossible, and naturally, many authors claim tiple vendors. However, it is specific to Microsoft Windows that OpenCL does not provide performance portability. This, and therefore it is not portable between host Operating Sys- along with the fact that GPUs are quickly evolving in complex- tems (OS). ity, has made tuning numerical libraries for them challenging. The OpenGL Shading language [11] is portable across both One approach (that we explore) to systematically resolve these GPU hardware and the host OS. However, it is specifically geared issues is the use of auto-tuning, a technique that in the context of towards programming new graphics effects – GLSL does not OpenCL would involve collecting and generating multiple ker- have a scientific focus. nel versions, implementing the same algorithm optimized for OpenCL [12] has been designed for general purpose com- different architectures, and heuristically selecting the best per- puting on GPUs (GPGPU). It is an open standard maintained forming one. Auto-tuning has been used intensively on CPUs by the Khronos group with the backing of major graphics hard- in the past to address these challenges to automatically gener- ware vendors as well as large computer industry vendors inter- ate near optimal numerical libraries, e.g., ATLAS [19, 20] and ested in off-loading computations to GPUs. As such, there ex- PHiPAC [21] used it to generate highly optimized BLAS. Work ist working OpenCL implementations for graphics cards and, on auto-tuning CUDA kernels for NVIDIA GPUs [22, 23] has multi-core processors; OpenCL offers portability across GPU shown that the technique is a very practical solution to easily hardware, OS software, and multicore processors. In this work, port existing algorithmic solutions on quickly evolving GPU BLAS routines are used to evaluate OpenCL’s usefulness in cre- architectures and to substantially speed up even highly tuned ating a portable high performance GPU numerical linear alge- hand-written kernels. bra library. In [24], the authors1 examined performance portability in The rest of the paper is organized as follows: Section 2 talks OpenCL. In their study, they compared CUDA and OpenCL about related work. Section 3 details OpenCL’s programming implementations of a Monte Carlo Chemistry application run- considerations. This involves a comparison between CUDA ning on an NVIDIA GTX285. They also compared the same and OpenCL as programming language for GPU, and the profil- application written in ATI’s now defunct Brook+ to an OpenCL ing analysis of the run-time of each component in OpenCL pro- version on a Firestream 9170 and Radeon 4870 respectively. Fi- gram. Sections 4 and 5 give an evaluation to both NVIDIA and nally they compared OpenCL to a C++ implementation running ATI GPU platforms by implementing fast GEMM and analyz- on multi-core Intel processors. The paper showed that while ing performance. Section 6 discusses cross-platform issues and OpenCL does provide code portability, it doesn’t necessarily section 7 lays out the basic structure of an auto-tuning system provide performance portability. Furthermore, they showed that for cross-platform GPU math library design. Section 8 shows platform-specific languages often, but not always, outperformed the performance result and section 9 concludes the paper. In OpenCL. the text we use the terms ”ATI card” and Radeon 5870; ”Fermi card” and Tesla C2050 interchangeably. 3. OpenCL as A Programming Tool 2. Related Work To evaluate OpenCL as a programming tool for implement- ing high performance linear algebra routines, we pick the tri- Synthetic benchmarking of GPUs was used extensively to angular solver (TRSM) routine from BLAS and profile each gain understanding of the graphics accelerators when the tech- component of the program: environment setup, program com- nical details of the hardware remains an industrial secret [13]. pilation, kernel extraction, and execution. In the context of scientific applications, such benchmarking ef- TRSM solves the linear equation Ax = b where A is an upper forts lead to algorithms that provide significant performance or lower triangular matrix and b is a known matrix of solutions. improvements [14]. Its implementation involves a blocking algorithm in which the A performance-oriented study of NVIDIA’s PTX (Parallel diagonal triangular blocks are inverted (TRTRI) in parallel fol- Thread eXecution) architecture [15] was done in the context of lowed by a series of matrix multiplications (GEMM). Porting the Ocelot project [16]. Code transformations of CUDA ker- these routines from CUDA to OpenCL requires some transla- nels were done by the MCUDA [17] project. The transforma- tion. tions enabled CUDA code to run efficiently on multicore CPUs. CUDA and OpenCL have many conceptual similarities but The inverse operation – porting multicore code to GPUs – was they diverge on terminology. Table 1 shows the correspond- difficult [16]. ing terms in both frameworks while 1 highlights differences in Work on optimizing CUDA implementations of basic lin- the CUDA and OpenCL software stacks. Similarly, ATI and ear algebra kernels has demonstrated the importance on “how NVIDIA GPUs have analogous platform definitions as shown sensitive the performance of a GPU is to the formulation of in Table 2. Table 3 shows the platform details of two different your kernel” [18] and that an enormous amount of well thought NVIDIA GPUs and one GPU from ATI/AMD.
Load more

Recommended publications
  • Other Apis What’S Wrong with Openmp?
    Threaded Programming Other APIs What’s wrong with OpenMP? • OpenMP is designed for programs where you want a fixed number of threads, and you always want the threads to be consuming CPU cycles. – cannot arbitrarily start/stop threads – cannot put threads to sleep and wake them up later • OpenMP is good for programs where each thread is doing (more-or-less) the same thing. • Although OpenMP supports C++, it’s not especially OO friendly – though it is gradually getting better. • OpenMP doesn’t support other popular base languages – e.g. Java, Python What’s wrong with OpenMP? (cont.) Can do this Can do this Can’t do this Threaded programming APIs • Essential features – a way to create threads – a way to wait for a thread to finish its work – a mechanism to support thread private data – some basic synchronisation methods – at least a mutex lock, or atomic operations • Optional features – support for tasks – more synchronisation methods – e.g. condition variables, barriers,... – higher levels of abstraction – e.g. parallel loops, reductions What are the alternatives? • POSIX threads • C++ threads • Intel TBB • Cilk • OpenCL • Java (not an exhaustive list!) POSIX threads • POSIX threads (or Pthreads) is a standard library for shared memory programming without directives. – Part of the ANSI/IEEE 1003.1 standard (1996) • Interface is a C library – no standard Fortran interface – can be used with C++, but not OO friendly • Widely available – even for Windows – typically installed as part of OS – code is pretty portable • Lots of low-level control over behaviour of threads • Lacks a proper memory consistency model Thread forking #include <pthread.h> int pthread_create( pthread_t *thread, const pthread_attr_t *attr, void*(*start_routine, void*), void *arg) • Creates a new thread: – first argument returns a pointer to a thread descriptor.
    [Show full text]
  • Opencl on Shared Memory Multicore Cpus
    OpenCL on shared memory multicore CPUs Akhtar Ali, Usman Dastgeer and Christoph Kessler Book Chapter N.B.: When citing this work, cite the original article. Part of: Proceedings of the 5th Workshop on MULTIPROG -2012. E. Ayguade, B. Gaster, L. Howes, P. Stenström, O. Unsal (eds), 2012. Copyright: HiPEAC Network of Excellence Available at: Linköping University Electronic Press http://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-93951 Published in: Proc. Fifth Workshop on Programmability Issues for Multi-Core Computers (MULTIPROG-2012) at HiPEAC-2012 conference, Paris, France, Jan. 2012. OpenCL for programming shared memory multicore CPUs Akhtar Ali, Usman Dastgeer, and Christoph Kessler PELAB, Dept. of Computer and Information Science, Linköping University, Sweden [email protected] {usman.dastgeer,christoph.kessler}@liu.se Abstract. Shared memory multicore processor technology is pervasive in mainstream computing. This new architecture challenges programmers to write code that scales over these many cores to exploit the full compu- tational power of these machines. OpenMP and Intel Threading Build- ing Blocks (TBB) are two of the popular frameworks used to program these architectures. Recently, OpenCL has been defined as a standard by Khronos group which focuses on programming a possibly heteroge- neous set of processors with many cores such as CPU cores, GPUs, DSP processors. In this work, we evaluate the effectiveness of OpenCL for programming multicore CPUs in a comparative case study with OpenMP and Intel TBB for five benchmark applications: matrix multiply, LU decomposi- tion, 2D image convolution, Pi value approximation and image histogram generation. The evaluation includes the effect of compiler optimizations for different frameworks, OpenCL performance on different vendors’ plat- forms and the performance gap between CPU-specific and GPU-specific OpenCL algorithms for execution on a modern GPU.
    [Show full text]
  • Heterogeneous Task Scheduling for Accelerated Openmp
    Heterogeneous Task Scheduling for Accelerated OpenMP ? ? Thomas R. W. Scogland Barry Rountree† Wu-chun Feng Bronis R. de Supinski† ? Department of Computer Science, Virginia Tech, Blacksburg, VA 24060 USA † Center for Applied Scientific Computing, Lawrence Livermore National Laboratory, Livermore, CA 94551 USA [email protected] [email protected] [email protected] [email protected] Abstract—Heterogeneous systems with CPUs and computa- currently requires a programmer either to program in at tional accelerators such as GPUs, FPGAs or the upcoming least two different parallel programming models, or to use Intel MIC are becoming mainstream. In these systems, peak one of the two that support both GPUs and CPUs. Multiple performance includes the performance of not just the CPUs but also all available accelerators. In spite of this fact, the models however require code replication, and maintaining majority of programming models for heterogeneous computing two completely distinct implementations of a computational focus on only one of these. With the development of Accelerated kernel is a difficult and error-prone proposition. That leaves OpenMP for GPUs, both from PGI and Cray, we have a clear us with using either OpenCL or accelerated OpenMP to path to extend traditional OpenMP applications incrementally complete the task. to use GPUs. The extensions are geared toward switching from CPU parallelism to GPU parallelism. However they OpenCL’s greatest strength lies in its broad hardware do not preserve the former while adding the latter. Thus support. In a way, though, that is also its greatest weak- computational potential is wasted since either the CPU cores ness. To enable one to program this disparate hardware or the GPU cores are left idle.
    [Show full text]
  • AMD Accelerated Parallel Processing Opencl Programming Guide
    AMD Accelerated Parallel Processing OpenCL Programming Guide November 2013 rev2.7 © 2013 Advanced Micro Devices, Inc. All rights reserved. AMD, the AMD Arrow logo, AMD Accelerated Parallel Processing, the AMD Accelerated Parallel Processing logo, ATI, the ATI logo, Radeon, FireStream, FirePro, Catalyst, and combinations thereof are trade- marks of Advanced Micro Devices, Inc. Microsoft, Visual Studio, Windows, and Windows Vista are registered trademarks of Microsoft Corporation in the U.S. and/or other jurisdic- tions. Other names are for informational purposes only and may be trademarks of their respective owners. OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission by Khronos. The contents of this document are provided in connection with Advanced Micro Devices, Inc. (“AMD”) products. AMD makes no representations or warranties with respect to the accuracy or completeness of the contents of this publication and reserves the right to make changes to specifications and product descriptions at any time without notice. The information contained herein may be of a preliminary or advance nature and is subject to change without notice. No license, whether express, implied, arising by estoppel or other- wise, to any intellectual property rights is granted by this publication. Except as set forth in AMD’s Standard Terms and Conditions of Sale, AMD assumes no liability whatsoever, and disclaims any express or implied warranty, relating to its products including, but not limited to, the implied warranty of merchantability, fitness for a particular purpose, or infringement of any intellectual property right. AMD’s products are not designed, intended, authorized or warranted for use as compo- nents in systems intended for surgical implant into the body, or in other applications intended to support or sustain life, or in any other application in which the failure of AMD’s product could create a situation where personal injury, death, or severe property or envi- ronmental damage may occur.
    [Show full text]
  • Implementing FPGA Design with the Opencl Standard
    Implementing FPGA Design with the OpenCL Standard WP-01173-3.0 White Paper Utilizing the Khronos Group’s OpenCL™ standard on an FPGA may offer significantly higher performance and at much lower power than is available today from hardware architectures such as CPUs, graphics processing units (GPUs), and digital signal processing (DSP) units. In addition, an FPGA-based heterogeneous system (CPU + FPGA) using the OpenCL standard has a significant time-to-market advantage compared to traditional FPGA development using lower level hardware description languages (HDLs) such as Verilog or VHDL. 1 OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission by Khronos. Introduction The initial era of programmable technologies contained two different extremes of programmability. As illustrated in Figure 1, one extreme was represented by single core CPU and digital signal processing (DSP) units. These devices were programmable using software consisting of a list of instructions to be executed. These instructions were created in a manner that was conceptually sequential to the programmer, although an advanced processor could reorder instructions to extract instruction-level parallelism from these sequential programs at run time. In contrast, the other extreme of programmable technology was represented by the FPGA. These devices are programmed by creating configurable hardware circuits, which execute completely in parallel. A designer using an FPGA is essentially creating a massively- fine-grained parallel application. For many years, these extremes coexisted with each type of programmability being applied to different application domains. However, recent trends in technology scaling have favored technologies that are both programmable and parallel. Figure 1.
    [Show full text]
  • Threads Chapter 4
    Threads Chapter 4 Reading: 4.1,4.4, 4.5 1 Process Characteristics ● Unit of resource ownership - process is allocated: ■ a virtual address space to hold the process image ■ control of some resources (files, I/O devices...) ● Unit of dispatching - process is an execution path through one or more programs ■ execution may be interleaved with other process ■ the process has an execution state and a dispatching priority 2 Process Characteristics ● These two characteristics are treated independently by some recent OS ● The unit of dispatching is usually referred to a thread or a lightweight process ● The unit of resource ownership is usually referred to as a process or task 3 Multithreading vs. Single threading ● Multithreading: when the OS supports multiple threads of execution within a single process ● Single threading: when the OS does not recognize the concept of thread ● MS-DOS support a single user process and a single thread ● UNIX supports multiple user processes but only supports one thread per process ● Solaris /NT supports multiple threads 4 Threads and Processes 5 Processes Vs Threads ● Have a virtual address space which holds the process image ■ Process: an address space, an execution context ■ Protected access to processors, other processes, files, and I/O Class threadex{ resources Public static void main(String arg[]){ ■ Context switch between Int x=0; processes expensive My_thread t1= new my_thread(x); t1.start(); ● Threads of a process execute in Thr_wait(); a single address space System.out.println(x) ■ Global variables are
    [Show full text]
  • Lecture 10: Introduction to Openmp (Part 2)
    Lecture 10: Introduction to OpenMP (Part 2) 1 Performance Issues I • C/C++ stores matrices in row-major fashion. • Loop interchanges may increase cache locality { … #pragma omp parallel for for(i=0;i< N; i++) { for(j=0;j< M; j++) { A[i][j] =B[i][j] + C[i][j]; } } } • Parallelize outer-most loop 2 Performance Issues II • Move synchronization points outwards. The inner loop is parallelized. • In each iteration step of the outer loop, a parallel region is created. This causes parallelization overhead. { … for(i=0;i< N; i++) { #pragma omp parallel for for(j=0;j< M; j++) { A[i][j] =B[i][j] + C[i][j]; } } } 3 Performance Issues III • Avoid parallel overhead at low iteration counts { … #pragma omp parallel for if(M > 800) for(j=0;j< M; j++) { aa[j] =alpha*bb[j] + cc[j]; } } 4 C++: Random Access Iterators Loops • Parallelization of random access iterator loops is supported void iterator_example(){ std::vector vec(23); std::vector::iterator it; #pragma omp parallel for default(none) shared(vec) for(it=vec.begin(); it< vec.end(); it++) { // do work with it // } } 5 Conditional Compilation • Keep sequential and parallel programs as a single source code #if def _OPENMP #include “omp.h” #endif Main() { #ifdef _OPENMP omp_set_num_threads(3); #endif for(i=0;i< N; i++) { #pragma omp parallel for for(j=0;j< M; j++) { A[i][j] =B[i][j] + C[i][j]; } } } 6 Be Careful with Data Dependences • Whenever a statement in a program reads or writes a memory location and another statement reads or writes the same memory location, and at least one of the two statements writes the location, then there is a data dependence on that memory location between the two statements.
    [Show full text]
  • The Problem with Threads
    The Problem with Threads Edward A. Lee Electrical Engineering and Computer Sciences University of California at Berkeley Technical Report No. UCB/EECS-2006-1 http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.html January 10, 2006 Copyright © 2006, by the author(s). All rights reserved. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission. Acknowledgement This work was supported in part by the Center for Hybrid and Embedded Software Systems (CHESS) at UC Berkeley, which receives support from the National Science Foundation (NSF award No. CCR-0225610), the State of California Micro Program, and the following companies: Agilent, DGIST, General Motors, Hewlett Packard, Infineon, Microsoft, and Toyota. The Problem with Threads Edward A. Lee Professor, Chair of EE, Associate Chair of EECS EECS Department University of California at Berkeley Berkeley, CA 94720, U.S.A. [email protected] January 10, 2006 Abstract Threads are a seemingly straightforward adaptation of the dominant sequential model of computation to concurrent systems. Languages require little or no syntactic changes to sup- port threads, and operating systems and architectures have evolved to efficiently support them. Many technologists are pushing for increased use of multithreading in software in order to take advantage of the predicted increases in parallelism in computer architectures.
    [Show full text]
  • Opencl SYCL 2.2 Specification
    SYCLTM Provisional Specification SYCL integrates OpenCL devices with modern C++ using a single source design Version 2.2 Revision Date: – 2016/02/15 Khronos OpenCL Working Group — SYCL subgroup Editor: Maria Rovatsou Copyright 2011-2016 The Khronos Group Inc. All Rights Reserved Contents 1 Introduction 13 2 SYCL Architecture 15 2.1 Overview . 15 2.2 The SYCL Platform and Device Model . 15 2.2.1 Platform Mixed Version Support . 16 2.3 SYCL Execution Model . 17 2.3.1 Execution Model: Queues, Command Groups and Contexts . 18 2.3.2 Execution Model: Command Queues . 19 2.3.3 Execution Model: Mapping work-items onto an nd range . 21 2.3.4 Execution Model: Execution of kernel-instances . 22 2.3.5 Execution Model: Hierarchical Parallelism . 24 2.3.6 Execution Model: Device-side enqueue . 25 2.3.7 Execution Model: Synchronization . 25 2.4 Memory Model . 26 2.4.1 Access to memory . 27 2.4.2 Memory consistency . 28 2.4.3 Atomic operations . 28 2.5 The SYCL programming model . 28 2.5.1 Basic data parallel kernels . 30 2.5.2 Work-group data parallel kernels . 30 2.5.3 Hierarchical data parallel kernels . 30 2.5.4 Kernels that are not launched over parallel instances . 31 2.5.5 Synchronization . 31 2.5.6 Error handling . 32 2.5.7 Scheduling of kernels and data movement . 32 2.5.8 Managing object lifetimes . 34 2.5.9 Device discovery and selection . 35 2.5.10 Interfacing with OpenCL . 35 2.6 Anatomy of a SYCL application .
    [Show full text]
  • An Introduction to Gpus, CUDA and Opencl
    An Introduction to GPUs, CUDA and OpenCL Bryan Catanzaro, NVIDIA Research Overview ¡ Heterogeneous parallel computing ¡ The CUDA and OpenCL programming models ¡ Writing efficient CUDA code ¡ Thrust: making CUDA C++ productive 2/54 Heterogeneous Parallel Computing Latency-Optimized Throughput- CPU Optimized GPU Fast Serial Scalable Parallel Processing Processing 3/54 Why do we need heterogeneity? ¡ Why not just use latency optimized processors? § Once you decide to go parallel, why not go all the way § And reap more benefits ¡ For many applications, throughput optimized processors are more efficient: faster and use less power § Advantages can be fairly significant 4/54 Why Heterogeneity? ¡ Different goals produce different designs § Throughput optimized: assume work load is highly parallel § Latency optimized: assume work load is mostly sequential ¡ To minimize latency eXperienced by 1 thread: § lots of big on-chip caches § sophisticated control ¡ To maXimize throughput of all threads: § multithreading can hide latency … so skip the big caches § simpler control, cost amortized over ALUs via SIMD 5/54 Latency vs. Throughput Specificaons Westmere-EP Fermi (Tesla C2050) 6 cores, 2 issue, 14 SMs, 2 issue, 16 Processing Elements 4 way SIMD way SIMD @3.46 GHz @1.15 GHz 6 cores, 2 threads, 4 14 SMs, 48 SIMD Resident Strands/ way SIMD: vectors, 32 way Westmere-EP (32nm) Threads (max) SIMD: 48 strands 21504 threads SP GFLOP/s 166 1030 Memory Bandwidth 32 GB/s 144 GB/s Register File ~6 kB 1.75 MB Local Store/L1 Cache 192 kB 896 kB L2 Cache 1.5 MB 0.75 MB
    [Show full text]
  • Opencl in Scientific High Performance Computing
    OpenCL in Scientific High Performance Computing —The Good, the Bad, and the Ugly Matthias Noack [email protected] Zuse Institute Berlin Distributed Algorithms and Supercomputing 2017-05-18, IWOCL 2017 https://doi.org/10.1145/3078155.3078170 1 / 28 Systems: • 2× Cray XC40 (#118 and #150 in top500, year 4/5 in lifetime) with Xeon CPUs • 80-node Xeon Phi (KNL) Cray XC40 test-and-development system • 2× 32-node Infiniband cluster with Nvidia K40 • 2 test-and-development systems with AMD FirePro W8100 What I do: • computer science research (methods) • development of HPC codes • evaluation of upcoming technologies • consulting/training with system users Context ZIB hosts part of the HLRN (Northern German Supercomputing Alliance) facilities. 2 / 28 What I do: • computer science research (methods) • development of HPC codes • evaluation of upcoming technologies • consulting/training with system users Context ZIB hosts part of the HLRN (Northern German Supercomputing Alliance) facilities. Systems: • 2× Cray XC40 (#118 and #150 in top500, year 4/5 in lifetime) with Xeon CPUs • 80-node Xeon Phi (KNL) Cray XC40 test-and-development system • 2× 32-node Infiniband cluster with Nvidia K40 • 2 test-and-development systems with AMD FirePro W8100 2 / 28 Context ZIB hosts part of the HLRN (Northern German Supercomputing Alliance) facilities. Systems: • 2× Cray XC40 (#118 and #150 in top500, year 4/5 in lifetime) with Xeon CPUs • 80-node Xeon Phi (KNL) Cray XC40 test-and-development system • 2× 32-node Infiniband cluster with Nvidia K40 • 2 test-and-development systems with AMD FirePro W8100 What I do: • computer science research (methods) • development of HPC codes • evaluation of upcoming technologies • consulting/training with system users 2 / 28 • intra-node parallelism dominated by OpenMP (e.g.
    [Show full text]
  • The Continuing Renaissance in Parallel Programming Languages
    The continuing renaissance in parallel programming languages Simon McIntosh-Smith University of Bristol Microelectronics Research Group [email protected] © Simon McIntosh-Smith 1 Didn’t parallel computing use to be a niche? © Simon McIntosh-Smith 2 When I were a lad… © Simon McIntosh-Smith 3 But now parallelism is mainstream Samsung Exynos 5 Octa: • 4 fast ARM cores and 4 energy efficient ARM cores • Includes OpenCL programmable GPU from Imagination 4 HPC scaling to millions of cores Tianhe-2 at NUDT in China 33.86 PetaFLOPS (33.86x1015), 16,000 nodes Each node has 2 CPUs and 3 Xeon Phis 3.12 million cores, $390M, 17.6 MW, 720m2 © Simon McIntosh-Smith 5 A renaissance in parallel programming Metal C++11 OpenMP OpenCL Erlang Unified Parallel C Fortress XC Go Cilk HMPP CHARM++ CUDA Co-Array Fortran Chapel Linda X10 MPI Pthreads C++ AMP © Simon McIntosh-Smith 6 Groupings of || languages Partitioned Global Address Space GPU languages: (PGAS): • OpenCL • Fortress • CUDA • X10 • HMPP • Chapel • Metal • Co-array Fortran • Unified Parallel C Object oriented: • C++ AMP CSP: XC • CHARM++ Message passing: MPI Multi-threaded: • Cilk Shared memory: OpenMP • Go • C++11 © Simon McIntosh-Smith 7 Emerging GPGPU standards • OpenCL, DirectCompute, C++ AMP, … • Also OpenMP 4.0, OpenACC, CUDA… © Simon McIntosh-Smith 8 Apple's Metal • A "ground up" parallel programming language for GPUs • Designed for compute and graphics • Potential to replace OpenGL compute shaders, OpenCL/GL interop etc. • Close to the "metal" • Low overheads • "Shading" language based
    [Show full text]