DOCSLIB.ORG

L18 SIMD (6Up).Pdf

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
L18 SIMD (6Up).Pdf 10/25/17 CS 61C: 61C Survey Great Ideas in Computer Architecture It would be nice to have a review lecture Lecture 18: every once in a while, actually showinG us Parallel Processing – SIMD how things fit in the biGGer picture Krste Asanović & Randy Katz http://inst.eecs.berkeley.edu/~cs61c CS 61c Lecture 18: Parallel ProcessinG - SIMD 2 Agenda 61C Topics so far … • What we learned: • 61C – the big picture 1. Binary numbers • Parallel processinG 2. C 3. Pointers • SinGle instruction, multiple data 4. Assembly lanGuaGe 5. Datapath architecture • SIMD matrix multiplication 6. PipelininG • Amdahl’s law 7. Caches 8. Performance evaluation • Loop unrollinG 9. FloatinG point • Memory access strateGy - blocking • What does this buy us? − Promise: execution speed • And in Conclusion, … − Let’s check! CS 61c Lecture 18: Parallel ProcessinG - SIMD 3 CS 61c Lecture 18: Parallel ProcessinG - SIMD 4 Reference ProBlem Application Example: Deep Learning • Matrix multiplication • ImaGe classification (cats …) −Basic operation in many engineering, data, and imaGinG processing tasks • Pick “best” vacation photos −ImaGe filterinG, noise reduction, … −Many closely related operations • Machine translation § E.G. stereo vision (project 4) • Clean up accent • dgemm −double-precision floating-point matrix • FinGerprint verification multiplication • Automatic Game playinG CS 61c Lecture 18: Parallel ProcessinG - SIMD 5 CS 61c Lecture 18: Parallel ProcessinG - SIMD 6 1 10/25/17 Matrices Matrix Multiplication � B • Square matrix of dimension NxN � 0 N-1 C = A*B 0 Cij = Σk (Aik * Bkj) � � � A C N-1 � CS 61c Lecture 18: Parallel ProcessinG - SIMD 7 CS 61c 8 Reference: Python C • Matrix multiplication in Python • c = a * b • a, b, c are N x N matrices N Python [Mflops] • 1 MFLOP = 1 Million floatinG- 32 5.4 point operations per second 160 5.5 (fadd, fmul) 480 5.4 3 960 5.3 • dgemm(N …) takes 2*N flops CS 61c Lecture 18: Parallel ProcessinG - SIMD 9 CS 61c Lecture 18: Parallel ProcessinG - SIMD 10 Timing Program Execution C versus Python N C [GFLOPS] Python [GFLOPS] 32 1.30 0.0054 160 1.30 0.0055 240x 480 1.32 0.0054 ! 960 0.91 0.0053 Which class gives you this kind of power? We could stop here … but why? Let’s do better! CS 61c Lecture 18: Parallel ProcessinG - SIMD 11 CS 61c Lecture 18: Parallel ProcessinG - SIMD 12 2 10/25/17 Agenda Why Parallel Processing? • 61C – the biG picture • CPU Clock Rates are no longer increasing • Parallel processing − Technical & economic challenGes § Advanced coolinG technology too expensive or impractical for most • SinGle instruction, multiple data applications § EnerGy costs are prohibitive • SIMD matrix multiplication • Amdahl’s law • Parallel processinG is only path to hiGher speed • Loop unrollinG − Compare airlines: • Memory access strateGy - blocking § Maximum speed limited by speed of sound and economics § Use more and larGer airplanes to increase throuGhput • And in Conclusion, … § And smaller seats … CS 61c Lecture 18: Parallel ProcessinG - SIMD 13 CS 61c Lecture 18: Parallel ProcessinG - SIMD 14 New-School Machine Structures Using Parallelism for Performance (It’s a bit more complicated!) Software Hardware • Two basic ways: • Parallel Requests AssiGned to computer Warehouse Smart − e.G., Search “Katz” Scale Phone MultiproGramminG Computer § run multiple independent proGrams in parallel • Parallel Threads Harness AssiGned to core Parallelism & § “Easy” e.G., Lookup, Ads Achieve High Computer • Parallel Instructions Performance Core Core −Parallel computinG >1 instruction @ one time … § run one proGram faster e.G., 5 pipelined instructions Memory (Cache) Input/Output § “Hard” • Parallel Data Core >1 data item @ one time Today’s Instruction Unit(s) Functional • We’ll focus on parallel computinG in the next e.G., Add of 4 pairs of words Unit(s) • Hardware descriptions Lecture A0+B0 A1+B1 A2+B2 A3+B3 few lectures All Gates @ one time Cache Memory • ProGramminG LanGuaGes LoGic Gates CS 61c Lecture 18: Parallel ProcessinG - SIMD 15 16 Single-Instruction/Single-Data Stream Single-Instruction/Multiple-Data Stream (SISD) (SIMD or “sim-dee”) • Sequential computer that • SIMD computer exploits exploits no parallelism in multiple data streams aGainst either the instruction or data a single instruction stream to streams. Examples of SISD operations that may be architecture are traditional naturally parallelized, e.G., uniprocessor machines Intel SIMD instruction ҆E.G. our trusted RISC-V pipeline extensions or NVIDIA Graphics ProcessinG Unit Processing Unit (GPU) This is what we did up to now in 61C Today’s topic. CS 61c Lecture 18: Parallel ProcessinG - SIMD 17 CS 61c Lecture 18: Parallel ProcessinG - SIMD 18 3 10/25/17 Multiple-Instruction/Multiple-Data Streams Multiple-Instruction/Single-Data Stream (MIMD or “mim-dee”) (MISD) • Multiple autonomous Instruction Pool processors simultaneously executinG different • Multiple-Instruction, PU instructions on different data. Single-Data stream • MIMD architectures include PU computer that exploits multicore and Warehouse-Scale multiple instruction PU Computers streams aGainst a sinGle Data Pool data stream. PU • Historical siGnificance Topic of Lecture 19 and beyond. This has few applications. Not covered in 61C. CS 61c Lecture 18: Parallel ProcessinG - SIMD 19 CS 61c Lecture 18: Parallel ProcessinG - SIMD 20 Flynn* Taxonomy, 1966 Agenda • 61C – the biG picture • Parallel processinG • Single instruction, multiple data • SIMD and MIMD are currently the most common • SIMD matrix multiplication parallelism in architectures – usually both in same • system! Amdahl’s law • Most common parallel processinG proGramminG style: • Loop unrollinG SinGle ProGram Multiple Data (“SPMD”) • Memory access strateGy - blocking − SinGle proGram that runs on all processors of a MIMD − Cross-processor execution coordination using • And in Conclusion, … synchronization primitives CS 61c Lecture 18: Parallel ProcessinG - SIMD 21 CS 61c Lecture 18: Parallel ProcessinG - SIMD 22 SIMD – “Single Instruction Multiple Data” SIMD Applications & Implementations • Applications − Scientific computinG § Matlab, NumPy − Graphics and video processinG § Photoshop, … − BiG Data § Deep learninG − GaminG − … • Implementations − x86 − ARM − RISC-V vector extensions CS 61c Lecture 18: Parallel ProcessinG - SIMD 23 CS 61c Lecture 18: Parallel ProcessinG - SIMD 24 4 10/25/17 First SIMD Extensions: MIT Lincoln LaBs TX-2, 1957 x86 SIMD Evolution • New instructions • New, wider, more registers • More parallelism http://svmoore.pbworks.com/w/file/fetch/70583970/VectorOps.pdf CS 61c 25 CS 61c Lecture 18: Parallel ProcessinG - SIMD 26 Laptop CPU Specs SIMD Registers $ sysctl -a | grep cpu hw.physicalcpu: 2 hw.logicalcpu: 4 machdep.cpu.brand_string: Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz machdep.cpu.features: FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE SSE3 PCLMULQDQ DTES64 MON DSCPL VMX EST TM2 SSSE3 FMA CX16 TPR PDCM SSE4.1 SSE4.2 x2APIC MOVBE POPCNT AES PCID XSAVE OSXSAVE SEGLIM64 TSCTMR AVX1.0 RDRAND F16C machdep.cpu.leaf7_features: SMEP ERMS RDWRFSGS TSC_THREAD_OFFSET BMI1 AVX2 BMI2 INVPCID SMAP RDSEED ADX IPT FPU_CSDS CS 61c Lecture 18: Parallel ProcessinG - SIMD 27 CS 61c Lecture 18: Parallel ProcessinG - SIMD 28 SIMD Data Types SIMD Vector Mode (Now also AVX-512 available) CS 61c Lecture 18: Parallel ProcessinG - SIMD 29 CS 61c Lecture 18: Parallel ProcessinG - SIMD 30 5 10/25/17 Agenda ProBlem • 61C – the biG picture • Today’s compilers (larGely) do not Generate SIMD • Parallel processinG code • SinGle instruction, multiple data • Back to assembly … • SIMD matrix multiplication • x86 (not usinG RISC-V as no vector hardware yet) • Amdahl’s law −Over 1000 instructions to learn … • Loop unrollinG −Green Book • Memory access strateGy - blocking • Can we use the compiler to Generate all non-SIMD • And in Conclusion, … instructions? CS 61c Lecture 18: Parallel ProcessinG - SIMD 31 CS 61c Lecture 18: Parallel ProcessinG - SIMD 32 x86 Intrinsics AVX Data Types Intrinsics AVX Code Nomenclature Intrinsics: Direct access to reGisters & assembly from C ReGister CS 61c Lecture 18: Parallel ProcessinG - SIMD 33 CS 61c Lecture 18: Parallel ProcessinG - SIMD 34 x86 SIMD “Intrinsics” Raw DouBle-Precision Throughput Characteristic Value assembly instruction CPU i7-5557U Clock rate (sustained) 3.1 GHz Instructions per clock (mul_pd) 2 4 parallel multiplies Parallel multiplies per instruction 4 Peak double FLOPS 24.8 GFLOPS 2 instructions per clock cycle (CPI = 0.5) https://www.karlrupp.net/2013/06/cpu-gpu-and-mic-hardware-characteristics-over-time/ Actual performance is lower because of overhead https://software.intel.com/sites/landinGpaGe/IntrinsicsGuide/ CS 61c Lecture 18: Parallel ProcessinG - SIMD 35 CS 61c Lecture 18: Parallel ProcessinG - SIMD 36 6 10/25/17 Vectorized Matrix Multiplication “Vectorized” dgemm for i …; i+=4 � for j ... Inner Loop: � � � i += 4 CS 61c 37 CS 61c Lecture 18: Parallel ProcessinG - SIMD 38 Performance Agenda • 61C – the biG picture Gflops N • Parallel processinG scalar avx • SinGle instruction, multiple data 32 1.30 4.56 • 160 1.30 5.47 SIMD matrix multiplication 480 1.32 5.27 • Amdahl’s law 960 0.91 3.64 • Loop unrollinG • Memory access strateGy - blocking • 4x faster • But still << theoretical 25 GFLOPS! • And in Conclusion, … CS 61c Lecture 18: Parallel ProcessinG - SIMD 39 CS 61c Lecture 18: Parallel ProcessinG - SIMD 40 A trip to LA Amdahl’s Law • Get enhancement E for your new PC Commercial airline: − E.G. floating-point rocket
Load more

Recommended publications
  • Balanced Multithreading: Increasing Throughput Via a Low Cost Multithreading Hierarchy
    Balanced Multithreading: Increasing Throughput via a Low Cost Multithreading Hierarchy Eric Tune Rakesh Kumar Dean M. Tullsen Brad Calder Computer Science and Engineering Department University of California at San Diego {etune,rakumar,tullsen,calder}@cs.ucsd.edu Abstract ibility of SMT comes at a cost. The register file and rename tables must be enlarged to accommodate the architectural reg- A simultaneous multithreading (SMT) processor can issue isters of the additional threads. This in turn can increase the instructions from several threads every cycle, allowing it to clock cycle time and/or the depth of the pipeline. effectively hide various instruction latencies; this effect in- Coarse-grained multithreading (CGMT) [1, 21, 26] is a creases with the number of simultaneous contexts supported. more restrictive model where the processor can only execute However, each added context on an SMT processor incurs a instructions from one thread at a time, but where it can switch cost in complexity, which may lead to an increase in pipeline to a new thread after a short delay. This makes CGMT suited length or a decrease in the maximum clock rate. This pa- for hiding longer delays. Soon, general-purpose micropro- per presents new designs for multithreaded processors which cessors will be experiencing delays to main memory of 500 combine a conservative SMT implementation with a coarse- or more cycles. This means that a context switch in response grained multithreading capability. By presenting more virtual to a memory access can take tens of cycles and still provide contexts to the operating system and user than are supported a considerable performance benefit.
    [Show full text]
  • Amdahl's Law Threading, Openmp
    Computer Science 61C Spring 2018 Wawrzynek and Weaver Amdahl's Law Threading, OpenMP 1 Big Idea: Amdahl’s (Heartbreaking) Law Computer Science 61C Spring 2018 Wawrzynek and Weaver • Speedup due to enhancement E is Exec time w/o E Speedup w/ E = ---------------------- Exec time w/ E • Suppose that enhancement E accelerates a fraction F (F <1) of the task by a factor S (S>1) and the remainder of the task is unaffected Execution Time w/ E = Execution Time w/o E × [ (1-F) + F/S] Speedup w/ E = 1 / [ (1-F) + F/S ] 2 Big Idea: Amdahl’s Law Computer Science 61C Spring 2018 Wawrzynek and Weaver Speedup = 1 (1 - F) + F Non-speed-up part S Speed-up part Example: the execution time of half of the program can be accelerated by a factor of 2. What is the program speed-up overall? 1 1 = = 1.33 0.5 + 0.5 0.5 + 0.25 2 3 Example #1: Amdahl’s Law Speedup w/ E = 1 / [ (1-F) + F/S ] Computer Science 61C Spring 2018 Wawrzynek and Weaver • Consider an enhancement which runs 20 times faster but which is only usable 25% of the time Speedup w/ E = 1/(.75 + .25/20) = 1.31 • What if its usable only 15% of the time? Speedup w/ E = 1/(.85 + .15/20) = 1.17 • Amdahl’s Law tells us that to achieve linear speedup with 100 processors, none of the original computation can be scalar! • To get a speedup of 90 from 100 processors, the percentage of the original program that could be scalar would have to be 0.1% or less Speedup w/ E = 1/(.001 + .999/100) = 90.99 4 Amdahl’s Law If the portion of Computer Science 61C Spring 2018 the program that Wawrzynek and Weaver can be parallelized is small, then the speedup is limited The non-parallel portion limits the performance 5 Strong and Weak Scaling Computer Science 61C Spring 2018 Wawrzynek and Weaver • To get good speedup on a parallel processor while keeping the problem size fixed is harder than getting good speedup by increasing the size of the problem.
    [Show full text]
  • Parallel Generation of Image Layers Constructed by Edge Detection
    International Journal of Applied Engineering Research ISSN 0973-4562 Volume 13, Number 10 (2018) pp. 7323-7332 © Research India Publications. http://www.ripublication.com Speed Up Improvement of Parallel Image Layers Generation Constructed By Edge Detection Using Message Passing Interface Alaa Ismail Elnashar Faculty of Science, Computer Science Department, Minia University, Egypt. Associate professor, Department of Computer Science, College of Computers and Information Technology, Taif University, Saudi Arabia. Abstract A. Elnashar [29] introduced two parallel techniques, NASHT1 and NASHT2 that apply edge detection to produce a set of Several image processing techniques require intensive layers for an input image. The proposed techniques generate computations that consume CPU time to achieve their results. an arbitrary number of image layers in a single parallel run Accelerating these techniques is a desired goal. Parallelization instead of generating a unique layer as in traditional case; this can be used to save the execution time of such techniques. In helps in selecting the layers with high quality edges among this paper, we propose a parallel technique that employs both the generated ones. Each presented parallel technique has two point to point and collective communication patterns to versions based on point to point communication "Send" and improve the speedup of two parallel edge detection techniques collective communication "Bcast" functions. The presented that generate multiple image layers using Message Passing techniques achieved notable relative speedup compared with Interface, MPI. In addition, the performance of the proposed that of sequential ones. technique is compared with that of the two concerned ones. In this paper, we introduce a speed up improvement for both Keywords: Image Processing, Image Segmentation, Parallel NASHT1 and NASHT2.
    [Show full text]
  • CS650 Computer Architecture Lecture 10 Introduction to Multiprocessors
    NJIT Computer Science Dept CS650 Computer Architecture CS650 Computer Architecture Lecture 10 Introduction to Multiprocessors and PC Clustering Andrew Sohn Computer Science Department New Jersey Institute of Technology Lecture 10: Intro to Multiprocessors/Clustering 1/15 12/7/2003 A. Sohn NJIT Computer Science Dept CS650 Computer Architecture Key Issues Run PhotoShop on 1 PC or N PCs Programmability • How to program a bunch of PCs viewed as a single logical machine. Performance Scalability - Speedup • Run PhotoShop on 1 PC (forget the specs of this PC) • Run PhotoShop on N PCs • Will it run faster on N PCs? Speedup = ? Lecture 10: Intro to Multiprocessors/Clustering 2/15 12/7/2003 A. Sohn NJIT Computer Science Dept CS650 Computer Architecture Types of Multiprocessors Key: Data and Instruction Single Instruction Single Data (SISD) • Intel processors, AMD processors Single Instruction Multiple Data (SIMD) • Array processor • Pentium MMX feature Multiple Instruction Single Data (MISD) • Systolic array • Special purpose machines Multiple Instruction Multiple Data (MIMD) • Typical multiprocessors (Sun, SGI, Cray,...) Single Program Multiple Data (SPMD) • Programming model Lecture 10: Intro to Multiprocessors/Clustering 3/15 12/7/2003 A. Sohn NJIT Computer Science Dept CS650 Computer Architecture Shared-Memory Multiprocessor Processor Prcessor Prcessor Interconnection network Main Memory Storage I/O Lecture 10: Intro to Multiprocessors/Clustering 4/15 12/7/2003 A. Sohn NJIT Computer Science Dept CS650 Computer Architecture Distributed-Memory Multiprocessor Processor Processor Processor IO/S MM IO/S MM IO/S MM Interconnection network IO/S MM IO/S MM IO/S MM Processor Processor Processor Lecture 10: Intro to Multiprocessors/Clustering 5/15 12/7/2003 A.
    [Show full text]
  • High-Performance Message Passing Over Generic Ethernet Hardware with Open-MX Brice Goglin
    High-Performance Message Passing over generic Ethernet Hardware with Open-MX Brice Goglin To cite this version: Brice Goglin. High-Performance Message Passing over generic Ethernet Hardware with Open-MX. Parallel Computing, Elsevier, 2011, 37 (2), pp.85-100. 10.1016/j.parco.2010.11.001. inria-00533058 HAL Id: inria-00533058 https://hal.inria.fr/inria-00533058 Submitted on 5 Nov 2010 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. High-Performance Message Passing over generic Ethernet Hardware with Open-MX Brice Goglin INRIA Bordeaux - Sud-Ouest – LaBRI 351 cours de la Lib´eration – F-33405 Talence – France Abstract In the last decade, cluster computing has become the most popular high-performance computing architec- ture. Although numerous technological innovations have been proposed to improve the interconnection of nodes, many clusters still rely on commodity Ethernet hardware to implement message passing within parallel applications. We present Open-MX, an open-source message passing stack over generic Ethernet. It offers the same abilities as the specialized Myrinet Express stack, without requiring dedicated support from the networking hardware. Open-MX works transparently in the most popular MPI implementations through its MX interface compatibility.
    [Show full text]
  • An Investigation of Symmetric Multi-Threading Parallelism for Scientific Applications
    An Investigation of Symmetric Multi-Threading Parallelism for Scientific Applications Installation and Performance Evaluation of ASCI sPPM Frank Castaneda [email protected] Nikola Vouk [email protected] North Carolina State University CSC 591c Spring 2003 May 2, 2003 Dr. Frank Mueller An Investigation of Symmetric Multi-Threading Parallelism for Scientific Applications Introduction Our project is to investigate the use of Symmetric Multi-Threading (SMT), or “Hyper-threading” in Intel parlance, applied to course-grain parallelism in large-scale distributed scientific applications. The processors provide the capability to run two streams of instruction simultaneously to fully utilize all available functional units in the CPU. We are investigating the speedup available when running two threads on a single processor that uses different functional units. The idea we propose is to utilize the hyper-thread for asynchronous communications activity to improve course-grain parallelism. The hypothesis is that there will be little contention for similar processor functional units when splitting the communications work from the computational work, thus allowing better parallelism and better exploiting the hyper-thread technology. We believe with minimal changes to the 2.5 Linux kernel we can achieve 25-50% speedup depending on the amount of communication by utilizing a hyper-thread aware scheduler and a custom communications API. Experiment Setup Software Setup Hardware Setup ?? Custom Linux Kernel 2.5.68 with ?? IBM xSeries 335 Single/Dual Processor Red Hat distribution 2.0 Ghz Xeon ?? Modification of Kernel Scheduler to run processes together ?? Custom Test Code In order to test our code we focus on the following scenarios: ?? A serial execution of communication / computation sections o Executing in serial is used to test the expected back-to-back run-time.
    [Show full text]
  • Computer Systems Architecture
    CS 352H: Computer Systems Architecture Topic 14: Multicores, Multiprocessors, and Clusters University of Texas at Austin CS352H - Computer Systems Architecture Fall 2009 Don Fussell Introduction Goal: connecting multiple computers to get higher performance Multiprocessors Scalability, availability, power efficiency Job-level (process-level) parallelism High throughput for independent jobs Parallel processing program Single program run on multiple processors Multicore microprocessors Chips with multiple processors (cores) University of Texas at Austin CS352H - Computer Systems Architecture Fall 2009 Don Fussell 2 Hardware and Software Hardware Serial: e.g., Pentium 4 Parallel: e.g., quad-core Xeon e5345 Software Sequential: e.g., matrix multiplication Concurrent: e.g., operating system Sequential/concurrent software can run on serial/parallel hardware Challenge: making effective use of parallel hardware University of Texas at Austin CS352H - Computer Systems Architecture Fall 2009 Don Fussell 3 What We’ve Already Covered §2.11: Parallelism and Instructions Synchronization §3.6: Parallelism and Computer Arithmetic Associativity §4.10: Parallelism and Advanced Instruction-Level Parallelism §5.8: Parallelism and Memory Hierarchies Cache Coherence §6.9: Parallelism and I/O: Redundant Arrays of Inexpensive Disks University of Texas at Austin CS352H - Computer Systems Architecture Fall 2009 Don Fussell 4 Parallel Programming Parallel software is the problem Need to get significant performance improvement Otherwise, just use a faster uniprocessor,
    [Show full text]
  • Computer Architecture: Parallel Processing Basics
    Computer Architecture: Parallel Processing Basics Onur Mutlu & Seth Copen Goldstein Carnegie Mellon University 9/9/13 Today What is Parallel Processing? Why? Kinds of Parallel Processing Multiprocessing and Multithreading Measuring success Speedup Amdhal’s Law Bottlenecks to parallelism 2 Concurrent Systems Embedded-Physical Distributed Sensor Claytronics Networks Concurrent Systems Embedded-Physical Distributed Sensor Claytronics Networks Geographically Distributed Power Internet Grid Concurrent Systems Embedded-Physical Distributed Sensor Claytronics Networks Geographically Distributed Power Internet Grid Cloud Computing EC2 Tashi PDL'09 © 2007-9 Goldstein5 Concurrent Systems Embedded-Physical Distributed Sensor Claytronics Networks Geographically Distributed Power Internet Grid Cloud Computing EC2 Tashi Parallel PDL'09 © 2007-9 Goldstein6 Concurrent Systems Physical Geographical Cloud Parallel Geophysical +++ ++ --- --- location Relative +++ +++ + - location Faults ++++ +++ ++++ -- Number of +++ +++ + - Processors + Network varies varies fixed fixed structure Network --- --- + + connectivity 7 Concurrent System Challenge: Programming The old joke: How long does it take to write a parallel program? One Graduate Student Year 8 Parallel Programming Again?? Increased demand (multicore) Increased scale (cloud) Improved compute/communicate Change in Application focus Irregular Recursive data structures PDL'09 © 2007-9 Goldstein9 Why Parallel Computers? Parallelism: Doing multiple things at a time Things: instructions,
    [Show full text]
  • Your Speedup Is a Megaflop!
    KOLUMNE 22 CHI MIA 47 (1993) Nr. II~ (lnnuor/Fcbru",) COMPUTATIONAL CHEMISTRY Column Editors: Prof. Dr. J. Weber, University of Geneva COLUMN Prof. Dr. H. Huber, University of Basel Dr. H. P. Weber, Sandoz AG, Basel Clzilllia 47 (/993) 22-23 ond instruction on the first number at the © Neue Sclz\i'eizerisclze Clzelllische Gesellschaft same time as the first worker performs the /SSN 0009-4293 first instruction on the second number, etc .. For the first 20 steps no product is leaving your assembly line (no result is Your Speedup is a Megaflop! leaving your vector-processor) but after that you get a product (result) in each step, 'I just bought a new personal compu- Another common measure is the mil- i.e. after a total of 120 steps your loop is ter, the LAST, with a speed of 50 MHz, lions of instructions performed per sec- processed! You gain a factor of about 17, whereas your old MADOS has only a 25 ond (Mips). Again this is not a very useful i.e. if your scalar processor performs I MHz frequency'. 'You are joking, the measure for number crunching as differ- Mflops, your vector-processor yields 17 LAST makes only 2 Mflops, whereas mine ent processors need a different number of Mflops, which would roughly con'espond makes 25 Mips.' 'Never mind, your old instructions to perform a useful calcula- to the performance advertized by the com- scalar processors will not do it for long, I tion, e.g. a multiplication. For example puter manufacturer. However, any scien- just read about a supercomputer of the workstations usually have risc-processors tific program has instructions other than Teraflop generation with a speedup of (reduced instruction set computer) which loops which do not vectorize and, there- over 100!' in average need slightly more instructions fore, the efficiency is in practice much to perform floating point operations than lower! Depending on your problem and on Not far from reality - perhaps virtual the 'classical' main-frame processors.
    [Show full text]
  • Chapter 4 Data-Level Parallelism in Vector, SIMD, and GPU Architectures
    Computer Architecture A Quantitative Approach, Fifth Edition Chapter 4 Data-Level Parallelism in Vector, SIMD, and GPU Architectures Copyright © 2012, Elsevier Inc. All rights reserved. 1 Contents 1. SIMD architecture 2. Vector architectures optimizations: Multiple Lanes, Vector Length Registers, Vector Mask Registers, Memory Banks, Stride, Scatter-Gather, 3. Programming Vector Architectures 4. SIMD extensions for media apps 5. GPUs – Graphical Processing Units 6. Fermi architecture innovations 7. Examples of loop-level parallelism 8. Fallacies Copyright © 2012, Elsevier Inc. All rights reserved. 2 Classes of Computers Classes Flynn’s Taxonomy SISD - Single instruction stream, single data stream SIMD - Single instruction stream, multiple data streams New: SIMT – Single Instruction Multiple Threads (for GPUs) MISD - Multiple instruction streams, single data stream No commercial implementation MIMD - Multiple instruction streams, multiple data streams Tightly-coupled MIMD Loosely-coupled MIMD Copyright © 2012, Elsevier Inc. All rights reserved. 3 Introduction Advantages of SIMD architectures 1. Can exploit significant data-level parallelism for: 1. matrix-oriented scientific computing 2. media-oriented image and sound processors 2. More energy efficient than MIMD 1. Only needs to fetch one instruction per multiple data operations, rather than one instr. per data op. 2. Makes SIMD attractive for personal mobile devices 3. Allows programmers to continue thinking sequentially SIMD/MIMD comparison. Potential speedup for SIMD twice that from MIMID! x86 processors expect two additional cores per chip per year SIMD width to double every four years Copyright © 2012, Elsevier Inc. All rights reserved. 4 Introduction SIMD parallelism SIMD architectures A. Vector architectures B. SIMD extensions for mobile systems and multimedia applications C.
    [Show full text]
  • Vector Processors
    VECTOR PROCESSORS Computer Science Department CS 566 – Fall 2012 1 Eman Aldakheel Ganesh Chandrasekaran Prof. Ajay Kshemkalyani OUTLINE What is Vector Processors Vector Processing & Parallel Processing Basic Vector Architecture Vector Instruction Vector Performance Advantages Disadvantages Applications Conclusion 2 VECTOR PROCESSORS A processor can operate on an entire vector in one instruction Work done automatically in parallel (simultaneously) The operand to the instructions are complete vectors instead of one element Reduce the fetch and decode bandwidth Data parallelism Tasks usually consist of: Large active data sets Poor locality Long run times 3 VECTOR PROCESSORS (CONT’D) Each result independent of previous result Long pipeline Compiler ensures no dependencies High clock rate Vector instructions access memory with known pattern Reduces branches and branch problems in pipelines Single vector instruction implies lots of work Example: for(i=0; i<n; i++) c(i) = a(i) + b(i); 4 5 VECTOR PROCESSORS (CONT’D) vadd // C code b[15]+=a[15] for(i=0;i<16; i++) b[i]+=a[i] b[14]+=a[14] b[13]+=a[13] b[12]+=a[12] // Vectorized code b[11]+=a[11] set vl,16 b[10]+=a[10] vload vr0,b b[9]+=a[9] vload vr1,a b[8]+=a[8] vadd vr0,vr0,vr1 b[7]+=a[7] vstore vr0,b b[6]+=a[6] b[5]+=a[5] b[4]+=a[4] Each vector instruction b[3]+=a[3] holds many units of b[2]+=a[2] independent operations b[1]+=a[1] 6 b[0]+=a[0] 1 Vector Lane VECTOR PROCESSORS (CONT’D) vadd // C code b[15]+=a[15] 16 Vector Lanes for(i=0;i<16; i++) b[14]+=a[14] b[i]+=a[i]
    [Show full text]
  • SIMD the Good, the Bad and the Ugly
    SIMD The Good, the Bad and the Ugly Viktor Leis Outline 1. Why does SIMD exist? 2. How to use SIMD? 3. When does SIMD help? 1 Why Does SIMD Exist? Hardware Trends • for decades, single-threaded performance doubled every 18-22 months • during the last decade, single-threaded performance has been stagnating • clock rates stopped growing due to heat/end of Dennard’s scaling • number of transistors is still growing, used for: • large caches • many cores (thread parallelism) • SIMD (data parallelism) 2 Number of Cores AMD Rome 64 48 AMD Naples 32 cores Skylake SP Broadwell EX Broadwell EP Haswell EP 16 Ivy Bridge EX Ivy Bridge EP Nehalem (Westmere EX) Nehalem (Beckton) Sandy Bridge EP Core (Kentsfield) Core (Lynnfield) NetBurst (Foster) NetBurst (Paxville) 1 2000 2004 2008 2012 2016 2020 year 3 Skylake CPU 4 Skylake Core 5 x86 SIMD History • 1997: MMX 64-bit (Pentium 1) • 1999: SSE 128-bit (Pentium 3) • 2011: AVX 256-bit float (Sandy Bridge) • 2013: AVX2 256-bit int (Haswell) • 2017: AVX-512 512 bit (Skylake Server) • instructions usually introduced by Intel • AMD typically followed quickly (except for AVX-512) 6 Single Instruction Multiple Data (SIMD) • data parallelism exposed by the CPU’s instruction set • CPU register holds multiple fixed-size values (e.g., 4×32-bit) • instructions (e.g., addition) are performed on two such registers (e.g., executing 4 additions in one instruction) 42 1 7 + 31 6 2 = 73 7 9 7 Registers • SSE: 16×128 bit (XMM0...XMM15) • AVX: 16×256 bit (YMM0...YMM15) • AVX-512: 32×512 bit (ZMM0...ZMM31) • XMM0 is lower half
    [Show full text]