DOCSLIB.ORG

Parallel Architectures Impact of Parallel Architectures

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Parallel Architectures Impact of Parallel Architectures An Introduction to Parallel Architectures Impact of Parallel Architectures • From cell phones to supercomputers • In regular CPUs as well as GPUs Parallel HW Processing • Why? • Which types? • Which novel issues? Why Multicores? The SPECint performance of the hottest chip grew by 52% per year from 1986 to 2002, and then grew only 20% in the next three years (about 6% per year). [from Patterson & Hennessy] Diminishing returns from uniprocessor designs Power Wall [from Patterson & Hennessy] • The design goal for the late 1990’s and early 2000’s was to drive the clock rate up. • by adding more transistors to a smaller chip. • Unfortunately, this increased the power dissipation of the CPU chip beyond the capacity of inexpensive cooling techniques The Multicore Approach Multiple cores on the same chip – Simpler – Slower – Less power demanding Transition to Multicore Intel Xeon (18cores) Sequential App Performance 5/4/2018 Spring 2011 ‐‐ Lecture #15 7 Flynn Taxonomy of parallel computers Data streams Single Parallel Single SISD SIMD Instruction Streams Multiple MISD MIMD 5/4/2018 Spring 2011 ‐‐ Lecture #13 8 Alternative Kinds of Parallelism: Single Instruction/Single Data Stream • Single Instruction, Single Data stream (SISD) – Sequential computer that exploits no parallelism in either the instruction or data Processing Unit streams. Examples of SISD architecture are traditional uniprocessor machines 5/4/2018 Spring 2011 ‐‐ Lecture #13 9 Alternative Kinds of Parallelism: Multiple Instruction/Single Data Stream • Multiple Instruction, Single Data streams (MISD) – Computer that exploits multiple instruction streams against a single data stream for data operations that can be naturally parallelized. For example, certain kinds of array processors. – No longer commonly encountered, mainly of 5/4/2018 Spring 2011 ‐‐ Lecture #13historical interest only 10 Alternative Kinds of Parallelism: Single Instruction/Multiple Data Stream • Single Instruction, Multiple Data streams (SIMD) – Computer that exploits multiple data streams against a single instruction stream to operations that may be naturally parallelized, e.g., SIMD instruction extensions or Graphics Processing Unit (GPU) 5/4/2018 Spring 2011 ‐‐ Lecture #13 11 Alternative Kinds of Parallelism: Multiple Instruction/Multiple Data Streams • Multiple Instruction, Multiple Data streams (MIMD) – Multiple autonomous processors simultaneously executing different instructions on different data. – MIMD architectures include multicore and Warehouse Scale Computers (datacenters) 5/4/2018 Spring 2011 ‐‐ Lecture #13 12 Flynn Taxonomy of parallel computers Data streams Single Parallel Single SISD: Intel Pentium 4 SIMD: SSE x86, ARM neon, GPGPUs, … Instruction Streams Multiple MISD: No examples today MIMD: SMP (Intel, ARM, …) • From 2011, SIMD and MIMD most common parallel computers • Most common parallel processing programming style: Single Program Multiple Data (“SPMD”) – Single program that runs on all processors of an MIMD – Cross‐processor execution coordination through conditional expressions (thread parallelism) • SIMD (aka hw‐level data parallelism): specialized function units, for handling lock‐step calculations involving arrays – Scientific computing, signal processing, multimedia (audio/video processing) 5/4/2018 Spring 2011 ‐‐ Lecture #13 13 SIMD Architectures • Data parallelism: executing one operation on multiple data streams – Single control unit – Multiple datapaths (processing elements – PEs) running in parallel • PEs are interconnected and exchange/share data as directed by the control unit • Each PE performs the same operation on its own local data • Example to provide context: – Multiplying a coefficient vector by a data vector (e.g., in filtering) y[i] := c[i] x[i], 0 i < n 5/4/2018 Spring 2011 ‐‐ Lecture #13 Slide 14 “Advanced Digital Media Boost” • To improve performance, SIMD instructions – Fetch one instruction, do the work of multiple instructions 5/4/2018 Spring 2011 ‐‐ Lecture #13 15 Example: SIMD Array Processing for each f in array f = sqrt(f) for each f in array { load f to the floating-point register SISD calculate the square root write the result from the register to memory } for each 4 members in array { load 4 members to the SIMD register SIMD calculate 4 square roots in one operation write the result from the register to memory } 5/4/2018 Spring 2011 ‐‐ Lecture #13 16 MIMD Architectures • Multicore architectures • At least 2 processors interconnected via a communication network – abstractions (HW/SW interface) – organizational structure to realize abstraction efficiently 5/4/2018 ACA H.Corporaal 17 MIMD Architectures • Thread‐Level parallelism – Have multiple program counters – Targeted for tightly‐coupled shared‐memory multiprocessors • For n processors, need n threads • Amount of computation assigned to each thread = grain size – Threads can be used for data‐level parallelism, but the overheads may outweigh the benefit Copyright © 2012, Elsevier Inc. All rights reserved. MIMD Architectures • And what about memory? • How is data to be accessed by multiple cores? • How to design accordingly the memory system? • How do traditional solutions from uniprocessor systems affect multicores? Copyright © 2012, Elsevier Inc. All rights reserved. Recall: The Memory Gap Memory CPU 100000 10000 1000 100 Performance 10 1 H&P 0 5 Fig. 5.2 985 990 198 1 1 1995 2000 200 • Bottom‐line: memory access is increasingly expensive and CA must devise new ways of hiding this cost The memory wall • How do multicores attach the memory wall? • Same issue as uniprocessor systems – By building on‐chip cache hierarchies Memory Hierarchy 5/4/2018 Spring 2011 ‐‐ Lecture #1 22 Communication models: Shared Memory Shared Memory (read, write) (read, write) Process P1 Process P2 • Coherence problem • Memory consistency issue • Synchronization problem 5/4/2018 ACA H.Corporaal 23 Communication models: Shared memory • Shared address space • Communication primitives: – load, store, atomic swap Two varieties: • Physically shared => Symmetric Multi- Processors (SMP) – usually combined with local caching • Physically distributed => Distributed Shared Memory (DSM) 5/4/2018 ACA H.Corporaal 24 SMP: Symmetric Multi-Processor • Memory: centralized with uniform access time (UMA) and bus interconnect, I/O • Examples: Sun Enterprise 6000, SGI Challenge, Intel can be 1 bus,Processor N Processor Processor Processor busses, or any network One or One or One or One or more cache more cache more cache more cache levels levels levels levels Main memory I/O System 5/4/2018 ACA H.Corporaal 25 The memory wall • How do multicores attach the memory wall? • Same issue as uniprocessor systems – By building on‐chip cache hierarchies • What novel issues arise? – Coherence – Consistency – Scalability COMPUTING SYSTEMS PERFORMANCE Chapter 1 — Computer Abstractions and Technology — 27 Response Time and Throughput Response time How long it takes to do a task Throughput Total work done per unit time e.g., tasks/transactions/… per hour How are response time and throughput affected by Replacing the processor with a faster version? Adding more processors? We’ll focus on response time for now… Chapter 1 — Computer Abstractions and Technology — 28 Relative Performance Define Performance = 1/Execution Time “X is n time faster than Y” PerformanceX PerformanceY Execution time Y Execution time X n Example: time taken to run a program 10s on A, 15s on B Execution TimeB / Execution TimeA = 15s / 10s = 1.5 So A is 1.5 times faster than B Chapter 1 — Computer Abstractions and Technology — 29 Measuring Execution Time Elapsed time Total response time, including all aspects Processing, I/O, OS overhead, idle time Determines system performance CPU time Time spent processing a given job Discounts I/O time, other jobs’ shares Comprises user CPU time and system CPU time Different programs are affected differently by CPU and system performance Chapter 1 — Computer Abstractions and Technology — 30 CPU Clocking Operation of digital hardware governed by a constant-rate clock Clock period Clock (cycles) Data transfer and computation Update state Clock period: duration of a clock cycle –12 e.g., 250ps = 0.25ns = 250×10 s Clock frequency (rate): cycles per second 9 e.g., 4.0GHz = 4000MHz = 4.0×10 Hz Chapter 1 — Computer Abstractions and Technology — 31 CPU Time CPU Time CPU Clock CyclesClock Cycle Time CPU Clock Cycles Clock Rate Performance improved by Reducing number of clock cycles Increasing clock rate Hardware designer must often trade off clock rate against cycle count Chapter 1 — Computer Abstractions and Technology — 32 CPU Time Example Computer A: 2GHz clock, 10s CPU time Designing Computer B Aim for 6s CPU time Can do faster clock, but causes 1.2 × clock cycles How fast must Computer B clock be? Clock CyclesB 1.2Clock CyclesA Clock RateB CPU TimeB 6s Clock CyclesA CPU Time A Clock RateA 10s2GHz 20109 1.220109 24109 Clock Rate 4GHz B 6s 6s Chapter 1 — Computer Abstractions and Technology — 33 Instruction Count and CPI Clock Cycles Instruction Count Cycles per Instruction CPU Time Instruction Count CPIClock Cycle Time Instruction Count CPI Clock Rate Instruction Count for a program Determined by program, ISA and compiler Average cycles per instruction Determined by CPU hardware If different instructions have different CPI Average CPI affected by instruction mix Chapter 1 — Computer Abstractions and Technology — 34 CPI Example Computer A: Cycle Time = 250ps, CPI = 2.0 Computer B: Cycle Time
Load more

Recommended publications
  • 2.5 Classification of Parallel Computers
    52 // Architectures 2.5 Classification of Parallel Computers 2.5 Classification of Parallel Computers 2.5.1 Granularity In parallel computing, granularity means the amount of computation in relation to communication or synchronisation Periods of computation are typically separated from periods of communication by synchronization events. • fine level (same operations with different data) ◦ vector processors ◦ instruction level parallelism ◦ fine-grain parallelism: – Relatively small amounts of computational work are done between communication events – Low computation to communication ratio – Facilitates load balancing 53 // Architectures 2.5 Classification of Parallel Computers – Implies high communication overhead and less opportunity for per- formance enhancement – If granularity is too fine it is possible that the overhead required for communications and synchronization between tasks takes longer than the computation. • operation level (different operations simultaneously) • problem level (independent subtasks) ◦ coarse-grain parallelism: – Relatively large amounts of computational work are done between communication/synchronization events – High computation to communication ratio – Implies more opportunity for performance increase – Harder to load balance efficiently 54 // Architectures 2.5 Classification of Parallel Computers 2.5.2 Hardware: Pipelining (was used in supercomputers, e.g. Cray-1) In N elements in pipeline and for 8 element L clock cycles =) for calculation it would take L + N cycles; without pipeline L ∗ N cycles Example of good code for pipelineing: §doi =1 ,k ¤ z ( i ) =x ( i ) +y ( i ) end do ¦ 55 // Architectures 2.5 Classification of Parallel Computers Vector processors, fast vector operations (operations on arrays). Previous example good also for vector processor (vector addition) , but, e.g. recursion – hard to optimise for vector processors Example: IntelMMX – simple vector processor.
    [Show full text]
  • Computer Hardware Architecture Lecture 4
    Computer Hardware Architecture Lecture 4 Manfred Liebmann Technische Universit¨atM¨unchen Chair of Optimal Control Center for Mathematical Sciences, M17 [email protected] November 10, 2015 Manfred Liebmann November 10, 2015 Reading List • Pacheco - An Introduction to Parallel Programming (Chapter 1 - 2) { Introduction to computer hardware architecture from the parallel programming angle • Hennessy-Patterson - Computer Architecture - A Quantitative Approach { Reference book for computer hardware architecture All books are available on the Moodle platform! Computer Hardware Architecture 1 Manfred Liebmann November 10, 2015 UMA Architecture Figure 1: A uniform memory access (UMA) multicore system Access times to main memory is the same for all cores in the system! Computer Hardware Architecture 2 Manfred Liebmann November 10, 2015 NUMA Architecture Figure 2: A nonuniform memory access (UMA) multicore system Access times to main memory differs form core to core depending on the proximity of the main memory. This architecture is often used in dual and quad socket servers, due to improved memory bandwidth. Computer Hardware Architecture 3 Manfred Liebmann November 10, 2015 Cache Coherence Figure 3: A shared memory system with two cores and two caches What happens if the same data element z1 is manipulated in two different caches? The hardware enforces cache coherence, i.e. consistency between the caches. Expensive! Computer Hardware Architecture 4 Manfred Liebmann November 10, 2015 False Sharing The cache coherence protocol works on the granularity of a cache line. If two threads manipulate different element within a single cache line, the cache coherency protocol is activated to ensure consistency, even if every thread is only manipulating its own data.
    [Show full text]
  • Threading SIMD and MIMD in the Multicore Context the Ultrasparc T2
    Overview SIMD and MIMD in the Multicore Context Single Instruction Multiple Instruction ● (note: Tute 02 this Weds - handouts) ● Flynn’s Taxonomy Single Data SISD MISD ● multicore architecture concepts Multiple Data SIMD MIMD ● for SIMD, the control unit and processor state (registers) can be shared ■ hardware threading ■ SIMD vs MIMD in the multicore context ● however, SIMD is limited to data parallelism (through multiple ALUs) ■ ● T2: design features for multicore algorithms need a regular structure, e.g. dense linear algebra, graphics ■ SSE2, Altivec, Cell SPE (128-bit registers); e.g. 4×32-bit add ■ system on a chip Rx: x x x x ■ 3 2 1 0 execution: (in-order) pipeline, instruction latency + ■ thread scheduling Ry: y3 y2 y1 y0 ■ caches: associativity, coherence, prefetch = ■ memory system: crossbar, memory controller Rz: z3 z2 z1 z0 (zi = xi + yi) ■ intermission ■ design requires massive effort; requires support from a commodity environment ■ speculation; power savings ■ massive parallelism (e.g. nVidia GPGPU) but memory is still a bottleneck ■ OpenSPARC ● multicore (CMT) is MIMD; hardware threading can be regarded as MIMD ● T2 performance (why the T2 is designed as it is) ■ higher hardware costs also includes larger shared resources (caches, TLBs) ● the Rock processor (slides by Andrew Over; ref: Tremblay, IEEE Micro 2009 ) needed ⇒ less parallelism than for SIMD COMP8320 Lecture 2: Multicore Architecture and the T2 2011 ◭◭◭ • ◮◮◮ × 1 COMP8320 Lecture 2: Multicore Architecture and the T2 2011 ◭◭◭ • ◮◮◮ × 3 Hardware (Multi)threading The UltraSPARC T2: System on a Chip ● recall concurrent execution on a single CPU: switch between threads (or ● OpenSparc Slide Cast Ch 5: p79–81,89 processes) requires the saving (in memory) of thread state (register values) ● aggressively multicore: 8 cores, each with 8-way hardware threading (64 virtual ■ motivation: utilize CPU better when thread stalled for I/O (6300 Lect O1, p9–10) CPUs) ■ what are the costs? do the same for smaller stalls? (e.g.
    [Show full text]
  • Computer Architecture: Parallel Processing Basics
    Computer Architecture: Parallel Processing Basics Onur Mutlu & Seth Copen Goldstein Carnegie Mellon University 9/9/13 Today What is Parallel Processing? Why? Kinds of Parallel Processing Multiprocessing and Multithreading Measuring success Speedup Amdhal’s Law Bottlenecks to parallelism 2 Concurrent Systems Embedded-Physical Distributed Sensor Claytronics Networks Concurrent Systems Embedded-Physical Distributed Sensor Claytronics Networks Geographically Distributed Power Internet Grid Concurrent Systems Embedded-Physical Distributed Sensor Claytronics Networks Geographically Distributed Power Internet Grid Cloud Computing EC2 Tashi PDL'09 © 2007-9 Goldstein5 Concurrent Systems Embedded-Physical Distributed Sensor Claytronics Networks Geographically Distributed Power Internet Grid Cloud Computing EC2 Tashi Parallel PDL'09 © 2007-9 Goldstein6 Concurrent Systems Physical Geographical Cloud Parallel Geophysical +++ ++ --- --- location Relative +++ +++ + - location Faults ++++ +++ ++++ -- Number of +++ +++ + - Processors + Network varies varies fixed fixed structure Network --- --- + + connectivity 7 Concurrent System Challenge: Programming The old joke: How long does it take to write a parallel program? One Graduate Student Year 8 Parallel Programming Again?? Increased demand (multicore) Increased scale (cloud) Improved compute/communicate Change in Application focus Irregular Recursive data structures PDL'09 © 2007-9 Goldstein9 Why Parallel Computers? Parallelism: Doing multiple things at a time Things: instructions,
    [Show full text]
  • Parallel Processing! 1! CSE 30321 – Lecture 23 – Introduction to Parallel Processing! 2! Suggested Readings! •! Readings! –! H&P: Chapter 7! •! (Over Next 2 Weeks)!
    CSE 30321 – Lecture 23 – Introduction to Parallel Processing! 1! CSE 30321 – Lecture 23 – Introduction to Parallel Processing! 2! Suggested Readings! •! Readings! –! H&P: Chapter 7! •! (Over next 2 weeks)! Lecture 23" Introduction to Parallel Processing! University of Notre Dame! University of Notre Dame! CSE 30321 – Lecture 23 – Introduction to Parallel Processing! 3! CSE 30321 – Lecture 23 – Introduction to Parallel Processing! 4! Processor components! Multicore processors and programming! Processor comparison! vs.! Goal: Explain and articulate why modern microprocessors now have more than one core andCSE how software 30321 must! adapt to accommodate the now prevalent multi- core approach to computing. " Introduction and Overview! Writing more ! efficient code! The right HW for the HLL code translation! right application! University of Notre Dame! University of Notre Dame! CSE 30321 – Lecture 23 – Introduction to Parallel Processing! CSE 30321 – Lecture 23 – Introduction to Parallel Processing! 6! Pipelining and “Parallelism”! ! Load! Mem! Reg! DM! Reg! ALU ! Instruction 1! Mem! Reg! DM! Reg! ALU ! Instruction 2! Mem! Reg! DM! Reg! ALU ! Instruction 3! Mem! Reg! DM! Reg! ALU ! Instruction 4! Mem! Reg! DM! Reg! ALU Time! Instructions execution overlaps (psuedo-parallel)" but instructions in program issued sequentially." University of Notre Dame! University of Notre Dame! CSE 30321 – Lecture 23 – Introduction to Parallel Processing! CSE 30321 – Lecture 23 – Introduction to Parallel Processing! Multiprocessing (Parallel) Machines! Flynn#s
    [Show full text]
  • Chapter 4 Data-Level Parallelism in Vector, SIMD, and GPU Architectures
    Computer Architecture A Quantitative Approach, Fifth Edition Chapter 4 Data-Level Parallelism in Vector, SIMD, and GPU Architectures Copyright © 2012, Elsevier Inc. All rights reserved. 1 Contents 1. SIMD architecture 2. Vector architectures optimizations: Multiple Lanes, Vector Length Registers, Vector Mask Registers, Memory Banks, Stride, Scatter-Gather, 3. Programming Vector Architectures 4. SIMD extensions for media apps 5. GPUs – Graphical Processing Units 6. Fermi architecture innovations 7. Examples of loop-level parallelism 8. Fallacies Copyright © 2012, Elsevier Inc. All rights reserved. 2 Classes of Computers Classes Flynn’s Taxonomy SISD - Single instruction stream, single data stream SIMD - Single instruction stream, multiple data streams New: SIMT – Single Instruction Multiple Threads (for GPUs) MISD - Multiple instruction streams, single data stream No commercial implementation MIMD - Multiple instruction streams, multiple data streams Tightly-coupled MIMD Loosely-coupled MIMD Copyright © 2012, Elsevier Inc. All rights reserved. 3 Introduction Advantages of SIMD architectures 1. Can exploit significant data-level parallelism for: 1. matrix-oriented scientific computing 2. media-oriented image and sound processors 2. More energy efficient than MIMD 1. Only needs to fetch one instruction per multiple data operations, rather than one instr. per data op. 2. Makes SIMD attractive for personal mobile devices 3. Allows programmers to continue thinking sequentially SIMD/MIMD comparison. Potential speedup for SIMD twice that from MIMID! x86 processors expect two additional cores per chip per year SIMD width to double every four years Copyright © 2012, Elsevier Inc. All rights reserved. 4 Introduction SIMD parallelism SIMD architectures A. Vector architectures B. SIMD extensions for mobile systems and multimedia applications C.
    [Show full text]
  • Instruction Level Parallelism Example
    Instruction Level Parallelism Example Is Jule peaty or weak-minded when highlighting some heckles foreground thenceforth? Homoerotic and commendatory Shelby still pinks his pronephros inly. Overneat Kermit never beams so quaveringly or fecundated any academicians effectively. Summary of parallelism create readable and as with a bit says if we currently being considered to resolve these two machine of a pretty cool explanation. Once plug, it book the parallel grammatical structure which creates a truly memorable phrase. In order to accomplish whereas, a hybrid approach is designed to whatever advantage of streaming SIMD instructions for each faction the awful that executes in parallel on independent cores. For the ILPA, there is one more type of instruction possible, which is the special instruction type for the dedicated hardware units. Advantages and high for example? Two which is already present data is, to process includes comprehensive career related services that instruction level parallelism example how many diverse influences on. Simple uses of parallelism create readable and understandable passages. Also note that a data dependent elements that has to be imported from another core in another processor is much higher than either of the previous two costs. Why the charge of the proton does not transfer to the neutron in the nuclei? The OPENMP code is implemented in advance way leaving each thread can climb up an element from first vector and compare after all the elements in you second vector and forth thread will appear able to execute simultaneously in parallel. To be ready to instruction level parallelism in this allows enormous reduction in memory.
    [Show full text]
  • Task Level Parallelism
    Task Level Parallelism The topic of this chapter is thread-level parallelism. While, thread-level parallelism falls within the textbook’s classification of ILP and data parallelism. It also falls into a broader topic of parallel and distributed computing. In the next set of slides, I will attempt to place you in the context of this broader computation space that is called task level parallelism. Of course a proper treatment of parallel computing or distributed computing is worthy of an entire semester (or two) course of study. I can only give you a brief exposure to this topic. The text highlighted in green in these slides contain external hyperlinks. 1 / 14 Classification of Parallelism Software Sequential Concurrent Serial Some problem written as a se- Some problem written as a quential program (the MATLAB concurrent program (the O/S example from the textbook). example from the textbook). Execution on a serial platform. Execution on a serial platform. Parallel Some problem written as a se- Some problem written as a quential program (the MATLAB concurrent program (the O/S Hardware example from the textbook). example from the textbook). Execution on a parallel plat- Execution on a parallel plat- form. form. 2 / 14 Flynn’s Classification of Parallelism CU: control unit SM: shared memory DS1 PU MM PU: processor unit IS: instruction stream 1 1 MM: memory unit DS: data stream DS2 PU MM IS 2 2 CU IS SM IS DS CU PU MM DSn PUn MMm (a) SISD computer IS (b) SIMD computer IS1 IS1 IS1 IS1 IS1 DS1 CU PU DS CU PU MM 1 1 1 1 1 SM IS2 IS2 IS2 IS2 DS2 CU PU CU PU MM IS2 2 2 2 2 2 MM MM MM 1 2 m SM ISn ISn ISn ISn ISn DSn IS CUnPU n DS 2 CUnPU n MMm IS1 ISn (c) MISD computer (d) MIMD computer 3 / 14 Task Level Parallelism I Task Level Parallelism: organizing a program or computing solution into a set of processes/tasks/threads for simultaneous execution.
    [Show full text]
  • An Introduction to CUDA/Opencl and Graphics Processors
    An Introduction to CUDA/OpenCL and Graphics Processors Bryan Catanzaro, NVIDIA Research Overview ¡ Terminology ¡ The CUDA and OpenCL programming models ¡ Understanding how CUDA maps to NVIDIA GPUs ¡ Thrust 2/74 Heterogeneous Parallel Computing Latency Throughput Optimized CPU Optimized GPU Fast Serial Scalable Parallel Processing Processing 3/74 Latency vs. Throughput Latency Throughput ¡ Latency: yoke of oxen § Each core optimized for executing a single thread ¡ Throughput: flock of chickens § Cores optimized for aggregate throughput, deemphasizing individual performance ¡ (apologies to Seymour Cray) 4/74 Latency vs. Throughput, cont. Specificaons Sandy Bridge- Kepler EP (Tesla K20) 8 cores, 2 issue, 14 SMs, 6 issue, 32 Processing Elements 8 way SIMD way SIMD @3.1 GHz @730 MHz 8 cores, 2 threads, 8 14 SMs, 64 SIMD Resident Strands/ way SIMD: vectors, 32 way Threads (max) SIMD: Sandy Bridge-EP (32nm) 96 strands 28672 threads SP GFLOP/s 396 3924 Memory Bandwidth 51 GB/s 250 GB/s Register File 128 kB (?) 3.5 MB Local Store/L1 Cache 256 kB 896 kB L2 Cache 2 MB 1.5 MB L3 Cache 20 MB - Kepler (28nm) 5/74 Why Heterogeneity? ¡ Different goals produce different designs § Manycore assumes work load is highly parallel § Multicore must be good at everything, parallel or not ¡ Multicore: minimize latency experienced by 1 thread § lots of big on-chip caches § extremely sophisticated control ¡ Manycore: maximize throughput of all threads § lots of big ALUs § multithreading can hide latency … so skip the big caches § simpler control, cost amortized over
    [Show full text]
  • Lecture 19: SIMD Processors
    Digital Design & Computer Arch. Lecture 19: SIMD Processors Prof. Onur Mutlu ETH Zürich Spring 2020 7 May 2020 We Are Almost Done With This… ◼ Single-cycle Microarchitectures ◼ Multi-cycle Microarchitectures ◼ Pipelining ◼ Issues in Pipelining: Control & Data Dependence Handling, State Maintenance and Recovery, … ◼ Out-of-Order Execution ◼ Other Execution Paradigms 2 Approaches to (Instruction-Level) Concurrency ◼ Pipelining ◼ Out-of-order execution ◼ Dataflow (at the ISA level) ◼ Superscalar Execution ◼ VLIW ◼ Systolic Arrays ◼ Decoupled Access Execute ◼ Fine-Grained Multithreading ◼ SIMD Processing (Vector and array processors, GPUs) 3 Readings for this Week ◼ Required ◼ Lindholm et al., "NVIDIA Tesla: A Unified Graphics and Computing Architecture," IEEE Micro 2008. ◼ Recommended ❑ Peleg and Weiser, “MMX Technology Extension to the Intel Architecture,” IEEE Micro 1996. 4 Announcement ◼ Late submission of lab reports in Moodle ❑ Open until June 20, 2020, 11:59pm (cutoff date -- hard deadline) ❑ You can submit any past lab report, which you have not submitted before its deadline ❑ It is NOT allowed to re-submit anything (lab reports, extra assignments, etc.) that you had already submitted via other Moodle assignments ❑ We will grade your reports, but late submission has a penalization of 1 point, that is, the highest possible score per lab report will be 2 points 5 Exploiting Data Parallelism: SIMD Processors and GPUs SIMD Processing: Exploiting Regular (Data) Parallelism Flynn’s Taxonomy of Computers ◼ Mike Flynn, “Very High-Speed Computing
    [Show full text]
  • Chapter 1 Introduction
    Chapter 1 Introduction 1.1 Parallel Processing There is a continual demand for greater computational speed from a computer system than is currently possible (i.e. sequential systems). Areas need great computational speed include numerical modeling and simulation of scientific and engineering problems. For example; weather forecasting, predicting the motion of the astronomical bodies in the space, virtual reality, etc. Such problems are known as grand challenge problems. On the other hand, the grand challenge problem is the problem that cannot be solved in a reasonable amount of time [1]. One way of increasing the computational speed is by using multiple processors in single case box or network of computers like cluster operate together on a single problem. Therefore, the overall problem is needed to split into partitions, with each partition is performed by a separate processor in parallel. Writing programs for this form of computation is known as parallel programming [1]. How to execute the programs of applications in very fast way and on a concurrent manner? This is known as parallel processing. In the parallel processing, we must have underline parallel architectures, as well as, parallel programming languages and algorithms. 1.2 Parallel Architectures The main feature of a parallel architecture is that there is more than one processor. These processors may communicate and cooperate with one another to execute the program instructions. There are diverse classifications for the parallel architectures and the most popular one is the Flynn taxonomy (see Figure 1.1) [2]. 1 1.2.1 The Flynn Taxonomy Michael Flynn [2] has introduced taxonomy for various computer architectures based on notions of Instruction Streams (IS) and Data Streams (DS).
    [Show full text]
  • INTRODUCTION to PARALLEL and GPU COMPUTING Carlo Nardone Sr
    INTRODUCTION TO PARALLEL AND GPU COMPUTING Carlo Nardone Sr. Solution Architect, NVIDIA EMEA 1 Intro 2 Processors Trends AGENDA 3 Multiprocessors 4 GPU Computing 2 PARALLEL COMPUTING, ILLUSTRATED 3 FROM MULTICORE TO MANYCORE 4 PARALLEL COMPUTING IS OLD Source: Tim Mattson 5 1 Intro 2 Processors Trends AGENDA 3 Multiprocessors 4 GPU Computing 6 MOORE’S LAW Source: U.Delaware CISC879 / J.Dongarra 7 MICROPROCESSOR FREQUENCY TREND source: CA5 Hennessy/Patterson 8 THE POWER WALL 9 MICROPROCESSORS PERFORMANCE TREND source: CA5 Hennessy/Patterson 11 THE PROCESSOR-MEMORY GAP source: CA5 Hennessy & Patterson 13 ARITHMETIC INTENSITY Arithmetic intensity, specified as the number of floating-point operations to run the program divided by the number of bytes accessed in main memory [Williams et al. 2009]. Some kernels have an arithmetic intensity that scales with problem size, such as dense matrix, but there are many kernels with arithmetic intensities independent of problem size. Source: CA5 Hennessy & Patterson 14 BANDWIDTH AND THE ROOFLINE MODEL Source: HPCA5 15 1 Intro 2 Processors Trends AGENDA 3 Multiprocessors 4 GPU Computing 16 FLYNN’S TAXONOMY: SISD source: CA5 Hennessy & Patterson 17 FLYNN’S TAXONOMY: SIMD source: CA5 Hennessy & Patterson 18 FLYNN’S TAXONOMY: SIMD (2) source: CA5 Hennessy & Patterson 19 FLYNN’S TAXONOMY source: CA5 Hennessy & Patterson 20 FLYNN’S TAXONOMY: MISD (?) source: CA5 Hennessy & Patterson 21 FLYNN’S TAXONOMY: MIMD source: CA5 Hennessy & Patterson 22 MULTIPROCESSORS source: CA5 Hennessy & Patterson 23 MULTIPROCESSOR
    [Show full text]