DOCSLIB.ORG

Introduction to Openmp! Agenda!

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Introduction to Openmp! Agenda! Introduction to OpenMP! Agenda! • Introduction to parallel computing! • Classification of parallel computers! • What is OpenMP? ! • OpenMP examples! !" #" Parallel computing! Demand for parallel computing! A form of computation in which many instructions are carried out simultaneously operating on the principle With the increased use of computers in every sphere of that large problems can often be divided into smaller human activity, computer scientists are faced with two ones, which are then solved concurrently (in parallel).! crucial issues today:! Parallel computing has become a dominant paradigm in • Processing has to be done faster ! computer architecture, mainly in the form of multi-core processors.! • Larger or more complex problems need to be solved! Why is it required?! $" %" Parallelism! Serial computing! A problem is broken into a discrete series of instructions. ! Increasing the number of transistors as per Moore’s law Instructions are executed one after another. ! is not a solution, as it increases power consumption.! Only one instruction may execute at any moment in time.! Power consumption has been a major issue recently, as it causes a problem of processor heating.! The prefect solution is parallelism, in hardware as well as software.! Moore's law (1965): The density of transistors will double every 18 months.! &" '" Parallel computing! Forms of parallelism! A problem is broken into discrete parts that can be solved concurrently. • Bit-level parallelism! !!!!!!! Each part is further broken down to a series of instructions. ! Instructions from each part execute simultaneously on different CPUs! !More than one bit is manipulated per clock cycle ! • Instruction-level parallelism! !!!! !Machine instructions are processed in multi-stage pipelines! • Data parallelism! !!!!!!! !Focuses on distributing the data across different parallel !computing nodes. It is also called loop-level parallelism! • Task parallelism! !!!!!!! !Focuses on distributing tasks across different parallel !computing nodes. It is also called functional or control !parallelism! (" )" Key difference between data ! Parallelism versus concurrency! and task parallelism! • !Concurrency is when two tasks can start, run and !complete in overlapping time periods. It doesn’t Data parallelism! Task parallelism! !necessarily mean that they will ever be running at A task ‘A’ is divided into sub- A task ‘A’ and a task ‘B’ are !the same instant of time. E.g., multitasking on a parts and then processed ! processed separately by !single-threaded machine.! different processors! • !Parallelism is when two tasks literally run at the !same time, e.g. on a multi-core computer.! *" !+" Granularity! Flynn’s classification of computers! • !Granularity is the ratio of computation to !communication.! • !Fine-grained parallelism means individual !tasks are relatively small in terms of code size !and execution time. The data are transferred !among processors frequently in amounts of !one or a few memory words.! • !Coarse-grained is the opposite: data are !communicated infrequently, after larger !amounts of computation. ! M. Flynn, 1966! !!" !#" Flynn’s taxonomy: SISD! Flynn’s taxonomy: MIMD! 678" ,-./"." ,-./"." 5.,,"9" ,-./">" 678" .//"0" .//"0" ,-./"." >?5">" 12-34"5"" 12-34"5"" :;<=","" :;<=@"" =!" =#" =A" A serial (non-parallel) computer! Examples: uni-core PCs, single CPU workstations and Multiple Instruction: Every Examples: Most modern Single Instruction: Only one instruction mainframes! processing unit may be executing computers fall into this stream is acted on by the CPU in one a different instruction at any given category! clock cycle! clock cycle! Single Data: Only one data stream is used Multiple Data: Each processing as input during any clock cycle! unit can operate on a different data element! !$" !%" Flynn’s taxonomy: SIMD! Flynn’s taxonomy: MISD! ,-./".B!C" ,-./".B#C" ,-./".BAC" 678" ,-./"." ,-./"." ,-./"." 678" .//"0B!C" .//"0B#C" .//"0BAC" <;,2"!" <;,2"#" <;,2"A" 12-34"5B!C"" 12-34"5B#C"" 12-34"5BAC"" 12-34"5B!C"" 12-34"5B#C"" 12-34"5BAC"" =!" =#" =A" =!" =#" =A" Single Instruction: All processing Best suited for problems of A single data stream is fed into Examples: Few actual units execute the same instruction high degree of regularity, multiple processing units! examples of this category at any given clock cycle! such as image processing! have ever existed! Each processing unit operates on Multiple Data: Each processing Examples: Connection the data independently via unit can operate on a different data Machine CM-2, Cray C90, independent instruction streams! element! NVIDIA! !&" !'" Flynn’s Taxonomy! (1966)! Parallel classification! MISD (Multiple-Instruction Single-Data)! MIMD (Multiple-Instruction Multiple-Data)! Parallel computers:! >!! > ! A >!! >A! = ! !"!"!" = ! • SIMD (Single Instruction Multiple Data): ! ! A =!! !"!"!" =A! !Synchronized execution of the same instruction on a ! /D! /E! / !/ ! / !/ ! !set of processors! D! E! DA EA SISD (Single-Instruction Single-Data)! SIMD (Single-Instruction Multiple-Data)! • MIMD (Multiple Instruction Multiple Data): ! !Asynchronous execution of different instructions! >! >! =!! !"!"!! =A! "=! / !/ ! / !/ ! D! E! DA EA / ! / ! D E !(" 18! Parallel machine memory models! Distributed memory! Distributed memory model! Shared memory model! Processors have local (non-shared) + Very scalable! Hybrid memory model! memory! + No cache coherence problems! + Use commodity processors! Requires communication network. Data from one processor must be - Lot of programmer responsibility! communicated if required ! - NUMA (Non-Uniform Memory !Access) times! Synchronization is programmer’s responsibility! !*" #+" Shared memory! Hybrid memory! Multiple processors operate + User friendly! independently but access global + UMA (Uniform Memory Access) Biggest machines today have both! memory space! !times. UMA = SMP (Symmetric !Multi-Processor)! Advantages and disadvantages are Changes in one memory location those of the individual parts! are visible to all processors! - Expense for large computers! Poor scalability is “myth”! #!" ##" Shared memory access! Uniform Memory Access – UMA: ! (Symmetric Multi-Processors – SMP)! • !centralized shared memory, accesses to global !memory! !from all processors have same latency! Non-uniform Memory Access – NUMA: ! (Distributed Shared Memory – DSM)! • !memory is distributed among the nodes, local accesses !much faster than remote accesses! #$" #%" Moore’s Law! (1965)! Parallelism in hardware and software! • Hardware! Moore’s Law, despite predictions of its demise, is still !going parallel (many cores)! in effect. Despite power issues, transistor densities are still doubling every 18 to 24 months. ! • Software?! With the end of frequency scaling, these new !Herb Sutter (Microsoft):! transistors can be used to add extra hardware, such as !“The free lunch is over. Software performance additional cores, to facilitate parallel computing. ! !will no longer increase from one generation to !the next as hardware improves ... unless it is !parallel software.”! #&" #'" Amdahl’s Law [1967]! Gene Amdahl, IBM designer! Assume fraction fp of execution time is parallelizable! !No overhead for scheduling, communication, synchronization, etc. ! Fraction fs = 1"fp is serial (not parallelizable)" Time on 1 processor:! T1 = fpT1 + (1! fp )T1 f T Time on N processors:"T = p 1 + (1! f )T N N p 1 T 1 1 Speedup "= 1 = ! T fp 1" f N + (1! f ) p N p #(" #)" 1 1 Speedup = Speedup = f f p + (1! f ) p + (1! f ) Example! N p N p Amdahl’s Law! 95% of a program’s execution time occurs inside a loop that can be executed in parallel. ! For mainframes Amdahl expected fs =1-fp =35%! What is the maximum speedup we should expect from a parallel version of the program executing on 8 processors?! !For a 4-processor: speedup # 2! !For an infinity-processor: speedup < 3 ! 1 Speedup ! " 5.9 !Therefore, stay with mainframes with only ! ! 0.95 / 8 0.05 + !one/few processors!! What is the maximum speedup achievable by this program, Amdahl’s Law prevented designers from exploiting regardless of how many processes are used?! parallelism for many years as even the most trivially 1 parallel programs contain a small amount of natural serial Speedup ! = 20 0.05 code! #*" $+" Limitations of Amdahl’s Law! Gustafson’s Law [1988]! Two key flaws in past arguments that Amdahl’s law is a The key is in observing that f and N are not independent of fatal limit to the future of parallelism are! s one another.! !(1) Its proof focuses on the steps in a particular ! Parallelism can be used to increase the size of the problem. ! !algorithm, and does not consider that other ! Time gained by using N processors (in stead of one) for the ! !algorithms with more parallelism may exist! parallel part of the program can be used in the serial part to solve a larger problem.! !(2) Gustafson’s Law: The proportion of the ! !! !computations that are sequential normally ! Gustafson’s speedup factor is called scaled speedup factor.! !! !decreases as the problem size increases! $!" $#" Speedup = N + (1 ! N ) fs Gustafson’s Law (cont’d)! Example! 95% of a program’s execution time occurs inside a loop that can be Assume TN is fixed to one, meaning the problem is executed in parallel. ! scaled up to fit the parallel computer. ! Then execution time on a single processor will be What is the scaled speedup on 8 processors?! fs+Nfp as the N parallel parts must be executed sequentially. Thus, ! Speedup = 8 + (1! 8)" 0.05 = 7.65 T1 fs + Nfp Speedup = = = fs + Nfp = N + (1! N) fs TN
Load more

Recommended publications
  • Data-Level Parallelism
    Fall 2015 :: CSE 610 – Parallel Computer Architectures Data-Level Parallelism Nima Honarmand Fall 2015 :: CSE 610 – Parallel Computer Architectures Overview • Data Parallelism vs. Control Parallelism – Data Parallelism: parallelism arises from executing essentially the same code on a large number of objects – Control Parallelism: parallelism arises from executing different threads of control concurrently • Hypothesis: applications that use massively parallel machines will mostly exploit data parallelism – Common in the Scientific Computing domain • DLP originally linked with SIMD machines; now SIMT is more common – SIMD: Single Instruction Multiple Data – SIMT: Single Instruction Multiple Threads Fall 2015 :: CSE 610 – Parallel Computer Architectures Overview • Many incarnations of DLP architectures over decades – Old vector processors • Cray processors: Cray-1, Cray-2, …, Cray X1 – SIMD extensions • Intel SSE and AVX units • Alpha Tarantula (didn’t see light of day ) – Old massively parallel computers • Connection Machines • MasPar machines – Modern GPUs • NVIDIA, AMD, Qualcomm, … • Focus of throughput rather than latency Vector Processors 4 SCALAR VECTOR (1 operation) (N operations) r1 r2 v1 v2 + + r3 v3 vector length add r3, r1, r2 vadd.vv v3, v1, v2 Scalar processors operate on single numbers (scalars) Vector processors operate on linear sequences of numbers (vectors) 6.888 Spring 2013 - Sanchez and Emer - L14 What’s in a Vector Processor? 5 A scalar processor (e.g. a MIPS processor) Scalar register file (32 registers) Scalar functional units (arithmetic, load/store, etc) A vector register file (a 2D register array) Each register is an array of elements E.g. 32 registers with 32 64-bit elements per register MVL = maximum vector length = max # of elements per register A set of vector functional units Integer, FP, load/store, etc Some times vector and scalar units are combined (share ALUs) 6.888 Spring 2013 - Sanchez and Emer - L14 Example of Simple Vector Processor 6 6.888 Spring 2013 - Sanchez and Emer - L14 Basic Vector ISA 7 Instr.
    [Show full text]
  • High Performance Computing Through Parallel and Distributed Processing
    Yadav S. et al., J. Harmoniz. Res. Eng., 2013, 1(2), 54-64 Journal Of Harmonized Research (JOHR) Journal Of Harmonized Research in Engineering 1(2), 2013, 54-64 ISSN 2347 – 7393 Original Research Article High Performance Computing through Parallel and Distributed Processing Shikha Yadav, Preeti Dhanda, Nisha Yadav Department of Computer Science and Engineering, Dronacharya College of Engineering, Khentawas, Farukhnagar, Gurgaon, India Abstract : There is a very high need of High Performance Computing (HPC) in many applications like space science to Artificial Intelligence. HPC shall be attained through Parallel and Distributed Computing. In this paper, Parallel and Distributed algorithms are discussed based on Parallel and Distributed Processors to achieve HPC. The Programming concepts like threads, fork and sockets are discussed with some simple examples for HPC. Keywords: High Performance Computing, Parallel and Distributed processing, Computer Architecture Introduction time to solve large problems like weather Computer Architecture and Programming play forecasting, Tsunami, Remote Sensing, a significant role for High Performance National calamities, Defence, Mineral computing (HPC) in large applications Space exploration, Finite-element, Cloud science to Artificial Intelligence. The Computing, and Expert Systems etc. The Algorithms are problem solving procedures Algorithms are Non-Recursive Algorithms, and later these algorithms transform in to Recursive Algorithms, Parallel Algorithms particular Programming language for HPC. and Distributed Algorithms. There is need to study algorithms for High The Algorithms must be supported the Performance Computing. These Algorithms Computer Architecture. The Computer are to be designed to computer in reasonable Architecture is characterized with Flynn’s Classification SISD, SIMD, MIMD, and For Correspondence: MISD. Most of the Computer Architectures preeti.dhanda01ATgmail.com are supported with SIMD (Single Instruction Received on: October 2013 Multiple Data Streams).
    [Show full text]
  • 2.5 Classification of Parallel Computers
    52 // Architectures 2.5 Classification of Parallel Computers 2.5 Classification of Parallel Computers 2.5.1 Granularity In parallel computing, granularity means the amount of computation in relation to communication or synchronisation Periods of computation are typically separated from periods of communication by synchronization events. • fine level (same operations with different data) ◦ vector processors ◦ instruction level parallelism ◦ fine-grain parallelism: – Relatively small amounts of computational work are done between communication events – Low computation to communication ratio – Facilitates load balancing 53 // Architectures 2.5 Classification of Parallel Computers – Implies high communication overhead and less opportunity for per- formance enhancement – If granularity is too fine it is possible that the overhead required for communications and synchronization between tasks takes longer than the computation. • operation level (different operations simultaneously) • problem level (independent subtasks) ◦ coarse-grain parallelism: – Relatively large amounts of computational work are done between communication/synchronization events – High computation to communication ratio – Implies more opportunity for performance increase – Harder to load balance efficiently 54 // Architectures 2.5 Classification of Parallel Computers 2.5.2 Hardware: Pipelining (was used in supercomputers, e.g. Cray-1) In N elements in pipeline and for 8 element L clock cycles =) for calculation it would take L + N cycles; without pipeline L ∗ N cycles Example of good code for pipelineing: §doi =1 ,k ¤ z ( i ) =x ( i ) +y ( i ) end do ¦ 55 // Architectures 2.5 Classification of Parallel Computers Vector processors, fast vector operations (operations on arrays). Previous example good also for vector processor (vector addition) , but, e.g. recursion – hard to optimise for vector processors Example: IntelMMX – simple vector processor.
    [Show full text]
  • Balanced Multithreading: Increasing Throughput Via a Low Cost Multithreading Hierarchy
    Balanced Multithreading: Increasing Throughput via a Low Cost Multithreading Hierarchy Eric Tune Rakesh Kumar Dean M. Tullsen Brad Calder Computer Science and Engineering Department University of California at San Diego {etune,rakumar,tullsen,calder}@cs.ucsd.edu Abstract ibility of SMT comes at a cost. The register file and rename tables must be enlarged to accommodate the architectural reg- A simultaneous multithreading (SMT) processor can issue isters of the additional threads. This in turn can increase the instructions from several threads every cycle, allowing it to clock cycle time and/or the depth of the pipeline. effectively hide various instruction latencies; this effect in- Coarse-grained multithreading (CGMT) [1, 21, 26] is a creases with the number of simultaneous contexts supported. more restrictive model where the processor can only execute However, each added context on an SMT processor incurs a instructions from one thread at a time, but where it can switch cost in complexity, which may lead to an increase in pipeline to a new thread after a short delay. This makes CGMT suited length or a decrease in the maximum clock rate. This pa- for hiding longer delays. Soon, general-purpose micropro- per presents new designs for multithreaded processors which cessors will be experiencing delays to main memory of 500 combine a conservative SMT implementation with a coarse- or more cycles. This means that a context switch in response grained multithreading capability. By presenting more virtual to a memory access can take tens of cycles and still provide contexts to the operating system and user than are supported a considerable performance benefit.
    [Show full text]
  • Cimple: Instruction and Memory Level Parallelism a DSL for Uncovering ILP and MLP
    Cimple: Instruction and Memory Level Parallelism A DSL for Uncovering ILP and MLP Vladimir Kiriansky, Haoran Xu, Martin Rinard, Saman Amarasinghe MIT CSAIL {vlk,haoranxu510,rinard,saman}@csail.mit.edu Abstract Processors have grown their capacity to exploit instruction Modern out-of-order processors have increased capacity to level parallelism (ILP) with wide scalar and vector pipelines, exploit instruction level parallelism (ILP) and memory level e.g., cores have 4-way superscalar pipelines, and vector units parallelism (MLP), e.g., by using wide superscalar pipelines can execute 32 arithmetic operations per cycle. Memory and vector execution units, as well as deep buffers for in- level parallelism (MLP) is also pervasive with deep buffering flight memory requests. These resources, however, often ex- between caches and DRAM that allows 10+ in-flight memory hibit poor utilization rates on workloads with large working requests per core. Yet, modern CPUs still struggle to extract sets, e.g., in-memory databases, key-value stores, and graph matching ILP and MLP from the program stream. analytics, as compilers and hardware struggle to expose ILP Critical infrastructure applications, e.g., in-memory databases, and MLP from the instruction stream automatically. key-value stores, and graph analytics, characterized by large In this paper, we introduce the IMLP (Instruction and working sets with multi-level address indirection and pointer Memory Level Parallelism) task programming model. IMLP traversals push hardware to its limits: large multi-level caches tasks execute as coroutines that yield execution at annotated and branch predictors fail to keep processor stalls low. The long-latency operations, e.g., memory accesses, divisions, out-of-order windows of hundreds of instructions are also or unpredictable branches.
    [Show full text]
  • Amdahl's Law Threading, Openmp
    Computer Science 61C Spring 2018 Wawrzynek and Weaver Amdahl's Law Threading, OpenMP 1 Big Idea: Amdahl’s (Heartbreaking) Law Computer Science 61C Spring 2018 Wawrzynek and Weaver • Speedup due to enhancement E is Exec time w/o E Speedup w/ E = ---------------------- Exec time w/ E • Suppose that enhancement E accelerates a fraction F (F <1) of the task by a factor S (S>1) and the remainder of the task is unaffected Execution Time w/ E = Execution Time w/o E × [ (1-F) + F/S] Speedup w/ E = 1 / [ (1-F) + F/S ] 2 Big Idea: Amdahl’s Law Computer Science 61C Spring 2018 Wawrzynek and Weaver Speedup = 1 (1 - F) + F Non-speed-up part S Speed-up part Example: the execution time of half of the program can be accelerated by a factor of 2. What is the program speed-up overall? 1 1 = = 1.33 0.5 + 0.5 0.5 + 0.25 2 3 Example #1: Amdahl’s Law Speedup w/ E = 1 / [ (1-F) + F/S ] Computer Science 61C Spring 2018 Wawrzynek and Weaver • Consider an enhancement which runs 20 times faster but which is only usable 25% of the time Speedup w/ E = 1/(.75 + .25/20) = 1.31 • What if its usable only 15% of the time? Speedup w/ E = 1/(.85 + .15/20) = 1.17 • Amdahl’s Law tells us that to achieve linear speedup with 100 processors, none of the original computation can be scalar! • To get a speedup of 90 from 100 processors, the percentage of the original program that could be scalar would have to be 0.1% or less Speedup w/ E = 1/(.001 + .999/100) = 90.99 4 Amdahl’s Law If the portion of Computer Science 61C Spring 2018 the program that Wawrzynek and Weaver can be parallelized is small, then the speedup is limited The non-parallel portion limits the performance 5 Strong and Weak Scaling Computer Science 61C Spring 2018 Wawrzynek and Weaver • To get good speedup on a parallel processor while keeping the problem size fixed is harder than getting good speedup by increasing the size of the problem.
    [Show full text]
  • Introduction to Multi-Threading and Vectorization Matti Kortelainen Larsoft Workshop 2019 25 June 2019 Outline
    Introduction to multi-threading and vectorization Matti Kortelainen LArSoft Workshop 2019 25 June 2019 Outline Broad introductory overview: • Why multithread? • What is a thread? • Some threading models – std::thread – OpenMP (fork-join) – Intel Threading Building Blocks (TBB) (tasks) • Race condition, critical region, mutual exclusion, deadlock • Vectorization (SIMD) 2 6/25/19 Matti Kortelainen | Introduction to multi-threading and vectorization Motivations for multithreading Image courtesy of K. Rupp 3 6/25/19 Matti Kortelainen | Introduction to multi-threading and vectorization Motivations for multithreading • One process on a node: speedups from parallelizing parts of the programs – Any problem can get speedup if the threads can cooperate on • same core (sharing L1 cache) • L2 cache (may be shared among small number of cores) • Fully loaded node: save memory and other resources – Threads can share objects -> N threads can use significantly less memory than N processes • If smallest chunk of data is so big that only one fits in memory at a time, is there any other option? 4 6/25/19 Matti Kortelainen | Introduction to multi-threading and vectorization What is a (software) thread? (in POSIX/Linux) • “Smallest sequence of programmed instructions that can be managed independently by a scheduler” [Wikipedia] • A thread has its own – Program counter – Registers – Stack – Thread-local memory (better to avoid in general) • Threads of a process share everything else, e.g. – Program code, constants – Heap memory – Network connections – File handles
    [Show full text]
  • Computer Hardware Architecture Lecture 4
    Computer Hardware Architecture Lecture 4 Manfred Liebmann Technische Universit¨atM¨unchen Chair of Optimal Control Center for Mathematical Sciences, M17 [email protected] November 10, 2015 Manfred Liebmann November 10, 2015 Reading List • Pacheco - An Introduction to Parallel Programming (Chapter 1 - 2) { Introduction to computer hardware architecture from the parallel programming angle • Hennessy-Patterson - Computer Architecture - A Quantitative Approach { Reference book for computer hardware architecture All books are available on the Moodle platform! Computer Hardware Architecture 1 Manfred Liebmann November 10, 2015 UMA Architecture Figure 1: A uniform memory access (UMA) multicore system Access times to main memory is the same for all cores in the system! Computer Hardware Architecture 2 Manfred Liebmann November 10, 2015 NUMA Architecture Figure 2: A nonuniform memory access (UMA) multicore system Access times to main memory differs form core to core depending on the proximity of the main memory. This architecture is often used in dual and quad socket servers, due to improved memory bandwidth. Computer Hardware Architecture 3 Manfred Liebmann November 10, 2015 Cache Coherence Figure 3: A shared memory system with two cores and two caches What happens if the same data element z1 is manipulated in two different caches? The hardware enforces cache coherence, i.e. consistency between the caches. Expensive! Computer Hardware Architecture 4 Manfred Liebmann November 10, 2015 False Sharing The cache coherence protocol works on the granularity of a cache line. If two threads manipulate different element within a single cache line, the cache coherency protocol is activated to ensure consistency, even if every thread is only manipulating its own data.
    [Show full text]
  • Parallel Generation of Image Layers Constructed by Edge Detection
    International Journal of Applied Engineering Research ISSN 0973-4562 Volume 13, Number 10 (2018) pp. 7323-7332 © Research India Publications. http://www.ripublication.com Speed Up Improvement of Parallel Image Layers Generation Constructed By Edge Detection Using Message Passing Interface Alaa Ismail Elnashar Faculty of Science, Computer Science Department, Minia University, Egypt. Associate professor, Department of Computer Science, College of Computers and Information Technology, Taif University, Saudi Arabia. Abstract A. Elnashar [29] introduced two parallel techniques, NASHT1 and NASHT2 that apply edge detection to produce a set of Several image processing techniques require intensive layers for an input image. The proposed techniques generate computations that consume CPU time to achieve their results. an arbitrary number of image layers in a single parallel run Accelerating these techniques is a desired goal. Parallelization instead of generating a unique layer as in traditional case; this can be used to save the execution time of such techniques. In helps in selecting the layers with high quality edges among this paper, we propose a parallel technique that employs both the generated ones. Each presented parallel technique has two point to point and collective communication patterns to versions based on point to point communication "Send" and improve the speedup of two parallel edge detection techniques collective communication "Bcast" functions. The presented that generate multiple image layers using Message Passing techniques achieved notable relative speedup compared with Interface, MPI. In addition, the performance of the proposed that of sequential ones. technique is compared with that of the two concerned ones. In this paper, we introduce a speed up improvement for both Keywords: Image Processing, Image Segmentation, Parallel NASHT1 and NASHT2.
    [Show full text]
  • CS650 Computer Architecture Lecture 10 Introduction to Multiprocessors
    NJIT Computer Science Dept CS650 Computer Architecture CS650 Computer Architecture Lecture 10 Introduction to Multiprocessors and PC Clustering Andrew Sohn Computer Science Department New Jersey Institute of Technology Lecture 10: Intro to Multiprocessors/Clustering 1/15 12/7/2003 A. Sohn NJIT Computer Science Dept CS650 Computer Architecture Key Issues Run PhotoShop on 1 PC or N PCs Programmability • How to program a bunch of PCs viewed as a single logical machine. Performance Scalability - Speedup • Run PhotoShop on 1 PC (forget the specs of this PC) • Run PhotoShop on N PCs • Will it run faster on N PCs? Speedup = ? Lecture 10: Intro to Multiprocessors/Clustering 2/15 12/7/2003 A. Sohn NJIT Computer Science Dept CS650 Computer Architecture Types of Multiprocessors Key: Data and Instruction Single Instruction Single Data (SISD) • Intel processors, AMD processors Single Instruction Multiple Data (SIMD) • Array processor • Pentium MMX feature Multiple Instruction Single Data (MISD) • Systolic array • Special purpose machines Multiple Instruction Multiple Data (MIMD) • Typical multiprocessors (Sun, SGI, Cray,...) Single Program Multiple Data (SPMD) • Programming model Lecture 10: Intro to Multiprocessors/Clustering 3/15 12/7/2003 A. Sohn NJIT Computer Science Dept CS650 Computer Architecture Shared-Memory Multiprocessor Processor Prcessor Prcessor Interconnection network Main Memory Storage I/O Lecture 10: Intro to Multiprocessors/Clustering 4/15 12/7/2003 A. Sohn NJIT Computer Science Dept CS650 Computer Architecture Distributed-Memory Multiprocessor Processor Processor Processor IO/S MM IO/S MM IO/S MM Interconnection network IO/S MM IO/S MM IO/S MM Processor Processor Processor Lecture 10: Intro to Multiprocessors/Clustering 5/15 12/7/2003 A.
    [Show full text]
  • Scheduling Task Parallelism on Multi-Socket Multicore Systems
    Scheduling Task Parallelism" on Multi-Socket Multicore Systems" Stephen Olivier, UNC Chapel Hill Allan Porterfield, RENCI Kyle Wheeler, Sandia National Labs Jan Prins, UNC Chapel Hill The University of North Carolina at Chapel Hill Outline" Introduction and Motivation Scheduling Strategies Evaluation Closing Remarks The University of North Carolina at Chapel Hill ! Outline" Introduction and Motivation Scheduling Strategies Evaluation Closing Remarks The University of North Carolina at Chapel Hill ! Task Parallel Programming in a Nutshell! • A task consists of executable code and associated data context, with some bookkeeping metadata for scheduling and synchronization. • Tasks are significantly more lightweight than threads. • Dynamically generated and terminated at run time • Scheduled onto threads for execution • Used in Cilk, TBB, X10, Chapel, and other languages • Our work is on the recent tasking constructs in OpenMP 3.0. The University of North Carolina at Chapel Hill ! 4 Simple Task Parallel OpenMP Program: Fibonacci! int fib(int n)! {! fib(10)! int x, y;! if (n < 2) return n;! #pragma omp task! fib(9)! fib(8)! x = fib(n - 1);! #pragma omp task! y = fib(n - 2);! #pragma omp taskwait! fib(8)! fib(7)! return x + y;! }! The University of North Carolina at Chapel Hill ! 5 Useful Applications! • Recursive algorithms cilksort cilksort cilksort cilksort cilksort • E.g. Mergesort • List and tree traversal cilkmerge cilkmerge cilkmerge cilkmerge cilkmerge cilkmerge • Irregular computations cilkmerge • E.g., Adaptive Fast Multipole cilkmerge cilkmerge
    [Show full text]
  • Demystifying Parallel and Distributed Deep Learning: an In-Depth Concurrency Analysis
    1 Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis TAL BEN-NUN and TORSTEN HOEFLER, ETH Zurich, Switzerland Deep Neural Networks (DNNs) are becoming an important tool in modern computing applications. Accelerating their training is a major challenge and techniques range from distributed algorithms to low-level circuit design. In this survey, we describe the problem from a theoretical perspective, followed by approaches for its parallelization. We present trends in DNN architectures and the resulting implications on parallelization strategies. We then review and model the different types of concurrency in DNNs: from the single operator, through parallelism in network inference and training, to distributed deep learning. We discuss asynchronous stochastic optimization, distributed system architectures, communication schemes, and neural architecture search. Based on those approaches, we extrapolate potential directions for parallelism in deep learning. CCS Concepts: • General and reference → Surveys and overviews; • Computing methodologies → Neural networks; Parallel computing methodologies; Distributed computing methodologies; Additional Key Words and Phrases: Deep Learning, Distributed Computing, Parallel Algorithms ACM Reference Format: Tal Ben-Nun and Torsten Hoefler. 2018. Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis. 47 pages. 1 INTRODUCTION Machine Learning, and in particular Deep Learning [143], is rapidly taking over a variety of aspects in our daily lives.
    [Show full text]