DOCSLIB.ORG

Experimenting with a Big Data Framework for Scaling a Data Quality Query System

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Experimenting with a Big Data Framework for Scaling a Data Quality Query System EXPERIMENTING WITH A BIG DATA FRAMEWORK FOR SCALING A DATA QUALITY QUERY SYSTEM A THESIS SUBMITTED TO THE UNIVERSITY OF MANCHESTER FOR THE DEGREE OF MASTER OF PHILOSOPHY IN THE FACULTY OF SCIENCE AND ENGINEERING 2016 By Sonia Cisneros Cabrera School of Computer Science Contents Abstract 13 Declaration 14 Copyright 15 Dedication 16 Acknowledgements 17 1 Introduction 18 1.1 Motivation . 18 1.2 Research Goals and Contributions . 20 1.3 Research Method . 21 1.4 Thesis Overview . 25 2 Background and Related Work 26 2.1 Data Quality: Wrangling and the Big Data Life Cycle . 26 2.2 Big Data and Parallelism . 31 2.3 Big Data Frameworks: Tools and Projects . 33 2.3.1 MapReduce, Hadoop and Spark . 35 2.4 Scalability of Data Management Operators . 37 3 Designing a Scalable DQ2S 42 3.1 The Data Quality Query System (DQ2S) . 42 3.1.1 The Framework . 43 3.1.2 The Query . 44 3.1.3 The Datasets . 46 3.1.4 The Algorithms Included in the DQ2S Database Algebra Im- plementation . 48 2 3.2 The Artifact . 55 3.2.1 The Java DQ2S......................... 55 3.2.2 The Python DQ2S Implementation . 58 3.2.3 The Optimised Python DQ2S Implementation . 60 3.2.4 The PySpark DQ2S Implementation . 62 The Apache Spark Generals . 62 Spark Python API . 67 3.3 Summary . 70 4 Experiment design 71 4.1 Objectives and Scope of the Experiment . 72 4.2 Data and Tools . 74 4.3 Setup: Spark Cluster and Application Settings . 76 4.3.1 The Apache Spark Local Mode . 76 4.3.2 The Apache Spark Standalone Mode . 77 4.4 Experiment Procedure . 79 4.5 Algorithm Comparison . 90 4.6 Experiment Evaluation . 92 4.6.1 Performance Metrics . 92 4.6.2 Validity Evaluation . 94 4.7 Summary . 99 5 Evaluation of Experimental Results 100 5.1 Experiments Results . 100 5.2 Algorithm Comparison . 118 5.3 Summary . 139 6 Conclusions and Future Work 140 6.1 Research Summary and Key Results . 140 6.2 Research Contributions and Impact . 141 6.3 Future Work . 142 Bibliography 146 A DQ2S Source code 162 A.1 The Java DQ2S ............................. 162 A.2 The Python DQ2S............................ 175 3 A.3 The Optimised Python DQ2S . 186 A.4 The PySpark DQ2S........................... 191 B Apache Spark set up 198 B.1 Configuration of Spark’s Basic Elements . 198 B.2 Single Machine Deployment . 199 B.3 PySpark Submission to the DPSF Cluster . 201 C Testing Results 205 C.1 Raw Results . 205 C.2 Performance Metrics Results . 224 D Apache Spark User Level Proposal 227 Word Count: 37 242 4 List of Tables 1.1 The Design Science Research guideliness . 23 2.1 Big data life cycle processes definitions. 30 2.2 Big data frameworks available for data quality tasks . 34 2.3 Summary of the most relevant state-of-the-art related studies. 41 3.1 Operators comprising the DQ2S Timeliness query. 50 3.2 Java DQ2S technical description . 58 3.3 Python DQ2S technical description . 60 3.4 Optimised Python DQ2S technical description . 62 3.5 PySpark DQ2S technical description. 70 4.1 Research Question’s solution plan . 73 4.2 Relation of number of rows and size per each input dataset utilised in the experimentation. 74 4.3 Hardware specification of the desktop machine (PC) and the cluster (DPSF) utilised in the experimentation. 75 4.4 Tools used in the empirical evaluation of the research . 76 4.5 Spark configuration locations . 79 4.6 General overview of the experiment’s aim . 79 4.7 Number of runtimes and averages collected by experiment. 80 4.8 Conclusion validity threats applicable to the research presented. De- tails and addressing approach. 96 4.9 Internal validity threats applicable to the research presented. Details and addressing approach. 97 4.10 Construct validity threats applicable to the research presented. Details and addressing approach. 98 4.11 External validity threats applicable to the research presented. Details and addressing approach. 99 5 5.1 Relation of the dataset sizes and results when executing Optimised Python DQ2S in a single machine for Experiment 4. 104 5.2 Size in bytes for the first dataset that caused a failure (12 000 000 rows), and the maximum number of rows processed by the Optimised Python DQ2S (11 500 000). 104 5.3 Relation of seconds decreased from different configurations using 1 worker (W) and 1 to 4 cores (C) with the Standalone mode in a single machine. 108 5.4 Times Slower: Relation between the runtime results of the PySpark in- stance with 1 (sequential) and 8 threads (best time). Where local[1]<local[8] indicates the PySpark DQ2S in local mode with 8 threads is slower than the same implementation with 8 threads as many times as indicated. 111 5.5 Size in GB for the datasets sizes processed on the cluster. 115 5.6 Relation of the dataset sizes and results when executing the Optimised Python DQ2S in the cluster for Experiment 9. 117 5.7 Times Slower: Relation between Original Java and Python runtime results. Where Python <Java indicates the Python DQ2S is slower than the Java implementation as many times as indicated. 118 5.8 Times Slower: Relation between Python and Optimised Python run- time results. Where Python <Optimised Python indicates the Python DQ2S is slower than the Optimised Python implementation as many times as indicated. 122 5.9 Times Slower: Relation between Original Java and Optimised Python runtime results. Where Java <Optimised Python indicates the Java DQ2S is slower than the Optimised Python implementation as many times as indicated. 122 5.10 Times Slower: Relation between the runtime results of the Optimised Python and PySpark with 1 thread. Where local[1]<Optimised Python indicates the PySpark DQ2S in local mode with 1 thread is slower than the Optimised implementation as many times as indicated. 124 5.11 Times Slower: Relation between the runtime results of the Optimised Python and PySpark with 8 threads. Where local[8]<Optimised Python indicates the PySpark DQ2S in local mode with 8 threads is slower than the Optimised implementation as many times as indicated. 124 6 5.12 Times Slower: Relation between the runtime results of the Optimised Python and PySpark with 1 and 8 threads for 11 500 000 rows. Where the notation implementation <implementation indicates the left im- plementation is slower than the implementation on the right as many times as indicated. 126 B.1 Apache Spark processes’ default values for Standalone mode, where NA stands for “Not Applicable”. 198 B.2 Apache Spark processes’ general configuration parameters and loca- tion, where NA stands for “Not Applicable”. 199 C.1 Original Java DQ2S execution times. 209 C.2 Python DQ2S execution times. 210 C.3 Optimised Python DQ2S execution times. 210 C.4 Optimised Python DQ2S execution times in nanoseconds and seconds for 11 500 000 rows. 211 C.5 Execution times obtained from PySpark DQ2S Standalone mode in a single machine over 1 worker and 1 core. 211 C.6 Execution times obtained from PySpark DQ2S Standalone mode in a single machine over 1 worker and 2 cores each one. 212 C.7 Execution times obtained from PySpark DQ2S Standalone mode in a single machine over 1 worker and 3 cores each one. 212 C.8 Execution times obtained from PySpark DQ2S Standalone mode in a single machine over 1 worker and 4 cores each one. 213 C.9 Execution times obtained from PySpark DQ2S Standalone mode in a single machine over 2 workers and 1 core each one. 213 C.10 Execution times obtained from PySpark DQ2S Standalone mode in a single machine over 2 workers and 2 cores each one. 214 C.11 Execution times obtained from PySpark DQ2S Standalone mode in a single machine over 3 workers and 1 core each one. 214 C.12 Execution times obtained from PySpark DQ2S Standalone mode in a single machine over 4 workers and 1 core each one. 215 C.13 Execution times obtained from PySpark DQ2S Local mode using 1 core in a single machine. 215 C.14 Execution times obtained from PySpark DQ2S Local mode using 2 cores in a single machine. 216 7 C.15 Execution times obtained from PySpark DQ2S Local mode using 3 cores in a single machine. 216 C.16 Execution times obtained from PySpark DQ2S Local mode using 4 cores in a single machine. 217 C.17 Execution times obtained from PySpark DQ2S Local mode using 5 cores in a single machine. 218 C.18 Execution times obtained from PySpark DQ2S Local mode using 6 cores in a single machine. 218 C.19 Execution times obtained from PySpark DQ2S Local mode using 7 cores in a single machine. 219 C.20 Execution times obtained from PySpark DQ2S Local mode using 8 cores in a single machine. 219 C.21 Execution times obtained from PySpark DQ2S Local mode using 1 to 6 cores in a single machine for 11 500 000 rows. 220 C.22 Execution times obtained from PySpark DQ2S Local mode using 7 to 10 cores in a single machine for 11 500 000 rows. 220 C.23 Execution times obtained from PySpark DQ2S Local mode using 1 to 6 cores in a single machine for 35 000 000 rows. 221 C.24 Execution times obtained from PySpark DQ2S Local mode using 7 to 10, 12, and 16 cores in a single machine for 35 000 000 rows. 221 C.25 Execution times obtained from PySpark DQ2S Standalone mode pro- cessing 35 000 000 rows with 1 to 4 worker nodes on a cluster.
Load more

Recommended publications
  • The Importance of Data
    The landscape of Parallel Programing Models Part 2: The importance of Data Michael Wong and Rod Burns Codeplay Software Ltd. Distiguished Engineer, Vice President of Ecosystem IXPUG 2020 2 © 2020 Codeplay Software Ltd. Distinguished Engineer Michael Wong ● Chair of SYCL Heterogeneous Programming Language ● C++ Directions Group ● ISOCPP.org Director, VP http://isocpp.org/wiki/faq/wg21#michael-wong ● [email protected] ● [email protected] Ported ● Head of Delegation for C++ Standard for Canada Build LLVM- TensorFlow to based ● Chair of Programming Languages for Standards open compilers for Council of Canada standards accelerators Chair of WG21 SG19 Machine Learning using SYCL Chair of WG21 SG14 Games Dev/Low Latency/Financial Trading/Embedded Implement Releasing open- ● Editor: C++ SG5 Transactional Memory Technical source, open- OpenCL and Specification standards based AI SYCL for acceleration tools: ● Editor: C++ SG1 Concurrency Technical Specification SYCL-BLAS, SYCL-ML, accelerator ● MISRA C++ and AUTOSAR VisionCpp processors ● Chair of Standards Council Canada TC22/SC32 Electrical and electronic components (SOTIF) ● Chair of UL4600 Object Tracking ● http://wongmichael.com/about We build GPU compilers for semiconductor companies ● C++11 book in Chinese: Now working to make AI/ML heterogeneous acceleration safe for https://www.amazon.cn/dp/B00ETOV2OQ autonomous vehicle 3 © 2020 Codeplay Software Ltd. Acknowledgement and Disclaimer Numerous people internal and external to the original C++/Khronos group, in industry and academia, have made contributions, influenced ideas, written part of this presentations, and offered feedbacks to form part of this talk. But I claim all credit for errors, and stupid mistakes. These are mine, all mine! You can’t have them.
    [Show full text]
  • (CITIUS) Phd DISSERTATION
    UNIVERSITY OF SANTIAGO DE COMPOSTELA DEPARTMENT OF ELECTRONICS AND COMPUTER SCIENCE CENTRO DE INVESTIGACION´ EN TECNOLOX´IAS DA INFORMACION´ (CITIUS) PhD DISSERTATION PERFORMANCE COUNTER-BASED STRATEGIES TO IMPROVE DATA LOCALITY ON MULTIPROCESSOR SYSTEMS: REORDERING AND PAGE MIGRATION TECHNIQUES Author: Juan Angel´ Lorenzo del Castillo PhD supervisors: Prof. Francisco Fernandez´ Rivera Dr. Juan Carlos Pichel Campos Santiago de Compostela, October 2011 Prof. Francisco Fernandez´ Rivera, professor at the Computer Architecture Group of the University of Santiago de Compostela Dr. Juan Carlos Pichel Campos, researcher at the Computer Architecture Group of the University of Santiago de Compostela HEREBY CERTIFY: That the dissertation entitled Performance Counter-based Strategies to Improve Data Locality on Multiprocessor Systems: Reordering and Page Migration Techniques has been developed by Juan Angel´ Lorenzo del Castillo under our direction at the Department of Electronics and Computer Science of the University of Santiago de Compostela in fulfillment of the requirements for the Degree of Doctor of Philosophy. Santiago de Compostela, October 2011 Francisco Fernandez´ Rivera, Profesor Catedratico´ de Universidad del Area´ de Arquitectura de Computadores de la Universidad de Santiago de Compostela Juan Carlos Pichel Campos, Profesor Doctor del Area´ de Arquitectura de Computadores de la Universidad de Santiago de Compostela HACEN CONSTAR: Que la memoria titulada Performance Counter-based Strategies to Improve Data Locality on Mul- tiprocessor Systems: Reordering and Page Migration Techniques ha sido realizada por D. Juan Angel´ Lorenzo del Castillo bajo nuestra direccion´ en el Departamento de Electronica´ y Computacion´ de la Universidad de Santiago de Compostela, y constituye la Tesis que presenta para optar al t´ıtulo de Doctor por la Universidad de Santiago de Compostela.
    [Show full text]
  • Intel® Oneapi Programming Guide
    Intel® oneAPI Programming Guide Intel Corporation www.intel.com Notices and Disclaimers Contents Notices and Disclaimers....................................................................... 5 Chapter 1: Introduction oneAPI Programming Model Overview ..........................................................7 Data Parallel C++ (DPC++)................................................................8 oneAPI Toolkit Distribution..................................................................9 About This Guide.......................................................................................9 Related Documentation ..............................................................................9 Chapter 2: oneAPI Programming Model Sample Program ..................................................................................... 10 Platform Model........................................................................................ 14 Execution Model ...................................................................................... 15 Memory Model ........................................................................................ 17 Memory Objects.............................................................................. 19 Accessors....................................................................................... 19 Synchronization .............................................................................. 20 Unified Shared Memory.................................................................... 20 Kernel Programming
    [Show full text]
  • Executable Modelling for Highly Parallel Accelerators
    Executable Modelling for Highly Parallel Accelerators Lorenzo Addazi, Federico Ciccozzi, Bjorn¨ Lisper School of Innovation, Design, and Engineering Malardalen¨ University -Vaster¨ as,˚ Sweden florenzo.addazi, federico.ciccozzi, [email protected] Abstract—High-performance embedded computing is develop- development are lagging behind. Programming accelerators ing rapidly since applications in most domains require a large and needs device-specific expertise in computer architecture and increasing amount of computing power. On the hardware side, low-level parallel programming for the accelerator at hand. this requirement is met by the introduction of heterogeneous systems, with highly parallel accelerators that are designed to Programming is error-prone and debugging can be very dif- take care of the computation-heavy parts of an application. There ficult due to potential race conditions: this is particularly is today a plethora of accelerator architectures, including GPUs, disturbing as many embedded applications, like autonomous many-cores, FPGAs, and domain-specific architectures such as AI vehicles, are safety-critical, meaning that failures may have accelerators. They all have their own programming models, which lethal consequences. Furthermore software becomes hard to are typically complex, low-level, and involve explicit parallelism. This yields error-prone software that puts the functional safety at port between different kinds of accelerators. risk, unacceptable for safety-critical embedded applications. In A convenient solution is to adopt a model-driven approach, this position paper we argue that high-level executable modelling where the computation-intense parts of the application are languages tailored for parallel computing can help in the software expressed in a high-level, accelerator-agnostic executable mod- design for high performance embedded applications.
    [Show full text]
  • Programming Multicores: Do Applications Programmers Need to Write Explicitly Parallel Programs?
    [3B2-14] mmi2010030003.3d 11/5/010 16:48 Page 2 .......................................................................................................................................................................................................................... PROGRAMMING MULTICORES: DO APPLICATIONS PROGRAMMERS NEED TO WRITE EXPLICITLY PARALLEL PROGRAMS? .......................................................................................................................................................................................................................... IN THIS PANEL DISCUSSION FROM THE 2009 WORKSHOP ON COMPUTER ARCHITECTURE Arvind RESEARCH DIRECTIONS,DAVID AUGUST AND KESHAV PINGALI DEBATE WHETHER EXPLICITLY Massachusetts Institute PARALLEL PROGRAMMING IS A NECESSARY EVIL FOR APPLICATIONS PROGRAMMERS, of Technology ASSESS THE CURRENT STATE OF PARALLEL PROGRAMMING MODELS, AND DISCUSS David August POSSIBLE ROUTES TOWARD FINDING THE PROGRAMMING MODEL FOR THE MULTICORE ERA. Princeton University Keshav Pingali Moderator’s introduction: Arvind in the 1970s and 1980s, when two main Do applications programmers need to write approaches were developed. Derek Chiou explicitly parallel programs? Most people be- The first approach required that the com- lieve that the current method of parallel pro- pilers do all the work in finding the parallel- University of Texas gramming is impeding the exploitation of ism. This was often referred to as the ‘‘dusty multicores. In other words, the number of decks’’ problem—that is, how to
    [Show full text]
  • BARC Score Enterprise BI and Analytics Platforms
    BARC Score Enterprise BI and Analytics Platforms Authors: Larissa Seidler, Christian Fuchs, Patrick Keller, Carsten Bange, Robert Tischler Publication: September 8th, 2017 Abstract This BARC document is the third edition of our BARC Score business intelligence vendor evaluation and ranking. This BARC Score evaluates enterprise BI and analytics platforms that are able to fulfill a broad set of BI requirements within the enterprise. Based on countless data points from The BI Survey and many analyst interactions, vendors are rated on a variety of criteria, from product capabilities and architecture to sales and marketing strategy, financial performance and customer feedback. This document is not to be shared, distributed or reproduced in any way without prior permission of BARC Table of Contents Overview ...................................................................................................................................................3 Inclusion Criteria .......................................................................................................................................3 Evaluation Criteria ....................................................................................................................................4 Portfolio Capabilities...................................................................................................................... 4 Market Execution ........................................................................................................................... 7 Score
    [Show full text]
  • Customer Engagement Summit 2016
    OFFICIAL CUSTOMER EVENT ENGAGEMENT GUIDE SUMMIT 2016 TUESDAY, 8 NOVEMBER 2016 WESTMINSTER PARK PLAZA, LONDON TECHNOLOGY THE GREAT ENABLER PLATINUM GOLD ORGANISED BY: CustomerEngagementSummit.com @EngageCustomer #EngageSummits Find out more: www.verint.com/customer-engagement THE TEAM WELCOME Steve Hurst Editorial Director E: [email protected] T: 01932 506 304 Nick Rust WELCOME Sales Director E: [email protected] T: 01932 506 301 James Cottee Sponsorship Sales E: [email protected] T: 01932 506 309 Claire O’Brien Sponsorship Sales E: [email protected] T: 01932 506 308 Robert Cox A warm welcome to our fifth Customer Engagement Sponsorship Sales E: [email protected] Summit, now firmly established as Europe’s premier T: 01932 302 110 and most highly regarded customer and employee Katie Donaldson Marketing Executive engagement conference. E: [email protected] T: 01932 506 302 Our flagship Summit has gained a reputation for delivering leading-edge world class case-study led content in a business-like yet informal atmosphere which is conducive to delegate, speaker Claire Poole and sponsor networking. It’s a place both to learn and to do business. The overarching theme of Conference Producer this year’s Summit is ‘Technology the Great Enabler’ as we continue to recognise the critical E: [email protected] role that new technologies and innovations are playing in the fields of customer and employee T: 01932 506 300 engagement. Dan Keen We have several new elements to further improve this year’s Summit – not least our move to Delegate Sales E: [email protected] this new bigger venue, the iconic London Westminster Bridge Park Plaza Hotel following on T: 01932 506 306 from four hugely successful Summits at the London Victoria Park Plaza.
    [Show full text]
  • An Outlook of High Performance Computing Infrastructures for Scientific Computing
    An Outlook of High Performance Computing Infrastructures for Scientific Computing [This draft has mainly been extracted from the PhD Thesis of Amjad Ali at Center for Advanced Studies in Pure and Applied Mathematics (CASPAM), Bahauddin Zakariya University, Multan, Pakitsan 60800. ([email protected])] In natural sciences, two conventional ways of carrying out studies and reserach are: theoretical and experimental.Thefirstapproachisabouttheoraticaltreatmment possibly dealing with the mathematical models of the concerning physical phenomena and analytical solutions. The other approach is to perform physical experiments to carry out studies. This approach is directly favourable for developing the products in engineering. The theoretical branch, although very rich in concepts, has been able to develop only a few analytical methods applicable to rather simple problems. On the other hand the experimental branch involves heavy costs and specific arrangements. These shortcomings with the two approaches motivated the scientists to look around for a third or complementary choice, i.e., the use of numerical simulations in science and engineering. The numerical solution of scientific problems, by its very nature, requires high computational power. As the world has been continuously en- joying substantial growth in computer technologies during the past three decades, the numerical approach appeared to be more and more pragmatic. This constituted acompletenewbranchineachfieldofscience,oftentermedasComputational Science,thatisaboutusingnumericalmethodsforsolvingthemathematicalformu- lations of the science problems. Developments in computational sciences constituted anewdiscipline,namelytheScientific Computing,thathasemergedwiththe spirit of obtaining qualitative predictions through simulations in ‘virtual labora- tories’ (i.e., the computers) for all areas of science and engineering. The scientific computing, due to its nature, exists at the overlapping region of the three disciplines, mathematical modeling, numerical mathematics and computer science,asshownin Fig.
    [Show full text]
  • Performance Tuning Workshop
    Performance Tuning Workshop Samuel Khuvis Scientifc Applications Engineer, OSC February 18, 2021 1/103 Workshop Set up I Workshop – set up account at my.osc.edu I If you already have an OSC account, sign in to my.osc.edu I Go to Project I Project access request I PROJECT CODE = PZS1010 I Reset your password I Slides are the workshop website: https://www.osc.edu/~skhuvis/opt21_spring 2/103 Outline I Introduction I Debugging I Hardware overview I Performance measurement and analysis I Help from the compiler I Code tuning/optimization I Parallel computing 3/103 Introduction 4/103 Workshop Philosophy I Aim for “reasonably good” performance I Discuss performance tuning techniques common to most HPC architectures I Compiler options I Code modifcation I Focus on serial performance I Reduce time spent accessing memory I Parallel processing I Multithreading I MPI 5/103 Hands-on Code During this workshop, we will be using a code based on the HPCCG miniapp from Mantevo. I Performs Conjugate Gradient (CG) method on a 3D chimney domain. I CG is an iterative algorithm to numerically approximate the solution to a system of linear equations. I Run code with srun -n <numprocs> ./test_HPCCG nx ny nz, where nx, ny, and nz are the number of nodes in the x, y, and z dimension on each processor. I Download with: wget go . osu .edu/perftuning21 t a r x f perftuning21 I Make sure that the following modules are loaded: intel/19.0.5 mvapich2/2.3.3 6/103 More important than Performance! I Correctness of results I Code readability/maintainability I Portability -
    [Show full text]
  • Regent: a High-Productivity Programming Language for Implicit Parallelism with Logical Regions
    REGENT: A HIGH-PRODUCTIVITY PROGRAMMING LANGUAGE FOR IMPLICIT PARALLELISM WITH LOGICAL REGIONS A DISSERTATION SUBMITTED TO THE DEPARTMENT OF COMPUTER SCIENCE AND THE COMMITTEE ON GRADUATE STUDIES OF STANFORD UNIVERSITY IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY Elliott Slaughter August 2017 © 2017 by Elliott David Slaughter. All Rights Reserved. Re-distributed by Stanford University under license with the author. This work is licensed under a Creative Commons Attribution- Noncommercial 3.0 United States License. http://creativecommons.org/licenses/by-nc/3.0/us/ This dissertation is online at: http://purl.stanford.edu/mw768zz0480 ii I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Alex Aiken, Primary Adviser I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Philip Levis I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Oyekunle Olukotun Approved for the Stanford University Committee on Graduate Studies. Patricia J. Gumport, Vice Provost for Graduate Education This signature page was generated electronically upon submission of this dissertation in electronic format. An original signed hard copy of the signature page is on file in University Archives. iii Abstract Modern supercomputers are dominated by distributed-memory machines. State of the art high-performance scientific applications targeting these machines are typically written in low-level, explicitly parallel programming models that enable maximal performance but expose the user to programming hazards such as data races and deadlocks.
    [Show full text]
  • Introduction to Parallel Processing
    IntroductionIntroduction toto ParallelParallel ProcessingProcessing • Parallel Computer Architecture: Definition & Broad issues involved – A Generic Parallel ComputerComputer Architecture • The Need And Feasibility of Parallel Computing Why? – Scientific Supercomputing Trends – CPU Performance and Technology Trends, Parallelism in Microprocessor Generations – Computer System Peak FLOP Rating History/Near Future • The Goal of Parallel Processing • Elements of Parallel Computing • Factors Affecting Parallel System Performance • Parallel Architectures History – Parallel Programming Models – Flynn’s 1972 Classification of Computer Architecture • Current Trends In Parallel Architectures – Modern Parallel Architecture Layered Framework • Shared Address Space Parallel Architectures • Message-Passing Multicomputers: Message-Passing Programming Tools • Data Parallel Systems • Dataflow Architectures • Systolic Architectures: Matrix Multiplication Systolic Array Example PCA Chapter 1.1, 1.2 EECC756 - Shaaban #1 lec # 1 Spring 2012 3-13-2012 ParallelParallel ComputerComputer ArchitectureArchitecture A parallel computer (or multiple processor system) is a collection of communicating processing elements (processors) that cooperate to solve large computational problems fast by dividing such problems into parallel tasks, exploiting Thread-Level Parallelism (TLP). i.e Parallel Processing • Broad issues involved: Task = Computation done on one processor – The concurrency and communication characteristics of parallel algorithms for a given computational
    [Show full text]
  • Insight MFR By
    Manufacturers, Publishers and Suppliers by Product Category 11/6/2017 10/100 Hubs & Switches ASCEND COMMUNICATIONS CIS SECURE COMPUTING INC DIGIUM GEAR HEAD 1 TRIPPLITE ASUS Cisco Press D‐LINK SYSTEMS GEFEN 1VISION SOFTWARE ATEN TECHNOLOGY CISCO SYSTEMS DUALCOMM TECHNOLOGY, INC. GEIST 3COM ATLAS SOUND CLEAR CUBE DYCONN GEOVISION INC. 4XEM CORP. ATLONA CLEARSOUNDS DYNEX PRODUCTS GIGAFAST 8E6 TECHNOLOGIES ATTO TECHNOLOGY CNET TECHNOLOGY EATON GIGAMON SYSTEMS LLC AAXEON TECHNOLOGIES LLC. AUDIOCODES, INC. CODE GREEN NETWORKS E‐CORPORATEGIFTS.COM, INC. GLOBAL MARKETING ACCELL AUDIOVOX CODI INC EDGECORE GOLDENRAM ACCELLION AVAYA COMMAND COMMUNICATIONS EDITSHARE LLC GREAT BAY SOFTWARE INC. ACER AMERICA AVENVIEW CORP COMMUNICATION DEVICES INC. EMC GRIFFIN TECHNOLOGY ACTI CORPORATION AVOCENT COMNET ENDACE USA H3C Technology ADAPTEC AVOCENT‐EMERSON COMPELLENT ENGENIUS HALL RESEARCH ADC KENTROX AVTECH CORPORATION COMPREHENSIVE CABLE ENTERASYS NETWORKS HAVIS SHIELD ADC TELECOMMUNICATIONS AXIOM MEMORY COMPU‐CALL, INC EPIPHAN SYSTEMS HAWKING TECHNOLOGY ADDERTECHNOLOGY AXIS COMMUNICATIONS COMPUTER LAB EQUINOX SYSTEMS HERITAGE TRAVELWARE ADD‐ON COMPUTER PERIPHERALS AZIO CORPORATION COMPUTERLINKS ETHERNET DIRECT HEWLETT PACKARD ENTERPRISE ADDON STORE B & B ELECTRONICS COMTROL ETHERWAN HIKVISION DIGITAL TECHNOLOGY CO. LT ADESSO BELDEN CONNECTGEAR EVANS CONSOLES HITACHI ADTRAN BELKIN COMPONENTS CONNECTPRO EVGA.COM HITACHI DATA SYSTEMS ADVANTECH AUTOMATION CORP. BIDUL & CO CONSTANT TECHNOLOGIES INC Exablaze HOO TOO INC AEROHIVE NETWORKS BLACK BOX COOL GEAR EXACQ TECHNOLOGIES INC HP AJA VIDEO SYSTEMS BLACKMAGIC DESIGN USA CP TECHNOLOGIES EXFO INC HP INC ALCATEL BLADE NETWORK TECHNOLOGIES CPS EXTREME NETWORKS HUAWEI ALCATEL LUCENT BLONDER TONGUE LABORATORIES CREATIVE LABS EXTRON HUAWEI SYMANTEC TECHNOLOGIES ALLIED TELESIS BLUE COAT SYSTEMS CRESTRON ELECTRONICS F5 NETWORKS IBM ALLOY COMPUTER PRODUCTS LLC BOSCH SECURITY CTC UNION TECHNOLOGIES CO FELLOWES ICOMTECH INC ALTINEX, INC.
    [Show full text]