DOCSLIB.ORG

Parallel Processing in Python

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Parallel Processing in Python PARALLEL PROCESSING IN PYTHON COSMOS - 1/28/2020 BY JOSEPH KREADY LAYOUT What is Parallel Parallel Processing Processing and Python History of Google Colab Parallel Example Computation Parallel Processing: Multiple objects at a time Serial Processing: One object at a time SERIAL PROCESSING VS. PARALLEL PROCESSING CPU ARCHITECTURE Serial Computation The method used for improving computer performance from 1980s – 2000s FREQUENCY Frequency Scaling Equation: Power consumption = capacitance * SCALING voltage^2 * frequency The increase to power consumption led to the demise of frequency scaling SUPERSCALAR ARCHITECTURE Executes more than one instruction during a clock cycle by simultaneously dispatching multiple instructions to different execution units on the processor. Each CPU is made of independent ‘Cores’ that can access the same memory concurrently MULTI-CORE Moore’s Law: The number of cores per PROCESSORS processor doubles every 18-24 months Operating systems ensure programs run on available cores, but developers must design their programs to take advantage of parallel processing. PROCESS VS. THREAD VS. MULTI-THREADING VS. HYPER-THREADING ¡ Processes are made of threads ¡ Threads of the same processes share memory. ¡ Processes run in separate memory. ¡ Hyper-threading: allows scheduling of 2 processes on 1 CPU core ¡ Multiple Instructions operate on separate data in parallel THE GLOBAL INTERPRETER LOCK!!! ¡ Global Interpreter lock (GIL): All Python processes must go through the GIL to execute; thus threads execute 1 at a time ¡ It is faster in the single-threaded case. ¡ It is faster in the multi-threaded case for i/o bound programs. (they are not GIL locked) ¡ It is faster in the multi-threaded case for CPU-bound programs that do their compute-intensive work in C libraries, ie Numpy. ¡ GIL only becomes a problem when doing CPU-intensive work in pure Python. ¡ Not all versions of Python use a GIL: Jython, IronPython, PyPy-STM YOUR CHOICES MULTI-THREADING MULTI-PROCESSING ¡ Threads act as sub-tasks of a single process ¡ Separate processes act as individual jobs ¡ Threads share the same memory space ¡ Processes run in their own memory space ¡ Great for background tasks / waiting for ¡ Great for complex calculations / running multiple asynchronous functions instances of a whole project ¡ Can lead to conflicts (Race Conditions), when ¡ Higher overhead, but more secure writing to the same memory location at the same time WHY DON’T WE JUST USE TREADS? Problem ¡ Race Conditions: Multiple threads reading and writing to the same object Will cause unexpected results ¡ The operating system handles the processing of threads dynamically. There’s no way to ensure the compute order. Solution ¡ Synchronization using Lock: You can define one part of your function ‘with thread.lock:’ which requires 1 thread processing at a time. ¡ This can lead to Deadlocks: where the lock is not released properly, or you call sub functions that are already locked in another thread WHY DON’T WE JUST USE PROCESSES? Problem ¡ Serialization using Pickles: converting python objects to byte streams ¡ Individual processes run in separate memory spaces and need a way to communicate. This is done through Pickles. ¡ Certain limitations, like limited function arguments and no class supported with pickling. Therefore, you must design your functions with pickling in mind. Solution ¡ Serialization using Dill ¡ Dill extends Pickles, allowing to send arbitrary classes and functions as byte streams. Dill can Pickle all the python objects! ¡ Pathos.Multiprocessing library is a fork of python’s multi- processing that uses Dills instead of Pickles. AMDAHL’S LAW ¡ The small part of a program that cannot be parallelized will limit the overall speedup ¡ S-latency is the potential speedup in latency of the execution of the whole task; ¡ s is the speedup in latency of the execution of the parallelizable part of the task; ¡ p is the percentage of the execution time of the whole task concerning the parallelizable part of the task before parallelization. https://colab.research.google.com/drive/1 TkjjiIrzq5wE1BF2DbOAqTKhmRgzSYVh Real Python. “An Intro to Threading in Python.” Real Python, Real Python, 25 May 2019, Works Cited realpython.com/intro-to-python-threading/. Accessed 31 Jan. 2020. “An Introduction to Parallel Programming Using Python's Multiprocessing Module.” Dr. Sebastian Raschka, 20 June 2014, Rocklin, Matthew. “Parallelism and Serialization How Poor Pickling Breaks Multiprocessing.” sebastianraschka.com/Articles/2014_multiprocessing.html. Accessed 31 Jan. 2020. “Dill.” PyPI, pypi.org/project/dill/. Accessed 31 Jan. 2020. Parallelism and Serialization, matthewrocklin.com/blog/work/2013/12/05/Parallelism- FomiteFomite 2, et al. “Why Was Python Written with the GIL?” Software Engineering Stack and-Serialization. Accessed 31 Jan. 2020. Exchange, 1 Feb. 1963, softwareengineering.stackexchange.com/questions/186889/why- was-python-written-with-the-gil. Accessed 31 Jan. 2020. “Superscalar Processor.” Wikipedia, Wikimedia Foundation, 7 May 2019, “Has the Python GIL Been Slain?” By, hackernoon.com/has-the-python-gil-been-slain- 9440d28fa93d. Accessed 31 Jan. 2020. en.wikipedia.org/wiki/Superscalar_processor. Accessed 31 Jan. 2020. “Hyper-Threading.” Wikipedia, Wikimedia Foundation, 19 Jan. 2020, en.wikipedia.org/wiki/Hyper-threading. Accessed 31 Jan. 2020. “Multithreading (Computer Architecture).” Wikipedia, Wikimedia Foundation, 2 Jan. 2020, en.wikipedia.org/wiki/Multithreading_(computer_architecture). Accessed 31 Jan. 2020. “Parallel Computing.” Wikipedia, Wikimedia Foundation, 26 Dec. 2019, en.wikipedia.org/wiki/Parallel_computing. Accessed 31 Jan. 2020. REFERENCES “Pickle - Python Object Serialization¶.” Pickle - Python Object Serialization - Python 3.8.1 Documentation, docs.python.org/3/library/pickle.html. Accessed 31 Jan. 2020. .
Load more

Recommended publications
  • Ironpython in Action
    IronPytho IN ACTION Michael J. Foord Christian Muirhead FOREWORD BY JIM HUGUNIN MANNING IronPython in Action Download at Boykma.Com Licensed to Deborah Christiansen <[email protected]> Download at Boykma.Com Licensed to Deborah Christiansen <[email protected]> IronPython in Action MICHAEL J. FOORD CHRISTIAN MUIRHEAD MANNING Greenwich (74° w. long.) Download at Boykma.Com Licensed to Deborah Christiansen <[email protected]> For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. Sound View Court 3B fax: (609) 877-8256 Greenwich, CT 06830 email: [email protected] ©2009 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15% recycled and processed without the use of elemental chlorine.
    [Show full text]
  • Multiprocessing Contents
    Multiprocessing Contents 1 Multiprocessing 1 1.1 Pre-history .............................................. 1 1.2 Key topics ............................................... 1 1.2.1 Processor symmetry ...................................... 1 1.2.2 Instruction and data streams ................................. 1 1.2.3 Processor coupling ...................................... 2 1.2.4 Multiprocessor Communication Architecture ......................... 2 1.3 Flynn’s taxonomy ........................................... 2 1.3.1 SISD multiprocessing ..................................... 2 1.3.2 SIMD multiprocessing .................................... 2 1.3.3 MISD multiprocessing .................................... 3 1.3.4 MIMD multiprocessing .................................... 3 1.4 See also ................................................ 3 1.5 References ............................................... 3 2 Computer multitasking 5 2.1 Multiprogramming .......................................... 5 2.2 Cooperative multitasking ....................................... 6 2.3 Preemptive multitasking ....................................... 6 2.4 Real time ............................................... 7 2.5 Multithreading ............................................ 7 2.6 Memory protection .......................................... 7 2.7 Memory swapping .......................................... 7 2.8 Programming ............................................. 7 2.9 See also ................................................ 8 2.10 References .............................................
    [Show full text]
  • Multiprocessing and Scalability
    Multiprocessing and Scalability A.R. Hurson Computer Science and Engineering The Pennsylvania State University 1 Multiprocessing and Scalability Large-scale multiprocessor systems have long held the promise of substantially higher performance than traditional uni- processor systems. However, due to a number of difficult problems, the potential of these machines has been difficult to realize. This is because of the: Fall 2004 2 Multiprocessing and Scalability Advances in technology ─ Rate of increase in performance of uni-processor, Complexity of multiprocessor system design ─ This drastically effected the cost and implementation cycle. Programmability of multiprocessor system ─ Design complexity of parallel algorithms and parallel programs. Fall 2004 3 Multiprocessing and Scalability Programming a parallel machine is more difficult than a sequential one. In addition, it takes much effort to port an existing sequential program to a parallel machine than to a newly developed sequential machine. Lack of good parallel programming environments and standard parallel languages also has further aggravated this issue. Fall 2004 4 Multiprocessing and Scalability As a result, absolute performance of many early concurrent machines was not significantly better than available or soon- to-be available uni-processors. Fall 2004 5 Multiprocessing and Scalability Recently, there has been an increased interest in large-scale or massively parallel processing systems. This interest stems from many factors, including: Advances in integrated technology. Very
    [Show full text]
  • Design of a Message Passing Interface for Multiprocessing with Atmel Microcontrollers
    DESIGN OF A MESSAGE PASSING INTERFACE FOR MULTIPROCESSING WITH ATMEL MICROCONTROLLERS A Design Project Report Presented to the Engineering Division of the Graduate School of Cornell University in Partial Fulfillment of the Requirements of the Degree of Master of Engineering (Electrical) by Kalim Moghul Project Advisor: Dr. Bruce R. Land Degree Date: May 2006 Abstract Master of Electrical Engineering Program Cornell University Design Project Report Project Title: Design of a Message Passing Interface for Multiprocessing with Atmel Microcontrollers Author: Kalim Moghul Abstract: Microcontrollers are versatile integrated circuits typically incorporating a microprocessor, memory, and I/O ports on a single chip. These self-contained units are central to embedded systems design where low-cost, dedicated processors are often preferred over general-purpose processors. Embedded designs may require several microcontrollers, each with a specific task. Efficient communication is central to such systems. The focus of this project is the design of a communication layer and application programming interface for exchanging messages among microcontrollers. In order to demonstrate the library, a small- scale cluster computer is constructed using Atmel ATmega32 microcontrollers as processing nodes and an Atmega16 microcontroller for message routing. The communication library is integrated into aOS, a preemptive multitasking real-time operating system for Atmel microcontrollers. Report Approved by Project Advisor:__________________________________ Date:___________ Executive Summary Microcontrollers are versatile integrated circuits typically incorporating a microprocessor, memory, and I/O ports on a single chip. These self-contained units are central to embedded systems design where low-cost, dedicated processors are often preferred over general-purpose processors. Some embedded designs may incorporate several microcontrollers, each with a specific task.
    [Show full text]
  • CIS 192: Lecture 12 Deploying Apps and Concurrency
    CIS 192: Lecture 12 Deploying Apps and Concurrency Lili Dworkin University of Pennsylvania Good Question from Way Back I All HTTP requests have 1) URL, 2) headers, 3) body I GET requests: parameters sent in URL I POST requests: parameters sent in body Can GET requests have a body? StackOverflow's response: \Yes, you can send a request body with GET but it should not have any meaning. If you give it meaning by parsing it on the server and changing your response based on its contents you're violating the HTTP/1.1 spec." Good Question from Last Week What is the difference between jsonify and json.dumps? def jsonify(*args, **kwargs): if __debug__: _assert_have_json() return current_app.response_class(json.dumps(dict(* args, **kwargs), indent=None if request.is_xhr else 2), mimetype='application/json') I jsonify returns a Response object I jsonify automatically sets content-type header I jsonify also sets the indentation Review Find a partner! Deploying Apps I We've been running Flask apps locally on a builtin development server I When you're ready to go public, you need to deploy to a production server I Easiest option: use one hosted by someone else! I We'll use Heroku, a platform as a service (PaaS) that makes it easy to deploy apps in a variety of languages Heroku Prerequisites: I Virtualenv (creates standalone Python environments) I Heroku toolbox I Heroku command-line client I Git (for version control and pushing to Heroku) Virtualenv I Allows us to create a virtual Python environment I Unique, isolated environment for each project I Use case: different versions of packages for different projects Virtualenv How to use it? prompt$ pip install virtualenv Now navigate to your project directory: prompt$ virtualenv --no-site-packages venv prompt$ source venv/bin/activate (<name>)prompt$ pip install Flask gunicorn (<name>)prompt$ deactivate prompt% Heroku Toolbox Once you make a Heroku account, install the Heroku toolbox.
    [Show full text]
  • Threading and GUI Issues for R
    Threading and GUI Issues for R Luke Tierney School of Statistics University of Minnesota March 5, 2001 Contents 1 Introduction 2 2 Concurrency and Parallelism 2 3 Concurrency and Dynamic State 3 3.1 Options Settings . 3 3.2 User Defined Options . 5 3.3 Devices and Par Settings . 5 3.4 Standard Connections . 6 3.5 The Context Stack . 6 3.5.1 Synchronization . 6 4 GUI Events And Blocking IO 6 4.1 UNIX Issues . 7 4.2 Win32 Issues . 7 4.3 Classic MacOS Issues . 8 4.4 Implementations To Consider . 8 4.5 A Note On Java . 8 4.6 A Strategy for GUI/IO Management . 9 4.7 A Sample Implementation . 9 5 Threads and GUI’s 10 6 Threading Design Space 11 6.1 Parallelism Through HL Threads: The MXM Options . 12 6.2 Light-Weight Threads: The XMX Options . 12 6.3 Multiple OS Threads Running One At A Time: MSS . 14 6.4 Variations on OS Threads . 14 6.5 SMS or MXS: Which To Choose? . 14 7 Light-Weight Thread Implementation 14 1 March 5, 2001 2 8 Other Issues 15 8.1 High-Level GUI Interfaces . 16 8.2 High-Level Thread Interfaces . 16 8.3 High-Level Streams Interfaces . 16 8.4 Completely Random Stuff . 16 1 Introduction This document collects some random thoughts on runtime issues relating to concurrency, threads, GUI’s and the like. Some of this is extracted from recent R-core email threads. I’ve tried to provide lots of references that might be of use.
    [Show full text]
  • Piotr Warczak Quarter: Fall 2011 Student ID: 99XXXXX Credit: 2 Grading: Decimal
    CSS 600: Independent Study Contract ­ Final Report Student: Piotr Warczak Quarter: Fall 2011 Student ID: 99XXXXX Credit: 2 Grading: Decimal Independent Study Title The GPU version of the MASS library. Focus and Goals The current version of the MASS library is written in java programming language and combines the power of every computing node on the network by utilizing both multithreading and multiprocessing. The new version will be implemented in C and C++. It will also support multithreading and multiprocessing and finally CUDA parallel computing architecture utilizing both single and multiple devices. This version will harness the power of the GPU a general purpose parallel processor. My specific goals of the past quarter were: • To create a baseline in C language using Wave2D application as an example. • To implement single thread Wave2D • To implement multithreading Wave2D • To implement CUDA enabled Wave2D Work Completed During this quarter I have created Wave2D application in C language using a single thread, multithreaded and CUDA enabled application. The reason for this is that CUDA is an extension of C and thus we need to create baseline against which other versions of the program can be measured. This baseline illustrates how much faster the CUDA enabled program is versus single thread and multithreaded versions. Furthermore, once the GPU version of the MASS library is implemented, we can compare the results to identify if there is any difference in program’s total execution time due to the potential MASS library overhead. And if any major delays in execution are found, the problem ares will be identified and corrected.
    [Show full text]
  • Due to Global Interpreter Lock (GIL), Python Threads Do Not Provide Efficient Multi-Core Execution, Unlike Other Languages Such As Golang
    Due to global interpreter lock (GIL), python threads do not provide efficient multi-core execution, unlike other languages such as golang. Asynchronous programming in python is focusing on single core execution. Happily, Nexedi python code base already supports high performance multi-core execution either through multiprocessing with distributed shared memory or by relying on components (NumPy, MariaDB) that already support multiple core execution. The impatient reader will find bellow a table that summarises current options to achieve high performance multi-core performance in python. High Performance multi-core Python Cheat Sheet Use python whenever you can rely on any of... Use golang whenever you need at the same time... low latency multiprocessing high concurrency cython's nogil option multi-core execution within single userspace shared multi-core execution within wendelin.core distributed memory shared memory something else than cython's parallel module or cython's parallel module nogil option Here are a set of rules to achieve high performance on a multi-core system in the context of Nexedi stack. use ERP5's CMFActivity by default, since it can solve 99% of your concurrency and performance problems on a cluster or multi-core system; use parallel module in cython if you need to use all cores with high performance within a single transaction (ex. HTTP request); use threads in python together with nogil option in cython in order to use all cores on a multithreaded python process (ex. Zope); use cython to achieve C/C++ performance without mastering C/C++ as long as you do not need multi-core asynchronous programming; rewrite some python in cython to get performance boost on selected portions of python libraries; use numba to accelerate selection portions of python code with JIT that may also work in a dynamic context (ex.
    [Show full text]
  • Specialising Dynamic Techniques for Implementing the Ruby Programming Language
    SPECIALISING DYNAMIC TECHNIQUES FOR IMPLEMENTING THE RUBY PROGRAMMING LANGUAGE A thesis submitted to the University of Manchester for the degree of Doctor of Philosophy in the Faculty of Engineering and Physical Sciences 2015 By Chris Seaton School of Computer Science This published copy of the thesis contains a couple of minor typographical corrections from the version deposited in the University of Manchester Library. [email protected] chrisseaton.com/phd 2 Contents List of Listings7 List of Tables9 List of Figures 11 Abstract 15 Declaration 17 Copyright 19 Acknowledgements 21 1 Introduction 23 1.1 Dynamic Programming Languages.................. 23 1.2 Idiomatic Ruby............................ 25 1.3 Research Questions.......................... 27 1.4 Implementation Work......................... 27 1.5 Contributions............................. 28 1.6 Publications.............................. 29 1.7 Thesis Structure............................ 31 2 Characteristics of Dynamic Languages 35 2.1 Ruby.................................. 35 2.2 Ruby on Rails............................. 36 2.3 Case Study: Idiomatic Ruby..................... 37 2.4 Summary............................... 49 3 3 Implementation of Dynamic Languages 51 3.1 Foundational Techniques....................... 51 3.2 Applied Techniques.......................... 59 3.3 Implementations of Ruby....................... 65 3.4 Parallelism and Concurrency..................... 72 3.5 Summary............................... 73 4 Evaluation Methodology 75 4.1 Evaluation Philosophy
    [Show full text]
  • Multithreaded Programming L-35 8April2016 1/41 Lifecycle of a Thread Multithreaded Programming
    Outline 1 Concurrent Processes processes and threads life cycle of a thread thread safety, critical sections, and deadlock 2 Multithreading in Python the thread module the Thread class 3 Producer Consumer Relation object-oriented design classes producer and consumer MCS 260 Lecture 35 Introduction to Computer Science Jan Verschelde, 8 April 2016 Intro to Computer Science (MCS 260) multithreaded programming L-35 8April2016 1/41 lifecycle of a thread multithreaded programming 1 Concurrent Processes processes and threads life cycle of a thread thread safety, critical sections, and deadlock 2 Multithreading in Python the thread module the Thread class 3 Producer Consumer Relation object-oriented design classes producer and consumer Intro to Computer Science (MCS 260) multithreaded programming L-35 8April2016 2/41 concurrency and parallelism First some terminology: concurrency Concurrent programs execute multiple tasks independently. For example, a drawing application, with tasks: ◮ receiving user input from the mouse pointer, ◮ updating the displayed image. parallelism A parallel program executes two or more tasks in parallel with the explicit goal of increasing the overall performance. For example: a parallel Monte Carlo simulation for π, written with the multiprocessing module of Python. Every parallel program is concurrent, but not every concurrent program executes in parallel. Intro to Computer Science (MCS 260) multithreaded programming L-35 8April2016 3/41 Parallel Processing processes and threads At any given time, many processes are running simultaneously on a computer. The operating system employs time sharing to allocate a percentage of the CPU time to each process. Consider for example the downloading of an audio file. Instead of having to wait till the download is complete, we would like to listen sooner.
    [Show full text]
  • Message Passing Fundamentals
    1. Message Passing Fundamentals Message Passing Fundamentals As a programmer, you may find that you need to solve ever larger, more memory intensive problems, or simply solve problems with greater speed than is possible on a serial computer. You can turn to parallel programming and parallel computers to satisfy these needs. Using parallel programming methods on parallel computers gives you access to greater memory and Central Processing Unit (CPU) resources not available on serial computers. Hence, you are able to solve large problems that may not have been possible otherwise, as well as solve problems more quickly. One of the basic methods of programming for parallel computing is the use of message passing libraries. These libraries manage transfer of data between instances of a parallel program running (usually) on multiple processors in a parallel computing architecture. The topics to be discussed in this chapter are The basics of parallel computer architectures. The difference between domain and functional decomposition. The difference between data parallel and message passing models. A brief survey of important parallel programming issues. 1.1. Parallel Architectures Parallel Architectures Parallel computers have two basic architectures: distributed memory and shared memory. Distributed memory parallel computers are essentially a collection of serial computers (nodes) working together to solve a problem. Each node has rapid access to its own local memory and access to the memory of other nodes via some sort of communications network, usually a proprietary high-speed communications network. Data are exchanged between nodes as messages over the network. In a shared memory computer, multiple processor units share access to a global memory space via a high-speed memory bus.
    [Show full text]
  • Composable Multi-Threading and Multi-Processing for Numeric Libraries
    18 PROC. OF THE 17th PYTHON IN SCIENCE CONF. (SCIPY 2018) Composable Multi-Threading and Multi-Processing for Numeric Libraries Anton Malakhov‡∗, David Liu‡, Anton Gorshkov‡†, Terry Wilmarth‡ https://youtu.be/HKjM3peINtw F Abstract—Python is popular among scientific communities that value its sim- Scaling parallel programs is challenging. There are two fun- plicity and power, especially as it comes along with numeric libraries such as damental laws which mathematically describe and predict scala- [NumPy], [SciPy], [Dask], and [Numba]. As CPU core counts keep increasing, bility of a program: Amdahl’s Law and Gustafson-Barsis’ Law these modules can make use of many cores via multi-threading for efficient [AGlaws]. According to Amdahl’s Law, speedup is limited by multi-core parallelism. However, threads can interfere with each other leading the serial portion of the work, which effectively puts a limit on to overhead and inefficiency if used together in a single application on machines scalability of parallel processing for a fixed-size job. Python is with a large number of cores. This performance loss can be prevented if all multi-threaded modules are coordinated. This paper continues the work started especially vulnerable to this because it makes the serial part of in [AMala16] by introducing more approaches to coordination for both multi- the same code much slower compared to implementations in other threading and multi-processing cases. In particular, we investigate the use of languages due to its deeply dynamic and interpretative nature. In static settings, limiting the number of simultaneously active [OpenMP] parallel addition, the GIL serializes operations that could be potentially regions, and optional parallelism with Intel® Threading Building Blocks (Intel® executed in parallel, further adding to the serial portion of a [TBB]).
    [Show full text]