Concurrency: what, why, how

Oleg Batrashev

February 10, 2014 Course description

Course ■ description what this course is about? Overview What is ◆ additional experience with concurrency concurrency ◆ try out some language concepts and techniques Reasons for concurrency ■ java.util.concurrent.Future? Classifications of concurrency ■ agents, software transactional memory (STM) Languages for Concurrent ■ programming Grading ◆ ~5 lab assignments: 20% (1 Java + 4 Clojure) ◆ home assignments: 2×15% (2 Clojure)

■ declarative concurrency/agents... ■ STM/distributed program... ◆ exam: 50% (5-10 questions/exercises for 1-2 hours) ■ Languages: Clojure, Oz, Haskell, Erlang, Scala, Go

Concurrency: what, why, how 2 / 35 Overview

Course ■ description Lecture about everything and nothing in particular Overview ■ Explain concurrency What is concurrency ◆ basic idea Reasons for concurrency ◆ concurrency (pseudoparallelism) vs. parallelism Classifications of concurrency ■ List reasons for using concurrency Languages for Concurrent ■ Present briefly different programming ◆ classifications ◆ approaches ■ Concurrent programming ◆ models and languages

Concurrency: what, why, how 3 / 35 Course description Overview What is concurrency Basic idea Dependency Some terms (1) Some terms (2) Some terms (3) Reasons for What is concurrency concurrency Classifications of concurrency Languages for Concurrent programming

Concurrency: what, why, how 4 / 35 Basic idea

Course ■ description intuitively Overview What is ◆ simultaneous execution of concurrency Basic idea ■ instructions (with CPU pipelines) Dependency Some terms (1) ■ actions (functions within a program) Some terms (2) ■ Some terms (3) programs (distributed application) Reasons for concurrency ■ what is simultaneous? Classifications of concurrency ◆ physically at the same time? Languages for ◆ Concurrent “nearly” at the same time? programming ■ 2 threads on a one single-core CPU?

Concurrency: what, why, how 5 / 35 Dependency

Course description If 2 actions Overview ■ What is do not need result of each other (independent) concurrency ■ do not interfere otherwise Basic idea Dependency Some terms (1) ◆ e.g. do not write to the same variable/file Some terms (2) Some terms (3) then the order of their execution does not matter: Reasons for concurrency Classifications of a = c+d concurrency b = c+e Languages for Concurrent programming ■ may be run in parallel (concurrent) ■ excessive optimism (about action dependency/interference) leads to troubles ■ are a and e aliased? Language semantics do matter!

Concurrency: what, why, how 6 / 35 Some terms (1)

Course description Not a rule, just to extend the understanding of concurrency. Overview What is ■ concurrency Parallel - execute simultaneously Basic idea ■ Concurrent - the order of execution does not matter Dependency Some terms (1) Some terms (2) From wikipedia: Some terms (3) Reasons for Parallel computing is a form of computation in concurrency which many calculations are carried out simultaneously. Classifications of concurrency Concurrent computing is a form of computing in Languages for Concurrent which programs are designed as collections of programming interacting computational processes that may be executed in parallel.

Concurrent sometimes referred to as pseudoparallel.

Concurrency: what, why, how 7 / 35 Some terms (2)

Course description Haskell community variants. Overview What is ■ concurrency Parallel - deterministic “data crunching” simultaneous Basic idea execution of the same type tasks Dependency ■ Some terms (1) Concurrent - non-deterministic execution of unrelated Some terms (2) communicating processes Some terms (3) Reasons for concurrency From Chapter 24. Concurrent and multicore programming: Classifications of concurrency A concurrent program needs to perform several Languages for Concurrent possibly unrelated tasks at the same time. programming In contrast, a parallel program solves a single problem.

Concurrency: what, why, how 8 / 35 Some terms (3)

Course description “Programming Clojure”, 5.1 State, Concurrency, Parallelism, Overview and Locking What is concurrency Basic idea A concurrent program models more than one thing Dependency Some terms (1) happening simultaneously. Some terms (2) Some terms (3) A parallel program takes an operation that could be Reasons for sequential and chooses to break it into separate pieces concurrency that can execute concurrently to speed overall Classifications of concurrency execution. Languages for Concurrent programming

Concurrency: what, why, how 9 / 35 Course description Overview What is concurrency Reasons for concurrency List of reasons Some real life “analogies” Faster programs Reasons for concurrency Hiding latency Better structure Classifications of concurrency Languages for Concurrent programming

Concurrency: what, why, how 10 / 35 List of reasons

Course ■ description Faster programs Overview What is ◆ running on several cores/cpus/computers concurrency Reasons for ■ More responsive programs concurrency List of reasons Some real life ◆ GUI interface “analogies” Faster programs ◆ hiding disk/network latency Hiding latency Better structure ■ Programs with natural concurrency Classifications of concurrency ◆ Languages for distributed programs (client-server, etc) Concurrent programming ■ Fault tolerant programs ◆ using redundancy ■ Better structured programs

Concurrency: what, why, how 11 / 35 Some real life “analogies”

Course ■ description Speed of a “process” Overview What is ◆ With 1 axe one friend can chop wood and the other concurrency collect it (conveier) Reasons for concurrency ◆ With 2 axes both friends can chop wood in parallel List of reasons Some real life “analogies” ■ Hiding latency Faster programs Hiding latency ◆ Better structure When we turn on a kettle we do not wait until it boils Classifications of concurrency ■ e.g. we go and take out cups from cupboard Languages for ■ then return to the kettle Concurrent programming ■ Better structured ◆ doing ironing and cooking concurrently is messy

■ assign to different people

Concurrency: what, why, how 12 / 35 Faster programs

Course ■ description Calculate elements of an array in parallel Overview ■ Perform calculations on several processors/nodes What is ■ concurrency Serving youtube videos from multiple servers Reasons for concurrency End of Moore’s law List of reasons Some real life “analogies” The number of transistors that can be placed Faster programs Hiding latency inexpensively on an integrated circuit has increased Better structure exponentially, doubling approximately every two years Classifications of concurrency ■ Languages for Every new laptop comes with (at least) dual core technology Concurrent programming ◆ usually stuck with 50% CPU usage

Concurrency: what, why, how 13 / 35 Hiding latency

Course ■ description disk/network take time Overview What is ◆ if a thread blocks while waiting for HTTP response, no concurrency other action can happen Reasons for concurrency List of reasons ■ no other network request Some real life “analogies” ■ not even UI events Faster programs Hiding latency ■ either Better structure Classifications of concurrency ◆ work asynchronously Languages for Concurrent ■ leads to sliced programs and “pyramids of doom” programming ◆ dedicated thread

■ asking for synchronization troubles.. leads to race conditions

Concurrency: what, why, how 14 / 35 Better structure

Course ■ description Assign different threads to unrelated tasks (if reasonable) Overview What is Data sharing server concurrency Reasons for ■ vertically, one thread per request concurrency List of reasons Some real life ◆ typical web processing (excl. database access) “analogies” Faster programs ■ Hiding latency horizontally (conveier) Better structure Classifications of ◆ dedicated thread(s) for reading requests concurrency ◆ dedicated thread(s) for searching data Languages for Concurrent ◆ new thread for sending data programming Mixing tasks of all threads in one thread ■ asynchronous behavior ■ structural nightmare

Concurrency: what, why, how 15 / 35 Course description Overview What is concurrency Reasons for concurrency Classifications of concurrency Task and data parallelism Classifications of concurrency Coarse and fine grained parallelism High and low level parallelism Explicit and implicit parallelism (1) Explicit and implicit parallelism (2) Formalizations By application areas By computation model Languages for Concurrent programming

Concurrency: what, why, how 16 / 35 Task and data parallelism

Course ■ description Task parallelism: different operations concurrently Overview What is ◆ calculate g and h in f(g(x),h(y)) concurrently concurrency ◆ threads in the same program Reasons for concurrency ◆ several programs running on the same computer Classifications of concurrency ■ Task and data Data parallelism: same operation for different data (SIMD) parallelism Coarse and fine ◆ forall i=1..N do a[i]=a[i]+1 grained loop operations: parallelism ◆ vectorised operations: MMX, SSE, etc High and low level parallelism Explicit and implicit A program may benefit from both! parallelism (1) Explicit and implicit parallelism (2) Formalizations By application areas By computation model Languages for Concurrent programming

Concurrency: what, why, how 17 / 35 Coarse and fine grained parallelism

Course description Ratio of computation and communication Overview ■ What is coarse-grain parallel programs compute most of the time concurrency Reasons for ◆ e.g distribute data, calculate, collect result (Google concurrency MapReduce) Classifications of concurrency Task and data ■ fine-grain parallel programs communicate frequently parallelism Coarse and fine grained ◆ lots of dependencies between distributed data parallelism High and low level parallelism ■ medium-grained parallelism Explicit and implicit parallelism (1) ◆ DOUG: lots of computation interchange with lots of Explicit and implicit communication parallelism (2) Formalizations By application areas By computation model Languages for Concurrent programming

Concurrency: what, why, how 18 / 35 High and low level parallelism

Course description Different granularity (unit of parallelism) Overview ■ What is instruction-level concurrency Reasons for ◆ conveiers and pipelines in CPU; MMX concurrency Classifications of ■ concurrency expression level Task and data parallelism ◆ run expression in separate thread Coarse and fine grained parallelism ■ function level High and low level parallelism ■ process level Explicit and implicit parallelism (1) Source of confusion: this sometimes referred as fine/coarse Explicit and implicit grained parallelism. Question of terminology? However, it is parallelism (2) possible to have two processes that communicate very frequently.. Formalizations By application areas By computation model Languages for Concurrent programming

Concurrency: what, why, how 19 / 35 Explicit and implicit parallelism (1)

Course description Models and languages for Parallel Computation; David B. Overview Skillicorn, Domenico Talia; 1998 What is concurrency ■ Parallelism explicit (hints for possible parallelism) Reasons for concurrency ◆ Loops: forall i in 1..N do a[i]=i Classifications of concurrency ◆ Fortran 90 matrix sum: C=A+B Task and data parallelism Coarse and fine ■ Decomposition explicit (specify parallel pieces) grained parallelism ■ Mapping explicit (map pieces to processors) High and low ■ level parallelism Communication explicit (specify sends/recvs) Explicit and ■ implicit Synchronization explicit (handle details of message-passing) parallelism (1) Explicit and implicit parallelism (2) Formalizations By application areas By computation model Languages for Concurrent programming

Concurrency: what, why, how 20 / 35 Explicit and implicit parallelism (2)

Course description Possibilities Overview What is 1. nothing explicit, parallelism implicit (OBJ, P3L) concurrency 2. parallelism explicit, decomposition implicit Reasons for concurrency ■ Loops - Fortran variants, Id, APL, NESL Classifications of concurrency Task and data 3. decomposition explicit, mapping implicit (BSP, LogP) parallelism Coarse and fine 4. mapping explicit, communication implicit (Linda) grained parallelism 5. communication explicit, synchronization implicit High and low level parallelism ■ Explicit and Actors, Concurrent smalltalk implicit parallelism (1) Explicit and 6. everything explicit implicit parallelism (2) ■ Formalizations PVM, MPI, fork By application areas By computation model Languages for Concurrent programming

Concurrency: what, why, how 21 / 35 Formalizations

Course description How to desribe (concurrent) computations? Overview ■ What is operational semantics – describe operations in Virtual concurrency Machine (VM) Reasons for concurrency ◆ Oz way Classifications of concurrency ◆ “reasoning for a programmer” Task and data parallelism Coarse and fine ■ denotational semantics – describe algebraic rules grained parallelism High and low ◆ concurrent lambda calculus, Pi calculus, CSP, Petri level parallelism Explicit and nets, DDA (Data Dependency Algebra) implicit parallelism (1) ◆ “reasoning for a matematician” Explicit and implicit ■ parallelism (2) axiomatic semantics – describe logical rules Formalizations By application ◆ areas TLA (Temporal Logic of Actions) By computation ◆ model “reasoning for a machine (a prover)” Languages for Concurrent programming

Concurrency: what, why, how 22 / 35 By application areas

Course ■ description Scientific computing Overview What is ◆ High-Performance Computing (HPC) concurrency ◆ High-Throughput Computing (HTC) Reasons for concurrency ■ Classifications of Distributed applications concurrency Task and data ◆ parallelism clients, servers Coarse and fine ◆ grained P2P parallelism ◆ telephone stations (Erlang PL) High and low level parallelism Explicit and ■ Desktop applications implicit parallelism (1) Explicit and ◆ responsive user interfaces implicit parallelism (2) ◆ utilizing multiple cores Formalizations By application areas By computation model Languages for Concurrent programming

Concurrency: what, why, how 23 / 35 By computation model

Course description What style of programming is supported? Overview ■ What is Declarative concurrent model concurrency Reasons for ◆ (pure) functional concurrency ◆ logical Classifications of concurrency Task and data ■ Message-passing model parallelism Coarse and fine grained ◆ synchronous, asynchronous, RPC parallelism High and low ◆ active objects, passive objects level parallelism Explicit and implicit ■ Shared-state (shared memory) model parallelism (1) Explicit and implicit ◆ locks parallelism (2) ◆ Formalizations transactions By application areas By computation model Languages for Concurrent programming

Concurrency: what, why, how 24 / 35 Course description Overview What is concurrency Reasons for concurrency Classifications of concurrency Languages for Languages for Concurrent programming Concurrent programming Why programming language? Oz Erlang Scala Clojure High-Performance Fortran NESL Concurrent and Parallel Haskell Intel TBB ArBB

Concurrency: what, why, how 25 / 35 Why programming language?

Course description Why not just library? Overview ■ What is cleaner syntax concurrency ■ safer semantics Reasons for ■ concurrency forces usage patterns Classifications of ■ control over compilation process concurrency Languages for In 198x there were hundreds PLs for concurrent programming, Concurrent programming now there are thousands. Why programming ■ language? the following slides describe some languages Oz Erlang Scala Clojure High-Performance Fortran NESL Concurrent and Parallel Haskell Intel TBB ArBB

Concurrency: what, why, how 26 / 35 Oz

Course ■ description roots in logic programming Overview What is ◆ dataflow variables (logical variables with suspension) concurrency Reasons for ■ multiparadigm (advertises different styles of programming) concurrency Classifications of ◆ concurrency functional programming Languages for ◆ object oriented programming Concurrent programming ◆ constraint programming (logic) Why programming language? ■ concurrency Oz Erlang ◆ thread Scala explicit task parallelism ( statement) Clojure ◆ explicit and implicit communication (through dataflow High-Performance Fortran variables) NESL ◆ Concurrent and for distributed and desktop programming Parallel Haskell Intel TBB ArBB

Concurrency: what, why, how 27 / 35 Erlang

Course ■ description Ericsson project from ~1990 Overview What is ◆ for telecom applications concurrency Reasons for ■ handle thousands of phone calls concurrency ■ robustness, distribution Classifications of concurrency Languages for ■ Concurrency Concurrent programming ◆ Why processes with message-passing (actors) programming ◆ language? focus on fault tolerance Oz Erlang Scala Clojure High-Performance Fortran NESL Concurrent and Parallel Haskell Intel TBB ArBB

Concurrency: what, why, how 28 / 35 Scala

Course ■ description 2008 year hot topic Overview What is ◆ interoperable with Java (runs on JVM) concurrency ◆ syntax similar to Java Reasons for concurrency ◆ mix of object oriented and functional programming Classifications of ◆ concurrency static typing, type inference, type parameters, ...

Languages for ■ Concurrent in general quite sophisticated programming Why ■ programming Concurrency language? Oz ◆ task parallelism Erlang Scala ◆ processes with message-passing (actors) Clojure High-Performance ■ not main focus, thus not mature Fortran NESL Concurrent and ◆ Akka library for actors http://akka.io/ Parallel Haskell Intel TBB ArBB

Concurrency: what, why, how 29 / 35 Clojure

Course ■ description 2008 year hot topic Overview What is ◆ targets the Java Virtual Machine concurrency ◆ Lisp syntax Reasons for concurrency ◆ functional programming, macro programming Classifications of concurrency ■ Concurrency Languages for Concurrent programming ◆ 4 reference models (atoms,vars,refs,agents) Why programming ◆ task parallelism language? ◆ Oz reactive Agent system Erlang ◆ software transactional memory Scala Clojure High-Performance Fortran NESL Concurrent and Parallel Haskell Intel TBB ArBB

Concurrency: what, why, how 30 / 35 High-Performance Fortran

Course ■ description since 1993, extension of Fortran 90 Overview ■ Concurrency What is concurrency ◆ data parallelism Reasons for concurrency ■ Classifications of A lot of extensions but no success concurrency Languages for Concurrent programming Why programming language? Oz Erlang Scala Clojure High-Performance Fortran NESL Concurrent and Parallel Haskell Intel TBB ArBB

Concurrency: what, why, how 31 / 35 NESL

Course ■ description since 1995 Overview ◆ What is available only on rare platforms concurrency ◆ a way to handle nested data Reasons for concurrency ■ sparse matrice storage Classifications of concurrency ■ in quicksort algorithm Languages for Concurrent ■ programming Concurrency Why programming ◆ language? nested data parallelism Oz Erlang Scala Clojure High-Performance Fortran NESL Concurrent and Parallel Haskell Intel TBB ArBB

Concurrency: what, why, how 32 / 35 Concurrent and Parallel Haskell

Course ■ description Parallel Haskell with par and pseq Overview What is ◆ deterministic speculative execution concurrency Reasons for ■ Concurrent Haskell with forkIO concurrency Classifications of ◆ concurrency locks, monitors, etc Languages for ◆ synchronization variables MVars Concurrent programming ◆ STM (software transactional memory) with atomically Why programming ◆ and more: mHaskell language? Oz ■ Erlang Data Parallel Haskell with parallel arrays Scala Clojure ◆ NDP (nested data parallelism) High-Performance Fortran NESL Concurrent and Parallel Haskell Intel TBB ArBB

Concurrency: what, why, how 33 / 35 Intel TBB

Course ■ description Intel Thread Building Blocks Overview What is ◆ recent C++ library concurrency Reasons for ■ Concurrency concurrency Classifications of ◆ concurrency task parallelism Languages for Concurrent programming Why programming language? Oz Erlang Scala Clojure High-Performance Fortran NESL Concurrent and Parallel Haskell Intel TBB ArBB

Concurrency: what, why, how 34 / 35 ArBB

Course ■ description Intel Array Building Blocks Overview What is ◆ compiler concurrency ◆ beta version Reasons for concurrency ■ Classifications of Concurrency concurrency Languages for ◆ immutable data (declarative model) Concurrent programming ◆ (nested) data parallelism Why programming language? Oz Erlang Scala Clojure High-Performance Fortran NESL Concurrent and Parallel Haskell Intel TBB ArBB

Concurrency: what, why, how 35 / 35