A Bit More Parallelism COS 326 David Walker Princeton University slides copyright 2013-2015 David Walker and Andrew W. Appel permission granted to reuse these slides for non-commercial educaonal purposes Last Time: Parallel CollecDons The parallel sequence abstracDon is powerful: • tabulate • nth • length • map • split • treeview • scan – used to implement prefix-sum – clever 2-phase implementaon – used to implement filters • sorng PARALLEL COLLECTIONS IN THE "REAL WORLD" Big Data If Google wants to index all the web pages (or images or gmails or google docs or ...) in the world, they have a lot of work to do • Same with Facebook for all the facebook pages/entries • Same with TwiXer • Same with Amazon • Same with ... Many of these tasks come down to map, filter, fold, reduce, scan Google Map-Reduce Google MapReduce (2004): a fault tolerant, MapReduce: Simplified Data Processing on Large Clusters Jeffrey Dean and Sanjay Ghemawat [email protected], [email protected] massively parallel funcDonal programming Google, Inc. Abstract given day, etc. Most such computations are conceptu- ally straightforward. However, the input data is usually paradigm MapReduce is a programming model and an associ- large and the computations have to be distributed across ated implementation for processing and generating large hundreds or thousands of machines in order to finish in data sets. Users specify a map function that processes a a reasonable amount of time. The issues of how to par- key/value pair to generate a set of intermediate key/value allelize the computation, distribute the data, and handle pairs, and a reduce function that merges all intermediate failures conspire to obscure the original simple compu- – based on our friends "map" and "reduce" values associated with the same intermediate key. Many tation with large amounts of complex code to deal with real world tasks are expressible in this model, as shown these issues. in the paper. As a reaction to this complexity, we designed a new Programs written in this functional style are automati- abstraction that allows us to express the simple computa- cally parallelized and executed on a large cluster of com- tions we were trying to perform but hides the messy de- – Hadoop is the open-source variant modity machines. The run-time system takes care of the tails of parallelization, fault-tolerance, data distribution details of partitioning the input data, scheduling the pro- and load balancing in a library. Our abstraction is in- gram’s execution across a set of machines, handling ma- spired by the map and reduce primitives present in Lisp chine failures, and managing the required inter-machine and many other functional languages. We realized that communication. This allows programmers without any most of our computations involved applying a map op- experience with parallel and distributed systems to eas- eration to each logical “record” in our input in order to – ily utilize the resources of a large distributed system. Database people complain that they compute a set of intermediate key/value pairs, and then Our implementation of MapReduce runs on a large applying a reduce operation to all the values that shared cluster of commodity machines and is highly scalable: the same key, in order to combine the derived data ap- a typical MapReduce computation processes many ter- propriately. Our use of a functional model with user- have been doing it for a while abytes of data on thousands of machines. Programmers specified map and reduce operations allows us to paral- find the system easy to use: hundreds of MapReduce pro- lelize large computations easily and to use re-execution grams have been implemented and upwards of one thou- as the primary mechanism for fault tolerance. sand MapReduce jobs are executed on Google’s clusters The major contributions of this work are a simple and every day. powerful interface that enables automatic parallelization • ... but it was hard to define and distribution of large-scale computations, combined with an implementation of this interface that achieves 1 Introduction high performance on large clusters of commodity PCs. Section 2 describes the basic programming model and Over the past five years, the authors and many others at gives several examples. Section 3 describes an imple- Google have implemented hundreds of special-purpose mentation of the MapReduce interface tailored towards Fun stats circa 2012: computations that process large amounts of raw data, our cluster-based computing environment. Section 4 de- such as crawled documents, web request logs, etc., to scribes several refinements of the programming model compute various kinds of derived data, such as inverted that we have found useful. Section 5 has performance indices, various representations of the graph structure measurements of our implementation for a variety of of web documents, summaries of the number of pages tasks. Section 6 explores the use of MapReduce within – Big clusters are ~4000 nodes crawled per host, the set of most frequent queries in a Google including our experiences in using it as the basis To appear in OSDI 2004 1 – Facebook had 100 PB in Hadoop – TritonSort (UCSD) sorts 900GB/minute on a 52-node, 800-disk hadoop cluster Data Model & Operaons • Map-reduce operates over collecDons of key-value pairs – millions of files (eg: web pages) drawn from the file system and parsed in parallel by many machines • The map-reduce engine is parameterized by 3 funcDons, which roughly speaking do this: map : key1 * value1 -> (key2 * value2) list combine : key2 * (value2 list) -> value2 option reduce : key2 * (value2 list) -> key3 * (value3 list) opDonal – ocen used to compress data before transfer from a mapper machine to a reducer machine Architecture Distributed ImplementationCombine Local Storage Local Storage Input Data Output Data Local Storage Map Shuffle/Sort Reduce Input Data Iterative Jobs are common… are Jobs Iterative Iterave Jobs are Common Working Set Output Data Input Data Output Data The Control Plane The control plane Input Input Input Data Data Data Jobs, Tasks and AXempts • A single job is split in to many tasks • each task may include many calls to map and reduce • Workers are long-running processes that are assigned many tasks • MulDple workers may a,empt the same task – each invocaon of the same task is called an aempt – the first worker to finish "wins" • Why have mulDple machines aempt the same task? – machines will fail • approximately speaking: 5% of high-end disks fail/year • if you have 1000 machines: 1 failure per week • repeated failures become the common case – machines can parDally fail or be slow for some reason • reducers can't start unDl all mappers complete Flow of Informaon The flow of information Heartbeats Job config. OK Tasks to start Completed A modernA Modern Socware Stack software stack Workload Manager High-level scripting language Distributed Execution Engine Key-value Distributed Filesystem store Cluster Cluster Cluster Cluster Node Node Node Node 8 Sort-of FuncDonal Programming in Java Hadoop interfaces: interface Mapper<K1,V1,K2,V2> { public void map (K1 key, V1 value, OutputCollector<K2,V2> output) ... } interface Reducer<K2,V2,K3,V3> { public void reduce (K2 key, Iterator<V2> values, OutputCollector<K3,V3> output) ... } Word Count in Java class WordCountMap implements Map { public void map(DocID key List<String> values, OutputCollector<String,Integer> output) { for (String s : values) output.collect(s,1); } } class WordCountReduce { public void reduce(String key, Iterator<Integer> values, OutputCollector<String,Integer> output) { int count = 0; for (int v : values) count += 1; output.collect(key, count) } PLeASe RELAX AND FOR THE SAke OF HYGIENe, WIPe THE JAVA CODe OFF YOUR BRAIN ASSIGNMENT #7: IMPLEMENTING AND USING PARALLEL COLLECTIONS US Census Queries end goal: develop a system for efficiently compuDng US populaon queries by geographic region Assignment 7 inverted index Libraries build layered concurrency Applications geographic abstractions queries Library parallel Library sequences message Library futures passing Library Library Library Concurrency primitives Hardware unix processes map-reduce API for Assignment 7 tabulate (f: int->’a) (n: int) : ‘a seq Create seq of length n, element i holds f(i) Parallel seq_of_array: ‘a array -> ‘a seq Create a sequence from an array Constant Dme array_of_seq: ‘a seq -> ‘a array Create an array from a sequence Constant Dme iter (f: ‘a -> unit): ‘a seq -> unit Applying f on each element in order. Useful for debugging. SequenDal length: ‘a seq -> int Return the length of the sequence Constant Dme empty: unit -> ‘a seq Return the empty sequence Constant Dme cons: ‘a -> ‘a seq -> ‘a seq (nondestrucDvely) cons a new element on the beginning SequenDal singleton: ‘a -> ‘a seq Return the sequence with a single element Constant Dme append: ‘a seq -> ‘a seq -> ‘a seq (nondestrucDvely) concatenate two sequences SequenDal nth: ‘a seq -> int -> ‘a Get the nth value in the sequence. Indexing is zero-based. Constant Dme map (f: ‘a -> ‘b) -> ‘a seq -> ‘a seq Map the funcDon f over a sequence Parallel reduce (f: ‘a -> ‘a -> ‘a) (base: ‘a): Fold a funcDon f over the sequence. Parallel ‘a seq -> ‘a f must be associave, and base must be the unit for f. mapreduce: (‘a->’b)->(‘b->’b->’b)-> Combine the map and reduce funcDons. Parallel ‘b -> ‘a seq -> ‘b flaen: ‘a seq seq -> ‘a seq flaen [[a0;a1]; [a2;a3]] = [a0;a1;a2;a3] SequenDal repeat (x: ‘a) (n: int) : ‘a seq repeat x 4 = [x;x;x;x] SequenDal zip: (‘a seq * ‘b seq) -> (‘a * ‘b) seq zip