COSC 416 - Dr. Ramon Lawrence
COSC 416 MapReduce and Hadoop NoSQL Databases MapReduce was invented by Google and has an open source implementation called Apache Hadoop (hadoop.apache.org/). Implemented in Java, Apache Top Level Project, most Hadoop and HDFS contributors from Yahoo
Hadoop is a software library designed for distributed processing of large data sets using computer clusters. Dr. Ramon Lawrence University of British Columbia Okanagan Key components: [email protected] HDFS – Hadoop Distributed File System Hadoop MapReduce – Parallel processing of data Many other related projects and systems (we will talk about Hive and Pig later). Page 2
COSC 416 - Dr. Ramon Lawrence COSC 416 - Dr. Ramon Lawrence
Hadoop/HDFS Architecture Hadoop/HDFS Architecture (2) Distributed architecture assumes inexpensive, unreliable Data: commodity hardware. Replication allows reliability. Data is organized into files and directories. Two types of nodes: Files are divided into fixed sized blocks and replicated across NameNode maintains file metadata cluster nodes. Replication factor on a per file basis (default 3). DataNodes manage storage NameNode stores the block replication information. Writes are pipelined to replicas. Reads from nearest replica. File access API: Mostly sequential access Master-Slave architecture: Single writers and no locking NameNode is master and manages file information, blocks, replication/mapping locations. DataNodes store blocks on underlying OS. Clients can access "Computation at the data": blocks directly on DataNodes without going through nameNode. Servers are for both storage and computation Page 3 Page 4
COSC 416 - Dr. Ramon Lawrence COSC 416 - Dr. Ramon Lawrence
HDFS Architecture MapReduce MapReduce/Hadoop parallel processing involves mapping and grouping file records by keys.
Data is stored in files. Users provide functions: reader(file) – converts file data into records map(records) – converts records into key-value pairs combine(key, list of values) – optional aggregation of pairs after map stage reduce(key, list of values) – summary on key values to produce output records write(file) – writes records to output file MapReduce (Hadoop) provides infrastructure for tying everything together and distributing work across machines. Source: hadoop.apache.org Page 5 Page 6 COSC 416 - Dr. Ramon Lawrence COSC 416 - Dr. Ramon Lawrence MapReduce Example MapReduce Example (2) Web Data Analysis Web Data Analysis
Chrome Data file records: URL, timestamp, browser yahoo.com, Chrome Chrome, {1,1,1} google.com, Firefox Map Reduce Chrome, {5} google.com, Chrome IE, {1} msdn.com, IE Goal: Determine the most popular browser used. yahoo.ca, Chrome Firefox, {1}
Chrome Chrome, {1,1} xyz.com, Chrome Reader Map Reduce Writer IE linkedin.com, Chrome IE, {1,1} google.ca, IE Map Reduce IE, {5} msdn.ca, Firefox msdn.com, IE Records IE Records Log Out Firefox, {1} Reader Map Reduce Writer File File
Firefox Firefox costco.ca, Firefox IE, {1,1} walmart.com, Firefox Reader Map Reduce Writer amazon.com, Firefox Map Reduce Firefox, {5} File is on Output msdn.ca, IE distributed file Converts file Groups by key Aggregates over Writes records Firefox, {1,1,1} in a file ubc.ca, IE system: data into (browser). all records with into file. GFS, HDFS. records. Distributes keys. given key. Page 7 Page 8
COSC 416 - Dr. Ramon Lawrence COSC 416 - Dr. Ramon Lawrence
WordCount Hadoop Example WordCount Hadoop Example (2)
import org.apache.hadoop.fs.Path; public static class Reduce extends MapReduceBase implements import org.apache.hadoop.conf.*; Reducer
COSC 416 - Dr. Ramon Lawrence COSC 416 - Dr. Ramon Lawrence
WordCount Hadoop Example (3) Job Configuration Parameters
public static void main(String[] args) throws Exception Job configuration parameters control the execution of Hadoop. { JobConf conf = new JobConf(WordCount.class); conf.setJobName("wordcount"); Main program Examples: conf.setOutputKeyClass(Text.class); core-site.xml conf.setOutputValueClass(IntWritable.class); fs.default.name – file system name node name conf.setMapperClass(Map.class); conf.setCombinerClass(Reduce.class); hdfs-site.xml conf.setReducerClass(Reduce.class); dfs.replication - # of block replicas dfs.data.dir – directory to store data blocks conf.setInputFormat(TextInputFormat.class); conf.setOutputFormat(TextOutputFormat.class); mapred-site.xml – URL of job tracker FileInputFormat.setInputPaths(conf, new Path(args[0])); mapred.job.tracker FileOutputFormat.setOutputPath(conf, new Path(args[1]));
JobClient.runJob(conf); } } Page 11 Page 12 COSC 416 - Dr. Ramon Lawrence COSC 416 - Dr. Ramon Lawrence
Hadoop Versions Conclusion As a relatively new open source project, Hadoop is rapidly Hadoop is an open source implementation of a highly scalable changing. There are many different versions, and distributed file system and MapReduce job processing incompatibilities and differences between them (including architecture. configuration files). It is designed for extremely large data processing on We are running 1.0.4 (latest stable release) that is consistent commodity hardware. with 0.23.X releases. When using web sources as references, watch for the version being used. Using MapReduce requires writing code that defines a mapping and reducer and executing it on the cluster. Efficient algorithms should be highly parallelizable and minimize the amount of data transfer.
Page 13 Page 14
COSC 416 - Dr. Ramon Lawrence
Objectives Understand the Hadoop/HDFS architecture and file/block allocation strategy.
Explain how a MapReduce program is designed.
Be able to understand a simple MapReduce program and write small amounts of code.
Page 15