Data Just Right The Addison-Wesley Data and Analytics Series Visit informit.com/awdataseries for a complete list of available publications. he Addison-Wesley Data and Analytics Series provides readers with practical Tknowledge for solving problems and answering questions with data. Titles in this series primarily focus on three areas: 1. Infrastructure: how to store, move, and manage data 2. Algorithms: how to mine intelligence or make predictions based on data 3. Visualizations: how to represent data and insights in a meaningful and compelling way The series aims to tie all three of these areas together to help the reader build end-to-end systems for fighting spam; making recommendations; building personalization; detecting trends, patterns, or problems; and gaining insight from the data exhaust of systems and user interactions. Make sure to connect with us! informit.com/socialconnect Data Just Right Introduction to Large-Scale Data & Analytics Michael Manoochehri Upper Saddle River, NJ • Boston • Indianapolis • San Francisco New York • Toronto • Montreal • London • Munich • Paris • Madrid Capetown • Sydney • Tokyo • Singapore • Mexico City Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and the publisher was aware of a trademark claim, the designations have been printed with initial capital letters or in all capitals. The author and publisher have taken care in the preparation of this book, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of the use of the information or programs contained herein. For information about buying this title in bulk quantities, or for special sales opportunities (which may include electronic versions; custom cover designs; and content particular to your business, training goals, marketing focus, or branding interests), please contact our corporate sales depart- ment at [email protected] or (800) 382-3419. For government sales inquiries, please contact [email protected]. For questions about sales outside the United States, please contact [email protected]. Visit us on the Web: informit.com/aw Library of Congress Cataloging-in-Publication Data Manoochehri, Michael. Data just right : introduction to large-scale data & analytics / Michael Manoochehri. pages cm Includes bibliographical references and index. ISBN 978-0-321-89865-4 (pbk. : alk. paper)—ISBN 0-321-89865-6 (pbk. : alk. paper) 1. Database design. 2. Big data. I. Title. QA76.9.D26M376 2014 005.74’3—dc23 2013041476 Copyright © 2014 Pearson Education, Inc. All rights reserved. Printed in the United States of America. This publication is protected by copy- right, and permission must be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or likewise. To obtain permission to use material from this work, please submit a written request to Pearson Education, Inc., Permissions Department, One Lake Street, Upper Saddle River, New Jersey 07458, or you may fax your request to (201) 236-3290. ISBN-13: 978-0-321-89865-4 ISBN-10: 0-321-89865-6 Text printed in the United States on recycled paper at RR Donnelley in Crawfordsville, Indiana. First printing, December 2013 ❖ This book is dedicated to my parents, Andrew and Cecelia Manoochehri, who put everything they had into making sure that I received an amazing education. ❖ This page intentionally left blank Contents Foreword xv Preface xvii Acknowledgments xxv About the Author xxvii I Directives in the Big Data Era 1 1 Four Rules for Data Success 3 When Data Became a BIG Deal 3 Data and the Single Server 4 The Big Data Trade-Off 5 Build Solutions That Scale (Toward Infinity) 6 Build Systems That Can Share Data (On the Internet) 7 Build Solutions, Not Infrastructure 8 Focus on Unlocking Value from Your Data 8 Anatomy of a Big Data Pipeline 9 The Ultimate Database 10 Summary 10 II Collecting and Sharing a Lot of Data 11 2 Hosting and Sharing Terabytes of Raw Data 13 Suffering from Files 14 The Challenges of Sharing Lots of Files 14 Storage: Infrastructure as a Service 15 The Network Is Slow 16 Choosing the Right Data Format 16 XML: Data, Describe Thyself 18 JSON: The Programmer’s Choice 18 Character Encoding 19 File Transformations 21 Data in Motion: Data Serialization Formats 21 Apache Thrift and Protocol Buffers 22 Summary 23 viii Contents 3 Building a NoSQL-Based Web App to Collect Crowd-Sourced Data 25 Relational Databases: Command and Control 25 The Relational Database ACID Test 28 Relational Databases versus the Internet 28 CAP Theorem and BASE 30 Nonrelational Database Models 31 Key–Value Database 32 Document Store 33 Leaning toward Write Performance: Redis 35 Sharding across Many Redis Instances 38 Automatic Partitioning with Twemproxy 39 Alternatives to Using Redis 40 NewSQL: The Return of Codd 41 Summary 42 4 Strategies for Dealing with Data Silos 43 A Warehouse Full of Jargon 43 The Problem in Practice 45 Planning for Data Compliance and Security 46 Enter the Data Warehouse 46 Data Warehousing’s Magic Words: Extract, Transform, and Load 48 Hadoop: The Elephant in the Warehouse 48 Data Silos Can Be Good 49 Concentrate on the Data Challenge, Not the Technology 50 Empower Employees to Ask Their Own Questions 50 Invest in Technology That Bridges Data Silos 51 Convergence: The End of the Data Silo 51 Will Luhn’s Business Intelligence System Become Reality? 52 Summary 53 Contents ix III Asking Questions about Your Data 55 5 Using Hadoop, Hive, and Shark to Ask Questions about Large Datasets 57 What Is a Data Warehouse? 57 Apache Hive: Interactive Querying for Hadoop 60 Use Cases for Hive 60 Hive in Practice 61 Using Additional Data Sources with Hive 65 Shark: Queries at the Speed of RAM 65 Data Warehousing in the Cloud 66 Summary 67 6 Building a Data Dashboard with Google BigQuery 69 Analytical Databases 69 Dremel: Spreading the Wealth 71 How Dremel and MapReduce Differ 72 BigQuery: Data Analytics as a Service 73 BigQuery’s Query Language 74 Building a Custom Big Data Dashboard 75 Authorizing Access to the BigQuery API 76 Running a Query and Retrieving the Result 78 Caching Query Results 79 Adding Visualization 81 The Future of Analytical Query Engines 82 Summary 83 7 Visualization Strategies for Exploring Large Datasets 85 Cautionary Tales: Translating Data into Narrative 86 Human Scale versus Machine Scale 89 Interactivity 89 Building Applications for Data Interactivity 90 Interactive Visualizations with R and ggplot2 90 matplotlib: 2-D Charts with Python 92 D3.js: Interactive Visualizations for the Web 92 Summary 96 x Contents IV Building Data Pipelines 97 8 Putting It Together: MapReduce Data Pipelines 99 What Is a Data Pipeline? 99 The Right Tool for the Job 100 Data Pipelines with Hadoop Streaming 101 MapReduce and Data Transformation 101 The Simplest Pipeline: stdin to stdout 102 A One-Step MapReduce Transformation 105 Extracting Relevant Information from Raw NVSS Data: Map Phase 106 Counting Births per Month: The Reducer Phase 107 Testing the MapReduce Pipeline Locally 108 Running Our MapReduce Job on a Hadoop Cluster 109 Managing Complexity: Python MapReduce Frameworks for Hadoop 110 Rewriting Our Hadoop Streaming Example Using mrjob 110 Building a Multistep Pipeline 112 Running mrjob Scripts on Elastic MapReduce 113 Alternative Python-Based MapReduce Frameworks 114 Summary 114 9 Building Data Transformation Workflows with Pig and Cascading 117 Large-Scale Data Workflows in Practice 118 It’s Complicated: Multistep MapReduce Transformations 118 Apache Pig: “Ixnay on the Omplexitycay” 119 Running Pig Using the Interactive Grunt Shell 120 Filtering and Optimizing Data Workflows 121 Running a Pig Script in Batch Mode 122 Cascading: Building Robust Data-Workflow Applications 122 Thinking in Terms of Sources and Sinks 123 Contents xi Building a Cascading Application 124 Creating a Cascade: A Simple JOIN Example 125 Deploying a Cascading Application on a Hadoop Cluster 127 When to Choose Pig versus Cascading 128 Summary 128 V Machine Learning for Large Datasets 129 10 Building a Data Classification System with Mahout 131 Can Machines Predict the Future? 132 Challenges of Machine Learning 132 Bayesian Classification 133 Clustering 134 Recommendation Engines 135 Apache Mahout: Scalable Machine Learning 136 Using Mahout to Classify Text 137 MLBase: Distributed Machine Learning Framework 139 Summary 140 VI Statistical Analysis for Massive Datasets 143 11 Using R with Large Datasets 145 Why Statistics Are Sexy 146 Limitations of R for Large Datasets 147 R Data Frames and Matrices 148 Strategies for Dealing with Large Datasets 149 Large Matrix Manipulation: bigmemory and biganalytics 150 ff: Working with Data Frames Larger than Memory 151 biglm: Linear Regression for Large Datasets 152 RHadoop: Accessing Apache Hadoop from R 154 Summary 155 xii Contents 12 Building Analytics Workflows Using Python and Pandas 157 The Snakes Are Loose in the Data Zoo 157 Choosing a Language for Statistical Computation 158 Extending Existing Code 159 Tools and Testing 160 Python Libraries for Data Processing 160 NumPy 160 SciPy: Scientific Computing for Python 162 The Pandas Data Analysis Library 163 Building More Complex Workflows 167 Working with Bad or Missing Records 169 iPython: Completing