DOCSLIB.ORG

Mastering Spark with R the Complete Guide to Large-Scale Analysis and Modeling

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Mastering Spark with R the Complete Guide to Large-Scale Analysis and Modeling Mastering Spark with R The Complete Guide to Large-Scale Analysis and Modeling Javier Luraschi, Kevin Kuo & Edgar Ruiz Foreword by Matei Zaharia Mastering Spark with R The Complete Guide to Large-Scale Analysis and Modeling Javier Luraschi, Kevin Kuo, and Edgar Ruiz Beijing Boston Farnham Sebastopol Tokyo Mastering Spark with R by Javier Luraschi, Kevin Kuo, and Edgar Ruiz Copyright © 2020 Javier Luraschi, Kevin Kuo, and Edgar Ruiz. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or [email protected]. Acquisition Editor: Jonathan Hassell Indexer: Judy McConville Development Editor: Melissa Potter Interior Designer: David Futato Production Editor: Elizabeth Kelly Cover Designer: Karen Montgomery Copyeditor: Octal Publishing, LLC Illustrator: Rebecca Demarest Proofreader: Rachel Monaghan October 2019: First Edition Revision History for the First Release 2019-10-04: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781492046370 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Mastering Spark with R, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors, and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 978-1-492-04637-0 [LSI] To Adrian, Clara, Julian, Max, Mila and Roman. Table of Contents Foreword. xi Preface. xiii 1. Introduction. 1 Overview 2 Hadoop 3 Spark 4 R 9 sparklyr 13 Recap 14 2. Getting Started. 15 Overview 15 Prerequisites 16 Installing sparklyr 17 Installing Spark 17 Connecting 18 Using Spark 19 Web Interface 20 Analysis 22 Modeling 23 Data 25 Extensions 25 Distributed R 26 Streaming 26 Logs 27 Disconnecting 28 v Using RStudio 28 Resources 30 Recap 31 3. Analysis. 33 Overview 33 Import 36 Wrangle 37 Built-in Functions 39 Correlations 40 Visualize 42 Using ggplot2 42 Using dbplot 44 Model 46 Caching 48 Communicate 49 Recap 52 4. Modeling. 53 Overview 54 Exploratory Data Analysis 56 Feature Engineering 63 Supervised Learning 66 Generalized Linear Regression 70 Other Models 72 Unsupervised Learning 72 Data Preparation 73 Topic Modeling 74 Recap 76 5. Pipelines. 77 Overview 77 Creation 79 Use Cases 80 Hyperparameter Tuning 82 Operating Modes 83 Interoperability 84 Deployment 86 Batch Scoring 87 Real-Time Scoring 88 Recap 90 vi | Table of Contents 6. Clusters. 93 Overview 93 On-Premises 95 Managers 95 Distributions 101 Cloud 103 Amazon 104 Databricks 106 Google 107 IBM 108 Microsoft 109 Qubole 110 Kubernetes 111 Tools 112 RStudio 113 Jupyter 114 Livy 114 Recap 115 7. Connections. 117 Overview 118 Edge Nodes 119 Spark Home 120 Local 120 Standalone 121 YARN 122 YARN Client 122 YARN Cluster 123 Livy 124 Mesos 126 Kubernetes 127 Cloud 128 Batches 128 Tools 129 Multiple Connections 129 Troubleshooting 130 Logging 130 Spark Submit 130 Windows 132 Recap 132 Table of Contents | vii 8. Data. 135 Overview 135 Reading Data 137 Paths 137 Schema 138 Memory 140 Columns 141 Writing Data 141 Copying Data 143 File Formats 144 CSV 145 JSON 146 Parquet 147 Others 148 File Systems 149 Storage Systems 150 Hive 150 Cassandra 151 JDBC 152 Recap 152 9. Tuning. 155 Overview 155 Graph 157 Timeline 159 Configuring 160 Connect Settings 161 Submit Settings 162 Runtime Settings 163 sparklyr Settings 164 Partitioning 167 Implicit Partitions 167 Explicit Partitions 168 Caching 169 Checkpointing 170 Memory 171 Shuffling 172 Serialization 172 Configuration Files 172 Recap 173 viii | Table of Contents 10. Extensions. 175 Overview 176 H2O 178 Graphs 182 XGBoost 187 Deep Learning 189 Genomics 192 Spatial 195 Troubleshooting 197 Recap 197 11. Distributed R. 199 Overview 200 Use Cases 202 Custom Parsers 202 Partitioned Modeling 204 Grid Search 205 Web APIs 206 Simulations 207 Partitions 209 Grouping 210 Columns 211 Context 212 Functions 213 Packages 214 Cluster Requirements 215 Installing R 216 Apache Arrow 217 Troubleshooting 219 Worker Logs 220 Resolving Timeouts 221 Inspecting Partitions 222 Debugging Workers 223 Recap 223 12. Streaming. 225 Overview 225 Transformations 228 Analysis 229 Modeling 230 Pipelines 231 Distributed R 232 Table of Contents | ix Kafka 233 Shiny 235 Recap 237 13. Contributing. 239 Overview 240 The Spark API 242 Spark Extensions 243 Using Scala Code 245 Recap 247 A. Supplemental Code References. 249 Index. 267 x | Table of Contents Foreword Apache Spark is a distributed computing platform built on extensibility: Spark’s APIs make it easy to combine input from many data sources and process it using diverse programming languages and algorithms to build a data application. R is one of the most powerful languages for data science and statistics, so it makes a lot of sense to connect R to Spark. Fortunately, R’s rich language features enable simple APIs for calling Spark from R that look similar to running R on local data sources. With a bit of background about both systems, you will be able to invoke massive computations in Spark or run your R code in parallel from the comfort of your favorite R program‐ ming environment. This book explores using Spark from R in detail, focusing on the sparklyr package that enables support for dplyr and other packages known to the R community. It covers all of the main use cases in detail, ranging from querying data using the Spark engine to exploratory data analysis, machine learning, parallel execution of R code, and streaming. It also has a self-contained introduction to running Spark and moni‐ toring job execution. The authors are exactly the right people to write about this topic—Javier, Kevin, and Edgar have been involved in sparklyr development since the project started. I was excited to see how well they’ve assembled this clear and focused guide about using Spark with R. I hope that you enjoy this book and use it to scale up your R workloads and connect them to the capabilities of the broader Spark ecosystem. And because all of the infra‐ structure here is open source, don’t hesitate to give the developers feedback about making these tools better. — Matei Zaharia Assistant Professor at Stanford University, Chief Technologist at Databricks, and original creator of Apache Spark xi Preface In a world where information is growing exponentially, leading tools like Apache Spark provide support to solve many of the relevant problems we face today. From companies looking for ways to improve based on data-driven decisions, to research organizations solving problems in health care, finance, education, and energy, Spark enables analyzing much more information faster and more reliably than ever before. Various books have been written for learning Apache Spark; for instance, Spark: The Definitive Guide is a comprehensive resource, and Learning Spark is an introductory book meant to help users get up and running (both are from O’Reilly). However, as of this writing, there is neither a book to learn Apache Spark using the R computing lan‐ guage nor a book specifically designed for the R user or the aspiring R user. There are some resources online to learn Apache Spark with R, most notably the spark.rstudio.com site and the Spark documentation site at spark.apache.org. Both sites are great online resources; however, the content is not intended to be read from start to finish and assumes you, the reader, have some knowledge of Apache Spark, R, and cluster computing. The goal of this book is to help anyone get started with Apache Spark using R. Addi‐ tionally, because the R programming language was created to simplify data analysis, it is also our belief that this book provides the easiest path for you to learn the tools used to solve data analysis problems with Spark. The first chapters provide an intro‐ duction to help anyone get up to speed with these concepts and present the tools required to work on these problems on your own computer. We then quickly ramp up to relevant data science topics, cluster computing, and advanced topics that should interest even the most experienced users. Therefore, this book is intended to be a useful resource for a wide range of users, from beginners curious to learn Apache Spark, to experienced readers seeking to understand why and how to use Apache Spark from R. This book has the following general outline: xiii Introduction In the first two chapters, Chapter 1, Introduction, and Chapter 2, Getting Started, you learn about Apache Spark, R and the tools to perform data analysis with Spark and R. Analysis In Chapter 3, Analysis, you learn how to analyze, explore, transform, and visual‐ ize data in Apache Spark with R. Modeling In the Chapter 4, Modeling and Chapter 5, Pipelines, you learn how to create stat‐ istical models with the purpose of extracting information, predicticting out‐ comes, and automating this process in production-ready workflows.
Load more

Recommended publications
  • Book of Abstracts
    Book of Abstracts June 27, 2015 1 Conference Sponsors Diamond Sponsor Platinum Sponsors Gold Sponsors Silver Sponsors Open Analytics Bronze Sponsors Media Sponsors 2 Conference program Time Tuesday Wednesday Thursday Friday 08:00 Registration opens Registration opens Registration opens Registration opens 08:30 – 09:00 Opening session (by Rector peR! M. Johansen, Aalborg University) Aalborghallen 09:00 – 10:00 Romain François Di Cook Thomas Lumley Aalborghallen Aalborghallen Aalborghallen 10:00 – 10:30 Coffee break Coffee break Coffee break (15 min) ee break Sponsored by Quantide Sponsored by Alteryx ff Session 1 Session 4 10:30 – 12:00 Sponsor session (10:15) Kaleidoscope 1 Kaleidoscope 4 Aalborghallen Aalborghallen Aalborghallen incl. co Morning Tutorials DataRobot Ecology Medicine Gæstesalen Gæstesalen RStudio Teradata Networks Regression Musiksalen Musiksalen Revolution Analytics Reproducibility Commercial Offerings alteryx Det Lille Teater Det Lille Teater TIBCO H O Interfacing Interactive graphics 2 Radiosalen Radiosalen HP 12:00 – 13:00 Sandwiches Lunch (standing buffet) Lunch (standing buffet) Break: 12:00 – 12:30 Sponsored by Sponsored by TIBCO ff Revolution Analytics Ste en Lauritzen (12:30) Aalborghallen Session 2 Session 5 13:00 – 14:30 13:30: Closing remarks Kaleidoscope 2 Kaleidoscope 5 Aalborghallen Aalborghallen 13:45: Grab ’n go lunch 14:00: Conference ends Case study Teaching 1 Gæstesalen Gæstesalen Clustering Statistical Methodology 1 Musiksalen Musiksalen ee break Data Management Machine Learning 1 ff Det Lille Teater Det Lille
    [Show full text]
  • Cloud Computing with E-Science Applications
    TERZO Information Technology • MOSSUCCA Cloud Computing with e-Science Applications The amount of data in everyday life has been exploding. This data increase has been especially signicant in scientic elds, where substantial amounts of data must be captured, communicated, aggregated, stored, and analyzed. Cloud Computing with e-Science Applications explains how cloud computing can improve data management in data-heavy elds such as bioinformatics, earth science, and computer science. Cloud Computing The book begins with an overview of cloud models supplied by the National Institute of Standards and Technology (NIST), and then: • Discusses the challenges imposed by big data on scientic data infrastructures, including security and trust issues • Covers vulnerabilities such as data theft or loss, privacy concerns, infected applications, threats in virtualization, and cross-virtual machine attack with • Describes the implementation of workows in clouds, proposing an architecture composed of two layers—platform and application • Details infrastructure-as-a-service (IaaS), platform-as-a-service (PaaS), e-Science Applications and software-as-a-service (SaaS) solutions based on public, private, and hybrid cloud computing models • Demonstrates how cloud computing aids in resource control, vertical and horizontal scalability, interoperability, and adaptive scheduling Featuring signicant contributions from research centers, universities, Cloud Computing and industries worldwide, Cloud Computing with e-Science Applications presents innovative cloud
    [Show full text]
  • User! 2020 Tutorials – Afternoon Session
    useR! 2020 Tutorials – Afternoon Session How green was my valley - Spatial analytics with PostgreSQL, PostGIS, R, and PL/R by J. Conway This tutorial combines the use of PostgreSQL with PostGIS and R spatial packages. It covers data ingestion including cleaning and transformation, and basic analysis, of vector and raster based geospatial data from open sources. Learning outcomes: At the end of the tutorial, the participants will: • learn the advantages and disadvantages of using PostgreSQL, PostGIS, R, and PL/R to process geospatial data; • perform a basic installation and setup of the technology stack, including PostgreSQL, PostGIS, R, and PL/R; • gain experience on the data cleaning, subsetting, and ingestion process for that data; • perform several simply types of visualization and analysis on the ingested data. Requirements: The audience should be comfortable with R, SQL, and spatial data at an intermediate level. The attendees need only a modern OS/web browser. The hands-on exercises will be done via Katacoda. Creating beautiful data visualization in R: a ggplot2 crash course by S. Tyner Learning ggplot2 brings joy and “aha moments” to new R users, keeping them more engaged and eager to grow their R skills. Newer R users will be and feel more empowered with data visualization skills. In addition to experiencing joy in creating beautiful graphics, advanced R users will learn to take advantage of ggplot2’s elegant defaults, saving time on manual plotting tasks like drawing legends. Thus, time and energy can be spent on advanced analyses,
    [Show full text]
  • Computational Reproducibility in Archaeological Research: Basic Principles and a Case Study of Their Implementation
    J Archaeol Method Theory (2017) 24:424–450 DOI 10.1007/s10816-015-9272-9 Computational Reproducibility in Archaeological Research: Basic Principles and a Case Study of Their Implementation Ben Marwick1,2 Published online: 7 January 2016 # Springer Science+Business Media New York 2016 Abstract The use of computers and complex software is pervasive in archaeology, yet their role in the analytical pipeline is rarely exposed for other researchers to inspect or reuse. This limits the progress of archaeology because researchers cannot easily reproduce each other’sworktoverifyorextendit.Fourgeneralprinciplesofrepro- ducible research that have emerged in other fields are presented. An archaeological case study is described that shows how each principle can be implemented using freely available software. The costs and benefits of implementing reproducible research are assessed. The primary benefit, of sharing data in particular, is increased impact via an increased number of citations. The primary cost is the additional time required to enhance reproducibility, although the exact amount is difficult to quantify. Keywords Reproducible research . Computer programming . Software engineering . Australian archaeology. Open science Introduction Archaeology, like all scientific fields, advances through rigorous tests of previously published studies. When numerous investigations are performed by different re- searchers and demonstrate similar results, we hold these results to be a reasonable approximation of a true account of past human behavior. This
    [Show full text]
  • R Studio R Code Execution Error
    R Studio R Code Execution Error When I try to open a freshly-installed RStudio, I am getting the following problem (and a blank application window): ERROR r error 4 (R code execution error). When I installed RRO and click Update packages in RStudio, an error message box says R code execution error and the console says. Error in. When starting up RStudio on OS X, you may see an error similar to the following: ERROR: r error 4 (R code execution error) (errormsg=Error in identical(call((1L)). Shadow graphics device error: r error 4 (R code execution error) like the following ones with no results (of course, I restarted Rstudio after any instalation). Inside Rstudio I would determine whether the error is limited to a package or a particular repository. Next test your R installation outside of the Rstudio e.. Introduction to the R language Installation and Configuration R Studio executed results of our code as well as any error messages in the execution of our code. This pane also shows the history of the R code execution which has been. R Studio R Code Execution Error Read/Download 2015 21:20:22 (rsession-cmohan) ERROR r error 4 (R code execution error) Happens in both 32 and 64 bit R, as well as in RStudio and base R. I started. This content is adapted from the RStudio R Markdown - Dynamic Documents of R, Markdown basics and R code chunks tutorials. is used to name the code chunk, useful for following the code execution and debugging. (r or errors are to be displayed with (r, message=TRUE, warning=TRUE, error=TRUE) or FALSE.
    [Show full text]
  • Package 'Drake'
    Package ‘drake’ December 11, 2018 Title A Pipeline Toolkit for Reproducible Computation at Scale Description A general-purpose computational engine for data analysis, drake rebuilds intermediate data objects when their dependencies change, and it skips work when the results are already up to date. Not every execution starts from scratch, there is native support for parallel and distributed computing, and completed projects have tangible evidence that they are reproducible. Extensive documentation, from beginner-friendly tutorials to practical examples and more, is available at the reference website <https://ropensci.github.io/drake/> and the online manual <https://ropenscilabs.github.io/drake-manual/>. Version 6.2.1 License GPL-3 URL https://github.com/ropensci/drake BugReports https://github.com/ropensci/drake/issues Depends R (>= 3.3.0) Imports codetools, digest, formatR, igraph, methods, pkgconfig, rlang (>= 0.2.0), storr (>= 1.1.0), tibble, utils Suggests abind, bindr, callr, clustermq, CodeDepends, crayon, datasets, DBI, downloader, future, future.apply, grDevices, ggplot2, ggraph, knitr, lubridate, networkD3, parallel, prettycode, RSQLite, stats, testthat, tidyselect (>= 0.2.4), txtq, rmarkdown, styler, visNetwork, webshot VignetteBuilder knitr Encoding UTF-8 Language en-US RoxygenNote 6.1.1 NeedsCompilation no 1 2 R topics documented: Author William Michael Landau [aut, cre] (<https://orcid.org/0000-0003-1878-3253>), Alex Axthelm [ctb], Jasper Clarkberg [ctb], Kirill Müller [ctb], Ben Marwick [rev], Peter Slaughter [rev], Ben Bond-Lamberty [ctb] (<https://orcid.org/0000-0001-9525-4633>), Eli Lilly and Company [cph] Maintainer William Michael Landau <[email protected]> Repository CRAN Date/Publication 2018-12-10 23:20:03 UTC R topics documented: drake-package .
    [Show full text]
  • The VGAM Package, Which Is Ca- Ter of the Publication Will Be Largely Unchanged
    News The Newsletter of the R Project Volume 8/2, October 2008 Editorial by John Fox pable of fitting a dizzying array of statistical mod- els. Paul Murrell shows how the compare package I had the opportunity recently to examine the devel- is used to locate similarities and differences among opment of the R Project. A dramatic index of this de- non-identical objects, and explains how the package velopment is the rapid growth in R packages, which may be applied to grading students’ assignments. is displayed in Figure 1; because the vertical axis in Arthur Allignol, Jan Beyersmann, and Martin Schu- this graph is on a log scale, the least-squares linear- macher describe the mvna package for the Nelson- regression line represents exponential growth. Aalen estimator of cumulative transition hazards in This issue of R News is another kind of testament multistate models. Finally, in the latest installment of to the richness and diversity of applications pro- the Programmers’ Niche column, Vince Carey reveals grammed in R: Hadley Wickham, Michael Lawrence, the mind-bending qualities of recursive computation Duncan Temple Lang, and Deborah Swayne describe in R without named self-reference. the rggobi package, which provides an interface to The “news” portion of this issue of R Rnews in- GGobi, software for dynamic and interactive sta- cludes a report on the recent useR! conference in tistical graphics. Carsten Dormann, Bernd Gruber, Dortmund, Germany; announcements of useR! 2009 and Jochen Fründ introduce the bipartite package in Rennes, France, and of DSC 2009 in Copenhagen, for examining interactions among species in ecolog- Denmark, both to be held in July; news from the R ical communities.
    [Show full text]
  • Business Analytics in R Introduction to Statistical Programming
    Business Analytics in R Introduction to Statistical Programming TIMOTHY WONG1 JAMES GAMMERMAN2 July 15, 2019 [email protected] [email protected] ii Contents 1 R Ecosystem 3 1.1 Programming Enviornment . .3 1.2 Packages . .5 2 Programming Concepts 7 2.1 Vector . .8 2.2 Character and Datetime . .9 2.3 Factor . .9 2.4 Logical Operator . 10 2.5 Special Numbers . 10 2.6 List . 11 2.7 Data Frame and Tibble . 11 2.8 Function . 13 2.9 Flow Control . 14 2.9.1 If-Else . 14 2.9.2 While . 15 2.9.3 For . 16 2.10 Apply . 16 iii iv CONTENTS 3 Data Transformation 19 3.1 Filtering . 22 3.2 Sorting . 24 3.3 Subsetting Variables . 24 3.4 Compute Variables . 26 3.5 Summarising . 27 4 Regression Models 31 4.1 Linear Regression . 31 4.2 Poisson Regression . 42 4.3 Logistic Regression . 45 5 Tree-based Methods 47 5.1 Decision Trees . 48 5.2 Random Forest . 50 6 Neural Networks 53 6.1 Multilayer Perceptron . 55 7 Time Series Analysis 61 7.1 Auto-Correlation Function . 61 7.2 Decomposition . 66 7.3 ARIMA Model . 69 8 Survival Analysis 75 8.1 Kaplan-Meier Estimator . 75 8.2 Cox Proportional Hazards Model . 77 CONTENTS v 9 Unsupervised Learning 81 9.1 K-means Clustering . 82 9.2 Hierarchical Clustering . 85 10 Extending R 93 10.1 R Markdown . 93 10.1.1 R Notebook . 97 10.2 Shiny Web Application . 97 10.3 Writing Packages . 102 10.4 Reproducibility .
    [Show full text]
  • Sparkr: Scaling R Programs with Spark
    SparkR: Scaling R Programs with Spark Shivaram Venkataraman1, Zongheng Yang1, Davies Liu2, Eric Liang2, Hossein Falaki2 Xiangrui Meng2, Reynold Xin2, Ali Ghodsi2, Michael Franklin1, Ion Stoica1;2, Matei Zaharia2;3 1AMPLab UC Berkeley, 2 Databricks Inc., 3 MIT CSAIL ABSTRACT support for reading input from a variety of systems including R is a popular statistical programming language with a number of HDFS, HBase, Cassandra and a number of formats like JSON, Par- extensions that support data processing and machine learning tasks. quet, etc. Integrating with the data source API enables R users to However, interactive data analysis in R is usually limited as the R directly process data sets from any of these data sources. runtime is single threaded and can only process data sets that fit in Performance Improvements: As opposed to a new distributed en- a single machine’s memory. We present SparkR, an R package that gine, SparkR can inherit all of the optimizations made to the Spark provides a frontend to Apache Spark and uses Spark’s distributed computation engine in terms of task scheduling, code generation, computation engine to enable large scale data analysis from the R memory management [3], etc. shell. We describe the main design goals of SparkR, discuss how SparkR is built as an R package and requires no changes to the high-level DataFrame API enables scalable computation and R. The central component of SparkR is a distributed data frame present some of the key details of our implementation. that enables structured data processing with a syntax familiar to R users [31](Figure 1).
    [Show full text]
  • Large-Scale Parallel Statistical Forecasting Computations in R
    Large-Scale Parallel Statistical Forecasting Computations in R Murray Stokely∗ Farzan Rohani∗ Eric Tassone∗ Abstract We demonstrate the utility of massively parallel computational infrastructure for statistical computing using the MapReduce paradigm for R. This framework allows users to write com- putations in a high-level language that are then broken up and distributed to worker tasks in Google datacenters. Results are collected in a scalable, distributed data store and returned to the interactive user session. We apply our approach to a forecasting application that fits a variety of models, prohibiting an analytical description of the statistical uncertainty associated with the overall forecast. To overcome this, we generate simulation-based uncer- tainty bands, which necessitates a large number of computationally intensive realizations. Our technique cut total run time by a factor of 300. Distributing the computation across many machines permits analysts to focus on statistical issues while answering questions that would be intractable without significant parallel computational infrastructure. We present real-world performance characteristics from our application to allow practitioners to better understand the nature of massively parallel statistical simulations in R. Key Words: Statistical Computing, R, forecasting, timeseries, parallelism 1. Introduction Large-scale statistical computing has become widespread at Internet companies in recent years, and the rapid growth of available data has increased the importance of scaling the tools for data analysis. Significant progress has been made in design- ing distributed systems to take advantage of massive clusters of shared machines for long-running batch jobs, but the development of higher-level abstractions and tools for interactive statistical analysis using this infrastructure has lagged.
    [Show full text]
  • Compiler and Runtime Techniques for Optimizing Dynamic Scripting Languages
    c 2015 by Haichuan Wang. All rights reserved. COMPILER AND RUNTIME TECHNIQUES FOR OPTIMIZING DYNAMIC SCRIPTING LANGUAGES BY HAICHUAN WANG DISSERTATION Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Computer Science in the Graduate College of the University of Illinois at Urbana-Champaign, 2015 Urbana, Illinois Doctoral Committee: Professor David A. Padua, Chair, Director of Research Professor Vikram Adve Professor Wen-Mei W. Hwu Doctor Peng Wu, Huawei Abstract This thesis studies the compilation and runtime techniques to improve the performance of dynamic scripting languages using R programming language as a test case. The R programming language is a convenient system for statistical computing. In this era of big data, R is becoming increasingly popular as a powerful data analytics tool. But the performance of R limits its usage in a broader context. The thesis introduces a classification of R programming styles into Looping over data(Type I), Vector programming(Type II), and Glue codes(Type III), and identified the most serious overhead of R is mostly manifested in Type I R codes. It pro- poses techniques to improve the performance R. First, it uses interpreter level specialization to do object allocation removal and path length reduction, and evaluates its effectiveness for GNU R VM. The approach uses profiling to translate R byte-code into a specialized byte-code to improve running speed, and uses data representation specialization to reduce the memory allocation and usage. Secondly, it proposes a lightweight approach that reduces the interpretation overhead of R through vectorization of the widely used Apply class of operations in R.
    [Show full text]
  • Speaking Serial R with a Parallel Accent
    Version 0.3-0 Programming with Big Data in R Speaking Serial R with a Parallel Accent Package Examples and Demonstrations pbdR Core Team Speaking Serial R with a Parallel Accent (Ver. 0.3-0) pbdR Package Examples and Demonstrations September 25, 2015 Drew Schmidt Business Analytics and Statistics University of Tennessee Wei-Chen Chen pbdR Core Team George Ostrouchov Computer Science and Mathematics Division Oak Ridge National Laboratory Pragneshkumar Patel National Institute for Computational Sciences University of Tennessee Version 0.3-0 c 2012–2015 pbdR Core Team. All rights reserved. Permission is granted to make and distribute verbatim copies of this vignette and its source provided the copyright notice and this permission notice are preserved on all copies. This manual may be incorrect or out-of-date. The authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein. Cover art from Join, or Die (Franklin, 1754). Illustrations were created using the ggplot2 package (Wickham, 2009), native R functions, and Microsoft Powerpoint. This publication was typeset using LATEX. Contents List of Figures ........................................ v List of Tables ......................................... vii Acknowledgements ...................................... viii Note About the Cover .................................... ix Disclaimer ........................................... 1 I Preliminaries 2 1 Introduction 3 1.1 What is pbdR? ..................................... 3
    [Show full text]