Efficient R Programming A Practical Guide to Smarter Programming Colin Gillespie and Robin Lovelace Efficient R Programming by Colin Gillespie and Robin Lovelace Copyright © 2017 Colin Gillespie, Robin Lovelace. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com/safari). For more information, contact our corporate/institutional sales department: 800-998-9938 or [email protected]. Editor: Nicole Tache Production Editor: Nicholas Adams Copyeditor: Gillian McGarvey Proofreader: Christina Edwards Indexer: WordCo Indexing Services Interior Designer: David Futato Cover Designer: Randy Comer Illustrator: Rebecca Demarest December 2016: First Edition Revision History for the First Edition 2016-11-29: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781491950784 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Efficient R Programming, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 978-1-491-95078-4 [LSI] Preface Efficient R Programming is about increasing the amount of work you can do with R in a given amount of time. It’s about both computational and programmer efficiency. There are many excellent R resources about topics such as visualization (e.g., Chang 2012), data science (e.g., Grolemund and Wickham 2016), and package development (e.g., Wickham 2015). There are even more resources on how to use R in particular domains, including Bayesian statistics, machine learning, and geographic information systems. However, there are very few unified resources on how to simply make R work effectively. Hints, tips, and decades of community knowledge on the subject are scattered across hundreds of internet pages, email threads, and discussion forums, making it challenging for R users to understand how to write efficient code. In our teaching we have found that this issue applies to beginners and experienced users alike. Whether it’s a question of understanding how to use R’s vector objects to avoid for loops, knowing how to set up your .Rprofile and .Renviron files, or the ability to harness R’s excellent C++ interface to do the heavy lifting, the concept of efficiency is key. The book aims to distill tips, warnings, and tricks of the trade into a single, cohesive whole that provides a useful resource to R programmers of all stripes for years to come. The content of the book reflects the questions that our students from a range of disciplines, skill levels, and industries have asked over the years to make their R work faster. How to set up my system optimally for R programming work? How can one apply general principles from computer science (such as do not repeat yourself, aka DRY) to the specifics of an R script? How can R code be incorporated into an efficient workflow, including project inception, collaboration, and write-up? And how can one quickly learn how to use new packages and functions? The book answers these questions and more in 10 self-contained chapters. Each chapter starts with the basics and gets progressively more advanced, so there is something for everyone in each one. While more advanced topics such as parallel programming and C++ may not be immediately relevant to R beginners, the book helps to navigate R’s infamously steep learning curve with a commitment to starting slow and building on strong foundations. Thus even experienced R users are likely to find previously hidden gems of advice. While teaching this material, we commonly hear “Why didn’t anyone tell me that before?” Efficient programming should not be seen as an optional extra, and the importance of efficiency grows with the size of projects and datasets. In fact, this book was devised while teaching a course called R for Big Data, when it quickly became apparent that if you want to work with large datasets, your code must work efficiently. Even with small datasets, efficient code that is both fast to write and fast to run is a vital component of successful R projects. We found that the concept of efficient programming is important in all branches of the R community. Whether you are a sporadic user of R (e.g., for its unbeatable range of statistical packages), looking to develop a package, or working on a large collaborative project in which efficiency is mission-critical, code efficiency will have a major impact on your productivity. Ultimately, efficiency is about getting more output for less work input. To take the analogy of a car, would you rather drive 1,000 km on a single tank (or a single charge of batteries) or refuel a heavy, clunky, ugly car every 50 km? Or would you prefer to choose an altogether more efficient vehicle and cycle? In the same way, efficient R code is better than inefficient R code in almost every way: it is easier to read, write, run, share, and maintain. This book cannot provide all the answers about how to produce such code, but it certainly can provide ideas, example code, and tips to make a start in the right direction of travel. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Bold Indicates the names of R packages. Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values determined by context. T IP This element signifies a tip or suggestion. NOT E This element signifies a general note. WARNING This element indicates a warning or caution. Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/csgillespie/efficient. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Efficient R Programming by Colin Gillespie and Robin Lovelace (O’Reilly). Copyright 2017 Colin Gillespie, Robin Lovelace, 978-1-491-95078-4.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at [email protected]. O’Reilly Safari Safari (formerly Safari Books Online) is a membership-based training and reference platform for enterprise, government, educators, and individuals. Members have access to thousands of books, training videos, Learning Paths, interactive tutorials, and curated playlists from over 250 publishers, including O’Reilly Media, Harvard Business Review, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press, Adobe, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, and Course Technology, among others. For more information, please visit http://oreilly.com/safari. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://bit.ly/efficient-r-programming. To comment or ask technical questions about this book, send email to [email protected]. For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://www.youtube.com/oreillymedia Acknowledgments This book was written in the open, and many people contributed pull requests to fix minor problems. Special thanks goes to O’Reilly who allowed this process and everyone who contributed via GitHub: @Delvis, @richelbilderbeek, @adamryczkowski, @CSJCampbell, @tktan, @nachti, Conor Lawless, @timcdlucas, Dirk Eddelbuettel, @wolfganglederer, @HenrikBengtsson, @giocomai, and @daattali. Many thanks also to the detailed feedback from the technical reviewers, Richard Cotton and Garrett Grolemund.