Full text available at: http://dx.doi.org/10.1561/2500000028 Programming with “Big Code” Martin Vechev ETH Zurich Switzerland [email protected] Eran Yahav Technion, Haifa Israel [email protected] Boston — Delft Full text available at: http://dx.doi.org/10.1561/2500000028 Foundations and Trends R in Programming Languages Published, sold and distributed by: now Publishers Inc. PO Box 1024 Hanover, MA 02339 United States Tel. +1-781-985-4510 www.nowpublishers.com [email protected] Outside North America: now Publishers Inc. PO Box 179 2600 AD Delft The Netherlands Tel. +31-6-51115274 The preferred citation for this publication is M. Vechev and E. Yahav. Programming with “Big Code”. Foundations and Trends R in Programming Languages, vol. 3, no. 4, pp. 231–284, 2016. R This Foundations and Trends issue was typeset in LATEX using a class file designed by Neal Parikh. Printed on acid-free paper. ISBN: 978-1-68083-231-0 c 2016 M. Vechev and E. Yahav All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, mechanical, photocopying, recording or otherwise, without prior written permission of the publishers. Photocopying. In the USA: This journal is registered at the Copyright Clearance Cen- ter, Inc., 222 Rosewood Drive, Danvers, MA 01923. Authorization to photocopy items for internal or personal use, or the internal or personal use of specific clients, is granted by now Publishers Inc for users registered with the Copyright Clearance Center (CCC). The ‘services’ for users can be found on the internet at: www.copyright.com For those organizations that have been granted a photocopy license, a separate system of payment has been arranged. Authorization does not extend to other kinds of copy- ing, such as that for general distribution, for advertising or promotional purposes, for creating new collective works, or for resale. In the rest of the world: Permission to pho- tocopy must be obtained from the copyright owner. Please apply to now Publishers Inc., PO Box 1024, Hanover, MA 02339, USA; Tel. +1 781 871 0245; www.nowpublishers.com; [email protected] now Publishers Inc. has an exclusive license to publish this material worldwide. Permission to use this content must be obtained from the copyright license holder. Please apply to now Publishers, PO Box 179, 2600 AD Delft, The Netherlands, www.nowpublishers.com; e-mail: [email protected] Full text available at: http://dx.doi.org/10.1561/2500000028 Foundations and Trends R in Programming Languages Volume 3, Issue 4, 2016 Editorial Board Editor-in-Chief Mooly Sagiv Tel Aviv University Israel Editors Martín Abadi Robert Harper Ganesan Ramalingam Google & CMU Microsoft Research UC Santa Cruz Tim Harris Mooly Sagiv Anindya Banerjee Oracle Tel Aviv University IMDEA Fritz Henglein Davide Sangiorgi Patrick Cousot University of Copenhagen University of Bologna ENS Paris & NYU Rupak Majumdar David Schmidt Oege De Moor MPI-SWS & UCLA Kansas State University University of Oxford Kenneth McMillan Peter Sewell Matthias Felleisen Microsoft Research University of Cambridge Northeastern University J. Eliot B. Moss Scott Stoller John Field UMass, Amherst Stony Brook University Google Andrew C. Myers Peter Stuckey Cormac Flanagan Cornell University University of Melbourne UC Santa Cruz Hanne Riis Nielson Jan Vitek Philippa Gardner TU Denmark Purdue University Imperial College Peter O’Hearn Philip Wadler Andrew Gordon UCL University of Edinburgh Microsoft Research & Benjamin C. Pierce David Walker University of Edinburgh UPenn Princeton University Dan Grossman Andrew Pitts Stephanie Weirich University of Washington University of Cambridge UPenn Full text available at: http://dx.doi.org/10.1561/2500000028 Editorial Scope Topics Foundations and Trends R in Programming Languages publishes survey and tutorial articles in the following topics: • Abstract interpretation • Programming languages for concurrency • Compilation and interpretation techniques • Programming languages for • Domain specific languages parallelism • Formal semantics, including • Program synthesis lambda calculi, process calculi, and process algebra • Program transformations and optimizations • Language paradigms • Mechanical proof checking • Program verification • Memory management • Runtime techniques for • Partial evaluation programming languages • Program logic • Software model checking • Programming language • Static and dynamic program implementation analysis • Programming language security • Type theory and type systems Information for Librarians Foundations and Trends R in Programming Languages, 2016, Volume 3, 4 issues. ISSN paper version 2325-1107. ISSN online version 2325-1131. Also available as a combined paper and online subscription. Full text available at: http://dx.doi.org/10.1561/2500000028 Foundations and Trends R in Programming Languages Vol. 3, No. 4 (2016) 231–284 c 2016 M. Vechev and E. Yahav DOI: 10.1561/2500000028 Programming with “Big Code” Martin Vechev Eran Yahav ETH Zurich Technion, Haifa Switzerland Israel [email protected] [email protected] Full text available at: http://dx.doi.org/10.1561/2500000028 Contents 1 Introduction2 2 Statistical Programming Systems6 2.1 Dimensions.......................... 6 3 Example System: Statistical Code Completion 15 3.1 Overview of SLANG .................... 17 3.2 Model ............................ 20 3.3 Statistical Language Models ................ 24 3.4 Synthesis ........................... 29 3.5 Implementation ....................... 34 3.6 Evaluation .......................... 36 3.7 Related work ......................... 43 4 Conclusion 47 References 48 ii Full text available at: http://dx.doi.org/10.1561/2500000028 Abstract The vast amount of code available on the web is increasing on a daily basis. Open-source hosting sites such as GitHub contain billions of lines of code. Community question-answering sites provide millions of code snippets with corresponding text and metadata. The amount of code available in executable binaries is even greater. Collectively, these in- creasing amounts of code have been referred to as “Big Code”. In this monograph, we cover some of the recent research trends on leveraging “Big Code” for performing various programming tasks that are difficult to accomplish with traditional techniques. M. Vechev and E. Yahav. Programming with “Big Code”. Foundations and Trends R in Programming Languages, vol. 3, no. 4, pp. 231–284, 2016. DOI: 10.1561/2500000028. Full text available at: http://dx.doi.org/10.1561/2500000028 1 Introduction The vast amount of code available on the web is increasing on a daily basis. Open-source hosting sites such as GitHub contain billions of lines of code. Community question-answering sites provide millions of code snippets with corresponding text and metadata. The amount of code available in executable binaries is even greater. Collectively, these in- creasing amounts of code have been referred to as “Big Code”. In this monograph, we cover some of the recent research trends on leveraging “Big Code” for performing various programming tasks that are difficult to accomplish with traditional techniques. Size of “Big Code”. As of January 2016, the GitHub open-source platform hosts over 30 million public repositories. The popular question-answering site Stack Overflow contains over 19 million answers, many of them containing code. Both of these sites, as well as other similar sites, are growing rapidly. For example, Fig. 1.1 shows the growth rate in the number of public repositories on GitHub. Looking beyond source code, the amount of binary executables available for download is also increasing rapidly (both in desktop applications and in mobile apps). 2 Full text available at: http://dx.doi.org/10.1561/2500000028 3 35 30 25 20 15 10 5 0 2008 2009 2010 2011 2012 2013 2014 2015 2016 Figure 1.1: Number of public repositories on GitHub (millions). The availability of these massive amounts of code and meta-data have the potential to revolutionize the way software is being developed, taught, debugged and analyzed. Ideas such as example-centric program- ming Brandt et al. [2010] or opportunistic programming Brandt et al. [2009], where the machine assists the programmer in leveraging exist- ing code repositories are not new. They are based on the insight that finding, checking, and adapting an existing solution is easier than creat- ing a solution from scratch. Indeed, various recommendation systems, and code completion tools Thummalapenta and Xie, Zhong et al., Al- nusair et al. [2010], Shoham et al. [2007a], Holmes et al., Mandelin et al. [2005], Gvero et al. [2011], Perelman et al. can improve program- mer productivity and increase code reuse. While these techniques have made significant progress, they have limited or no support for reusing existing codebases. Big Code vs. Big Data Learning from “Big Code” inherits all of the challenges associated with learning from “Big Data” (e.g., images, videos) (see Jagadish et al. [2014]). However, it introduces an additional set of unique problems specific to the domain of programs. The reason is that unlike a tradi- tional data sample (e.g., an image), a data sample in “Big Code” is a program that can represent an infinite number of behaviors. Further, it is well known that many seemingly basic questions about programs Full text available at: http://dx.doi.org/10.1561/2500000028 4 Introduction are in fact undecidable and thus the exact program semantics (that one has to