University of California Santa Barbara Automated Black Box Generation of Structured Inputs for Use in Software Testing A dissertation submitted in partial satisfaction of the requirements for the degree Doctor of Philosophy in Computer Science by Kyle Thomas Dewey Committee in charge: Professor Ben Hardekopf, Chair Professor Chandra Krintz Professor Tevfik Bultan September 2017 The Dissertation of Kyle Thomas Dewey is approved. Professor Chandra Krintz Professor Tevfik Bultan Professor Ben Hardekopf, Committee Chair June 2017 Automated Black Box Generation of Structured Inputs for Use in Software Testing Copyright © 2017 by Kyle Thomas Dewey iii Dedicated to Melanie, whose love and support has exceeded far beyond anything I could imagine. iv Acknowledgements In many ways, getting to this point was a team effort, and it would not have been possible if not for the vast amount of help I received along the way. Here I point out specific people who have been instrumental in this process, for whom I am immensely thankful. • Ben Hardekopf: For being an all-around excellent adviser and teaching me how to do research in the first place. Your patience speaks tomes. • Tevfik Bultan: For first exposing me to research in software engineering, and plenty of related critical guidance. • Chandra Krintz: For plenty of crucial guidance early on, and convincing me being here was a good idea. • Phill Conrad: For providing tons of mentoring both for teaching and for CS education research. • Ben Wiedermann: For teaching-related mentoring, along with giving CLP-based testing its first practical application). • Vineeth Kashyap: For being my first line of defense against having no idea what was going on in my first two years. • Madhukar Kedlaya: For being my first line of defense against having no idea what was going on in my third year. • Lawton Nichols: For first convincing me that I might actually have an idea of what’s going on. • Niko Matsakis, along with the entire Rust development team: For answering barrage after barrage of questions about Rust. We could not have fuzzed Rust without this help. v • Robert Rothman: For being brutally honest with me during my undergrad, and letting me know my place was in Computer Science. • Michael Christensen, Mehmet Emre, Miroslav (Mika) Gavrilov: For being wonderful fellow Ph.D. compatriots. You have made this a fun-filled experience. Case in point: Enter Sandman, Sad but True, Holier Than Thou, The Unforgiven, Wherever I May Roam, Don’t Tread on Me, Through the Never, Nothing Else Matters, Of Wolf and Man, The God That Failed, My Friend of Misery, The Struggle Within • Jared Roesch, Elena Morozova, Dylan Lynch, Ethan Kuefner, Berkeley Churchill, Davina Zamanzadeh, Dianne Wagner, Ben Campbell: The undergraduate students whom I worked closely with for multiple quarters, if not years. Beyond your research contributions, you all collectively helped me learn how to advise students. • The PL Lab: All members not already mentioned, past and present. • My Parents: For remaining sane and supportive when their only child suddenly decided to move to California. I apologize if I missed any names; it is not intentional. My memory resembles that of a goldfish. vi Curriculum Vitæ Kyle Thomas Dewey Education 2011 - 2017 Ph.D. in Computer Science, University of California, Santa Barbara. 2007 - 2011 M.S. in Bioinformatics, Rochester Institute of Technology, Rochester, NY. 2007 - 2011 B.S. in Bioinformatics, Rochester Institute of Technology, Rochester, NY. Courses Taught at the University of California, Santa Barbara Spring 2017 CS 162: Programming Languages Winter 2017 CS 162: Programming Languages Summer 2016 CS 56: Advanced Applications Programming (co-instructor) Winter 2016 CS 64: Computer Organization and Design Logic Fall 2015 CS 64: Computer Organization and Design Logic Winter 2015 CS 162: Programming Languages Summer 2014 CS 24: Problem Solving with Computers II Summer 2012 CS 16: Problem Solving with Computers I Courses TA’d at the University of California, Santa Barbara Spring 2014 CS 162: Programming Languages Winter 2014 CS 162: Programming Languages Spring 2013 CS 162: Programming Languages Winter 2013 CS 162: Programming Languages Spring 2012 CS 189B: Capstone Project, Part B Winter 2012 CS 189A: Capstone Project, Part A Fall 2011 CS 170: Operating Systems Awards and Professional Service June 2016 Student Volunteer for PLDI’16 Winter 2014 Outstanding Teaching Assistant for CS 162: Programming Lan- guages vii Peer-Reviewed Publications Evaluating Test Suite Effectiveness and Assessing Student Code via Constraint Logic Programming Kyle Dewey, Phill Conrad, Michelle Craig, Elena Morozova Conference on Innovation and Technology in Computer Science Education (ITiCSE), 2017 Fuzzing the Rust Typechecker Using CLP Kyle Dewey, Jared Roesch, Ben Hardekopf Conference on Automated Software Engineering (ASE), 2015 Automated Data Structure Generation: Refuting Common Wisdom Kyle Dewey, Lawton Nichols, Ben Hardekopf International Conference on Software Engineering (ICSE), 2015 A Parallel Abstract Interpreter for JavaScript Kyle Dewey, Vineeth Kashyap, Ben Hardekopf Symposium on Code Generation and Optimization (CGO), 2015 Language Fuzzing Using Constraint Logic Programming Kyle Dewey, Jared Roesch, Ben Hardekopf Conference on Automated Software Engineering (ASE), 2014 JSAI: A Static Analysis Platform for JavaScript Vineeth Kashyap, Kyle Dewey, Ethan A. Kuefner, John Wagner, Kevin Gibbons, John Sarracino, Ben Wiedermann, Ben Hardekopf Symposium on Foundations of Software Engineering (FSE), 2014 viii Abstract Automated Black Box Generation of Structured Inputs for Use in Software Testing by Kyle Thomas Dewey A common problem in automated software testing is the need to generate many inputs with complex structure in a black-box fashion. For example, a library for manipulating red-black trees may require that inputs are themselves valid red-black trees, meaning anything invalid is not suitable for testing. As another example, in order to test code generation in a compiler, it is necessary to use input programs which are both syntac- tically valid and well-typed. Despite the importance of this problem, we observe that existing solutions are few in number and have severe drawbacks, including unreasonably slow performance and a lack of generality to testing different systems. This thesis presents a solution to this problem of black-box structured input gen- eration. I observe that test inputs can be described as solutions to systems of logical constraints, and that more expressive constraints can lead to more complex tests. In order to test effectively and generate many tests, we need high-performance constraint solvers capable of finding many solutions to these constraints. I observe that constraint logic programming (CLP) offers an expressive constraint language paired with a high- performance constraint solver, and thus serves as a potential solution to this problem. Via a series of case studies, I have found that CLP (1) is applicable to testing a wide vari- ety of systems; (2) can scale to more complex constraints than ever previously described; and (3) is often orders of magnitude faster than competing solutions. These case studies have also exposed dozens of bugs in high-profile software, including the Rust compiler and the Z3 SMT solver. ix Contents Curriculum Vitae vii Abstract ix List of Figures xiii List of Tables xvi 1 Introduction and Motivation 1 1.1 What is Black-Box Fuzzing? . .2 1.2 Why (Not) Black-Box Fuzzing? . .3 1.3 Related Work . .4 1.4 Problem with Black-Box Fuzzing: Highly Structured Input Generation . .9 1.5 Key Insights . 10 1.6 Potential Solutions . 14 1.7 Overarching Thesis . 20 2 Case Study: Generating Interesting JavaScript Programs 24 2.1 Introduction . 24 2.2 CLP for Program Generation . 25 2.3 Generating JavaScript . 31 2.4 Evaluation . 37 2.5 Conclusions . 41 3 Case Study: Generating Complex Data Structures 43 3.1 Introduction . 43 3.2 Example . 45 3.3 CLP Compared to Other Data Structure Generators . 46 3.4 Data Structures and Properties . 49 3.5 Evaluation . 55 x 3.6 Conclusions . 62 4 Case Study: Type-Based Fuzzing of the Rust Compiler 63 4.1 Introduction . 63 4.2 Generating Well-Typed Programs . 64 4.3 Finding Typechecker Bugs . 69 4.4 Testing the Rust Typechecker . 76 4.5 Evaluation . 83 4.6 Conclusions . 89 5 Case Study: Semantics-Based Fuzzing of SMT Solvers 90 5.1 Introduction . 90 5.2 Generating Satisfiable Formulas . 92 5.3 Generating Unsatisfiable Formulas . 98 5.4 Application to Bitvectors and Floating Point . 101 5.5 Evaluation . 105 5.6 Discussion . 110 5.7 Conclusions . 113 6 Case Study: Intelligent Fuzzing of Student Tokenizers and Parsers 115 6.1 Introduction . 115 6.2 Testing Tokenizers, Parsers, and Arithmetic Evaluators With CLP . 116 6.3 Student Programming Assignment . 122 6.4 Evaluation . 124 6.5 Conclusions . 131 7 Case Study: Generating Polymorphic Programs for Testing Student Typecheckers 132 7.1 Introduction . 132 7.2 SimpleScala Language . 133 7.3 A Naive CLP-Based Generator for Well-Typed SimpleScala Programs . 140 7.4 Optimizing the Naive CLP-Based Generator for Well-Typed SimpleScala Programs . 156 7.5 Results . 169 7.6 Conclusion . 169 8 Improving CLP for Testing: Typed-Prolog 171 8.1 Introduction and Motivation . 171 8.2 Related Work . 174 8.3 Problems with CLP for Test Case Generation . 175 8.4 Type System . 183 xi 8.5 Compiling Higher-Order Relations . 189 8.6 Module System . 195 8.7 Results and Discussion . 198 8.8 Conclusions . 201 9 Improving CLP for Testing: Bounding and Search-Oriented Metainter- preter 202 9.1 Introduction and Motivation . 202 9.2 Related Work . 206 9.3 Background on CLP Metainterpreters . 208 9.4 A Metainterpreter for CLP that Parameterizes Search and Bounding . 213 9.5 Composing Search and Bounding Strategies . 227 9.6 Results and Discussion . 234 9.7 Conclusions and Future Work . 235 10 Conclusions and Future Work 237 A CLP Preliminaries 238 A.1 CLP Background .