Master of Science in Software Engineering May 2020 To Force a Bug: Extending Hybrid Fuzzing Johan Näslund Henrik Nero Faculty of Computing, Blekinge Institute of Technology, 371 79 Karlskrona, Sweden This thesis is submitted to the Faculty of Computing at Blekinge Institute of Technology in partial fulfilment of the requirements for the degree of Master of Science in Software Engineering. The thesis is equivalent to 20 weeks of full time studies. The authors declare that they are the sole authors of this thesis and that they have not used any sources other than those listed in the bibliography and identified as references. They further declare that they have not submitted this thesis at any other institution to obtain a degree. Contact Information: Authors: Johan Näslund E-mail: [email protected] Henrik Nero E-mail: [email protected] University advisor: Doctor Dragos Ilie Department of Computer Science Faculty of Computing Internet : www.bth.se Blekinge Institute of Technology Phone : +46 455 38 50 00 SE–371 79 Karlskrona, Sweden Fax : +46 455 38 50 57 Abstract One of the more promising solutions for automated binary testing today is hybrid fuzzing, a combination of the two acknowledged approaches, fuzzing and symbolic execution, for detecting errors in code. Hybrid fuzzing is one of the pioneering works coming from the authors of Angr and Driller, opening up for the possibility for more specialized tools such as QSYM to come forth. These hybrid fuzzers are coverage guided, meaning they measure their success in how much code they have covered. This is a typical approach but, as with many, it is not flawless. Just because a region of code has been covered does not mean it has been fully tested. Some flaws depend on the context in which the code is being executed, such as double-free vulnerabilities. Even if the free routine has been invoked twice, it does not mean that a double-free bug has occurred. To cause such a vulnerability, one has to free the same memory chunk twice (without it being reallocated between the two invocations to free). In this research, we will extend one of the current state-of-the-art hybrid fuzzers, QSYM, which is an open source project. We do this extension, adding double- free detection, in a tool we call QSIMP. We will then investigate our hypothesis, stating that it is possible to implement such functionality without losing so much performance that it would make the tool impractical. To test our hypothesis we have designed two experiments. One experiment tests the ability of our tool to find double-free bugs (the type of context-sensitive bug that we have chosen to test with). In our second experiment, we explore the scalability of the tool when this functionality is executed. Our experiments showed that we were able to implement context-sensitive bug detection within QSYM. We can find most double-free vulnerabilities we have tested it on, although not all, because of some optimizations that we were unable to build past. This has been done with small effects on scalability according to our tests. Our tool can find the same bugs that the original QSYM while adding functionality to find double-free vulnerabilities. Keywords: Symbolic execution, fuzzing, context-sensitive, bug i Sammanfattning En av de mer lovande lösningarna för automatiserad binärtestning är i dagsläget hy- brid fuzzing, en kombination av två vedertagna tillvägagångssätt, fuzzing och sym- bolisk exekvering. Forskarna som utvecklade Angr och Driller anses ofta vara några av de första med att testa denna approach. Detta har i sin tur öppnat upp för fler mer specialiserade verktyg som QSYM. Dessa hybrid fuzzers mäter oftast sin framgång i hänsyn till hur mycket kod som nås under testningen. Detta är ett typiskt tillvägagångssätt, men som med många metoder är det inte felfri. Kod som har exekverats, utan att en bugg utlösts, är inte nödvändigtvis felfri. Vissa buggar beror på vilken kontext maskininstruktioner exekveras i – ett exempel är double-free sårbarheter. Att minne har frigjorts flera gånger betyder inte ovillkorligen att en double-free sårbarhet har uppstått. För att en sådan sårbarhet ska uppstå måste samma minne frigöras flera gånger (utan att detta minne omallokerats mellan anropen till free). I detta projekt breddar vi en av de främsta hybrid fuzzers, QSYM, ett projekt med öppen källkod. Det vi tillför är detektering av double-free i ett verktyg vi kallar QSIMP. Vi undersöker sedan vår hypotes, som säger att det är möjligt att implementera sådan funktionalitet utan att förlora så mycket prestanda att det gör verktyget opraktiskt. För att bepröva hypotesen har vi designat två experiment. Ett experiment tes- tar verktygets förmåga att detektera double-free sårbarheter (den sortens kontext- känsliga sårbarheter vi har valt att fokusera på). I det andra experimentet utforskar vi huruvida verktyget är skalbart då den nya funktionaliteten körs. Våra experiment visar att vi har möjliggjort detektering av kontext-känsliga bug- gar genom vidareutveckling av verktyget QSYM. QSIMP hittar double-free buggar, dock inte alla, på grund av optimiseringar som vi ej har lyckats arbeta runt. Detta har gjorts utan större effekter på skalbarheten av verktyget enligt resultaten från våra experiment. Vårt verktyg hittar samma buggar som orignal verktyget QSYM, samtidigt som vi tillägger funktionalitet för att hitta double-free sårbarheter. Nyckelord: symbolisk exekvering, fuzzing, kontext-känslig, bug iii Acknowledgments We would like to greatly thank Dr. Dragos Ilie for his assistance, counsel, and proof- readings throughout our research. We also want to express our gratitude to Erik Bergenholtz for, alongside his own research, actively participating in our meetings and discussions. Additionally, we would like to thank Martin Strand for his supervision, engage- ment, and excellent guidance on the practical matters of binary analysis. Also, we are very grateful for the help of Dr. Carl Löndahl with getting started, both theoret- ically and academically, proofreading, and opening up the possibility to work with TrueSec and Säkerhetskontoret. Finally, we want to thank Mikael Lagström and TrueSec for giving us an inspir- ing workplace for our thesis, and for lending us the hardware needed to run the experiments. v Contents Abstract i Sammanfattning iii Acknowledgments v 1 Introduction 1 1.1 Motivation . 1 1.2 Problem Formulation . 2 1.2.1 Research Questions . 3 1.2.2 Hypothesis . 3 1.2.3 Scope . 3 1.3 Thesis Outline . 3 2 Background 5 2.1 Bugs . 5 2.2 Fuzzing . 6 2.3 Dynamic Symbolic Execution . 7 2.3.1 Soundness . 7 2.3.2 Concretization . 8 2.4 Hybrid Fuzzing . 8 2.5 Context-Sensitive Bugs . 10 3 Related Work 13 3.1 Satisfiable Modulo Theory Solvers . 13 3.2 Automated Vulnerability Discovery . 13 3.3 Hybrid Fuzzing . 14 3.4 Symbolic Memory . 15 4 Method 17 4.1 Current State of the Art . 17 4.2 Implementation . 18 4.2.1 DF Detection Algorithm . 18 4.2.2 Libdislocator . 20 4.2.3 Symbolic Load . 20 4.3 Dataset . 20 4.4 Evaluation . 22 4.4.1 Context-Sensitive Binaries Experiment . 23 vii 4.4.2 Scalability Experiment . 23 4.5 Equipment . 24 5 Results and Analysis 25 5.1 Context-Sensitive Bugs Experiment . 25 5.2 Scalability Experiment . 25 5.3 Analysis . 27 5.3.1 Context-Sensitive Bugs Experiment . 30 5.3.2 Scalability and Efficiency Experiment . 30 5.3.3 Combined Result Analysis . 31 6 Discussion 33 6.1 Conducted Research . 33 6.1.1 RQ1: DF Triggering Using SMT . 33 6.1.2 RQ2: Scalability . 34 6.2 General Discussion . 34 6.2.1 Further Applicability . 35 6.2.2 Implementation Weaknesses . 36 6.3 Limitations . 36 6.3.1 Equipment around QSYM . 36 6.3.2 Symbolic Memory . 37 6.3.3 Datasets . 37 6.3.4 Libdislocator . 38 6.4 Validity Threats . 38 7 Conclusions and Future Work 39 7.1 Conclusion . 39 7.2 Future Work . 39 7.2.1 Further optimizations . 39 7.2.2 Other interesting ideas . 40 A Graphs of path coverage 47 viii Chapter 1 Introduction Software is being run everywhere today. Almost every large company has code run- ning and security breaches commonly involve exploitation of software. Everyday objects, previously lacking networking capabilities, are being connected to the Inter- net – a growing concept commonly referred to as the Internet of Things. Day by day this digital revolution increases the risk and impact of software vulnerabilities. Be- cause the amount of code written is constantly increasing, and the difficulty finding such vulnerabilities, automation of vulnerability detection has become a necessity. Manually testing binaries can be time consuming, which is why effort is still being put into automation. Tool-assisted detection today is done via fuzzing, static analysis such as symbolic execution, and taint analysis. One of the most utilized approaches for automated testing fits very well in with the “A good programmer is a lazy programmer” paradigm, namely fuzzing. Letting the computer generate “random” input and see how the program reacts. State-of-the-art fuzzers today use evolutionary algorithms and minor instrumentation of the program to create input that is more likely to cause the program to crash [25]. The strength of fuzzers lies in their near-native execution performance. They are good at generating general input but inefficient when encountering specific constraints in binaries. Another method employed is symbolic execution (usually dynamic), simulating all possible paths of the binary while collecting variables and constraints that depend on user input. By using satisfiability modulo theory, mathematical constraint solving in computing, a symbolic execution engine can use the collected variables to see what input is needed to reach a certain branch of the program. This technique is good at handling fine-grained constraints but suffers from the path explosion problem invoked by having to explore every possible path in the binary [30, 23].