Air Force Institute of Technology AFIT Scholar Theses and Dissertations Student Graduate Works 6-19-2014 Using State Merging and State Pruning to Address the Path Explosion Problem Faced by Symbolic Execution Patrick T. Copeland Follow this and additional works at: https://scholar.afit.edu/etd Recommended Citation Copeland, Patrick T., "Using State Merging and State Pruning to Address the Path Explosion Problem Faced by Symbolic Execution" (2014). Theses and Dissertations. 517. https://scholar.afit.edu/etd/517 This Thesis is brought to you for free and open access by the Student Graduate Works at AFIT Scholar. It has been accepted for inclusion in Theses and Dissertations by an authorized administrator of AFIT Scholar. For more information, please contact [email protected]. USING STATE MERGING AND STATE PRUNING TO ADDRESS THE PATH EXPLOSION PROBLEM FACED BY SYMBOLIC EXECUTION THESIS Patrick T. Copeland, Civilian AFIT-ENG-T-14-J-3 DEPARTMENT OF THE AIR FORCE AIR UNIVERSITY AIR FORCE INSTITUTE OF TECHNOLOGY Wright-Patterson Air Force Base, Ohio DISTRIBUTION STATEMENT A: APPROVED FOR PUBLIC RELEASE; DISTRIBUTION UNLIMITED The views expressed in this thesis are those of the author and do not reflect the official policy or position of the United States Air Force, the Department of Defense, or the United States Government. This material is declared a work of the U.S. Government and is not subject to copyright protection in the United States. AFIT-ENG-T-14-J-3 USING STATE MERGING AND STATE PRUNING TO ADDRESS THE PATH EXPLOSION PROBLEM FACED BY SYMBOLIC EXECUTION THESIS Presented to the Faculty Department of Electrical and Computer Engineering Graduate School of Engineering and Management Air Force Institute of Technology Air University Air Education and Training Command in Partial Fulfillment of the Requirements for the Degree of Master of Science in Cyber Operations Patrick T. Copeland, B.A.C.S. Civilian June 2014 DISTRIBUTION STATEMENT A: APPROVED FOR PUBLIC RELEASE; DISTRIBUTION UNLIMITED AFIT-ENG-T-14-J-3 USING STATE MERGING AND STATE PRUNING TO ADDRESS THE PATH EXPLOSION PROBLEM FACED BY SYMBOLIC EXECUTION Patrick T. Copeland, B.A.C.S. Civilian Approved: /signed/ 02 Jun 2014 Gilbert L. Peterson, PhD (Chairman) Date /signed/ 30 May 2014 Maj Thomas E. Dube, PhD (Member) Date /signed/ 30 May 2014 Barry E. Mullins, PhD (Member) Date AFIT-ENG-T-14-J-3 Abstract Symbolic execution is a promising technique to discover software vulnerabilities and improve the quality of code. However, symbolic execution suffers from a path explosion problem where the number of possible paths within a program grows exponentially with respect to loops and conditionals. New techniques are needed to address the path explosion problem. This research presents a novel algorithm which combines the previously researched techniques of state merging and state pruning. A prototype of the algorithm along with a pure state merging and pure state pruning are implemented in the KLEE symbolic execution tool with the goal of increasing the code coverage. Each algorithm is tested over 66 of the GNU COREUTILS utilities. State merging combined with state pruning outperforms the unmodified version of KLEE on 53% of the COREUTILS. These results confirm that state merging with pruning has viability in addressing the path explosion problem of symbolic execution. iv For my beautiful fianc´eeand loving parents. v Acknowledgments I would like to thank all those who have made this work possible. A special thanks goes to my fiancee´ and my parents who have supported me every step of the way. I would also like to thank my research advisor, Dr. Gilbert Peterson, for agreeing to take on this work so late in the game and providing the necessary support and guidance. Patrick T. Copeland vi Table of Contents Page Abstract......................................... iv Dedication........................................v Acknowledgments.................................... vi Table of Contents.................................... vii List of Figures...................................... xi List of Tables...................................... xii List of Acronyms.................................... xiii I. Introduction.....................................1 1.1 Motivation...................................1 1.2 Research Objectives..............................2 1.2.1 Primary Objective 1.........................2 1.2.2 Primary Objective 2.........................2 1.2.3 Primary Objective 3.........................2 1.2.4 Primary Objective 4.........................2 1.3 Methodology.................................3 1.4 Assumptions and Limitations.........................3 1.5 Thesis Structure................................3 II. Background.....................................4 2.1 Symbolic Execution..............................4 2.1.1 Path Explosion Problem.......................5 2.2 KLEE.....................................6 2.2.1 KLEE Architecture..........................6 2.2.2 KLEE Searcher............................7 2.2.3 Measuring code coverage.......................7 2.3 Path Explosion Mitigation Techniques....................7 2.3.1 Search Heuristics...........................7 2.3.1.1 User-guided........................8 2.3.1.2 Depth-First Search.....................8 vii Page 2.3.1.3 Breadth-First Search....................8 2.3.1.4 Best-First Search.....................9 2.3.1.5 Random Path Selection..................9 2.3.1.6 Generational Search .................... 10 2.3.1.7 Fitness-Guided....................... 11 2.3.2 State Pruning............................. 12 2.3.3 State Merging............................. 12 2.3.4 Concolic Testing........................... 14 2.3.5 Compositional Analysis....................... 16 2.3.6 Parallel Execution.......................... 17 2.4 Summary................................... 18 III. Methodology.................................... 20 3.1 Problem Definition.............................. 20 3.1.1 Goals and Hypothesis........................ 20 3.1.2 Approach............................... 21 3.2 State Merging................................. 22 3.2.1 Dynamic State Merging....................... 22 3.2.1.1 State Fast Forwarding................... 23 3.2.2 Costs and benefits of merging.................... 25 3.3 State Pruning................................. 25 3.4 Combining state merging and state pruning................. 26 3.5 System Boundaries.............................. 27 3.5.1 KLEE Flags.............................. 27 3.5.2 Reducing non-determinism...................... 29 3.6 Workload................................... 30 3.6.1 GNU COREUTILS.......................... 30 3.7 Performance Metrics............................. 31 3.7.1 Code coverage............................ 33 3.7.2 Instruction reduction......................... 33 3.7.3 Path reduction............................ 34 3.7.4 Average query cost.......................... 34 3.7.5 Merge time.............................. 34 3.7.6 Fast forward time........................... 34 3.8 System Parameters.............................. 35 3.9 Factors..................................... 36 3.10 Evaluation Technique............................. 38 3.11 Experimental Design............................. 38 3.12 Methodology Summary............................ 38 viii Page IV. Results and Analysis................................ 39 4.1 Overview................................... 39 4.2 KLEE-BASE................................. 40 4.3 Merging Results................................ 40 4.3.1 KLEE-MERGE-5........................... 41 4.3.1.1 Code Coverage...................... 42 4.3.1.2 Instruction Reduction................... 43 4.3.1.3 Average Query Cost.................... 43 4.3.1.4 SMT Solver Time..................... 43 4.3.1.5 Merge and Fast-Forward Success............. 43 4.3.2 KLEE-MERGE-20.......................... 44 4.3.2.1 Code Coverage...................... 44 4.3.2.2 Instruction Reduction................... 46 4.3.2.3 Average Query Cost.................... 46 4.3.2.4 SMT Solver Time..................... 46 4.3.2.5 Merge and Fast-Forward Success............. 46 4.3.3 KLEE-NO-FF-MERGE-5...................... 47 4.3.3.1 Code Coverage...................... 47 4.3.3.2 Instruction Reduction................... 48 4.3.3.3 Average Query Cost.................... 49 4.3.3.4 SMT Solver Time..................... 49 4.3.3.5 Merge Success....................... 49 4.3.4 KLEE-NO-FF-MERGE-20...................... 49 4.3.4.1 Code Coverage...................... 49 4.3.4.2 Instruction Reduction................... 50 4.3.4.3 Average Query Cost.................... 50 4.3.4.4 SMT Solver Time..................... 51 4.3.4.5 Merge Success....................... 52 4.3.5 Merging Summary.......................... 52 4.4 Pruning Results................................ 52 4.4.1.1 Code Coverage...................... 53 4.4.1.2 Instruction Reduction................... 53 4.4.1.3 Average Query Cost.................... 54 4.4.1.4 SMT Solver Time..................... 55 4.4.1.5 Pruned States....................... 55 4.4.2 Pruning Summary.......................... 55 4.5 Pruning Combined with Merging Results.................. 55 4.5.1.1 Code Coverage...................... 55 4.5.1.2 Instruction Reduction................... 56 4.5.1.3 Average Query Cost.................... 57 4.5.1.4 SMT Solver Time..................... 58 ix Page 4.5.1.5 Merge and Fast-Forward Success............. 58 4.5.1.6 Pruned States....................... 58 4.5.2 Pruning Combined with Merging Summary............. 58 4.6