Iowa State University Capstones, Theses and Graduate Theses and Dissertations Dissertations 2021 Trust, transforms, and control flow: A graph-theoretic method to verifying source and binary control flow equivalence Ryan Christopher Goluch Iowa State University Follow this and additional works at: https://lib.dr.iastate.edu/etd Recommended Citation Goluch, Ryan Christopher, "Trust, transforms, and control flow: A graph-theoretic method to verifying source and binary control flow equivalence" (2021). Graduate Theses and Dissertations. 18498. https://lib.dr.iastate.edu/etd/18498 This Thesis is brought to you for free and open access by the Iowa State University Capstones, Theses and Dissertations at Iowa State University Digital Repository. It has been accepted for inclusion in Graduate Theses and Dissertations by an authorized administrator of Iowa State University Digital Repository. For more information, please contact [email protected]. Trust, transforms, and control flow: A graph-theoretic method to verifying source and binary control flow equivalence by Ryan Christopher Goluch A thesis submitted to the graduate faculty in partial fulfillment of the requirements for the degree of MASTER OF SCIENCE Major: Computer Engineering (Secure and Reliable Computing) Program of Study Committee: Suresh Kothari, Major Professor Samik Basu Akhilesh Tyagi The student author, whose presentation of the scholarship herein was approved by the program of study committee, is solely responsible for the content of this thesis. The Graduate College will ensure this thesis is globally accessible and will not permit alterations after a degree is conferred. Iowa State University Ames, Iowa 2021 Copyright © Ryan Christopher Goluch, 2021. All rights reserved. ii DEDICATION I would like to dedicate this work to my family, friends, colleagues, and mentors. I am grateful to know you all and have you in my life. You all have helped get me to this point, become who I am today, and contribute a verse. Thank you. iii TABLE OF CONTENTS Page LIST OF TABLES . .v LIST OF FIGURES . vi ACKNOWLEDGMENTS . ix ABSTRACT . .x CHAPTER 1. OVERVIEW . .1 1.1 Research Theme . .1 1.2 Thesis Overview . .2 CHAPTER 2. A NEW APPROACH . .3 2.1 Background and Related Work . .3 2.2 Control Flow . .4 2.2.1 Extracting Control Flow from Binary Code . .5 2.2.2 Control Flow Based Obfuscation . .7 2.2.3 Deobfuscation . .8 2.2.4 Control Flow Based Software Security . .9 2.3 Graph Isomorphism . .9 2.3.1 Graph Isomorphism Techniques . 10 2.4 Our Approach . 11 2.5 Practicality of Our Approach . 13 CHAPTER 3. TRANSFORM MOTIVATION . 15 3.1 Practicality of Graph Transforms . 16 3.2 Transform Cases . 16 3.2.1 Case 1 . 16 3.2.2 Case 2 . 19 3.2.3 Case 3 . 21 3.3 Transform Algorithm Overview . 23 3.3.1 Transform Algorithm . 25 CHAPTER 4. STATIC FUNCTION TRANSFORM . 27 4.1 XINU Example . 28 4.2 Static Function Transform Overview . 31 4.2.1 Finding Static Function Calls . 33 4.2.2 Create and Store Original Static CFGs . 34 iv 4.2.3 Create Transformed Graph . 34 4.2.4 Application to XINU Example . 35 4.3 Implications of the Static Transform . 35 CHAPTER 5. SWITCH STATEMENT TRANSFORM . 44 5.1 Switch Statement Transform . 44 5.2 Working Example . 45 5.2.1 Finding Switch and Case Nodes . 46 5.2.2 Sorting Cases in Ascending Order . 48 5.2.3 Creation of 2-way Branches . 48 5.2.4 Switch Block Replacement . 51 5.2.5 Transformed Working Example . 53 5.3 XINU Example . 56 5.4 Implications of the Switch Transform . 58 CHAPTER 6. SHORT CIRCUIT TRANSFORM . 60 6.1 Working Example . 61 6.1.1 Finding CC Nodes . 61 6.1.2 Processing non-CC Nodes to Retain . 63 6.1.3 Process CC Nodes . 65 6.1.4 Transformed Working Example . 66 6.2 Implications of Design Decision . 68 CHAPTER 7. ISOMORPHISM ALGORITHM TO CHECK CONTROL FLOW EQUIVA- LENCE .............................................. 74 7.1 Introduction . 74 7.2 Description of the Isomorphic Algorithm . 74 CHAPTER 8. RESULTS . 81 8.1 Categorization using Isomorphism . 81 8.1.1 G-Iso Example . 82 8.1.2 L-Iso Example . 82 8.1.3 A-Iso Example . 83 8.1.4 N-Iso Example . 84 8.2 XINU Results . 85 8.3 Discussion of Results . 87 CHAPTER 9. CONCLUSION AND FUTURE RESEARCH DIRECTIONS . 92 9.1 Future Research Directions . 93 BIBLIOGRAPHY . 95 v LIST OF TABLES Page Table 6.1 CFG Edge Combinations . 63 Table 8.1 XINU Isomorphism Results . 85 Table 8.2 XINU Overlap Results . 87 vi LIST OF FIGURES Page Figure 3.1 sgetch.c Source CFG . 18 Figure 3.2 sgetch.c Binary CFG . 18 Figure 3.3 sgetch.c Source CFG . 18 Figure 3.4 sgetch.c Binary CFG . 18 Figure 3.5 sgetch.c Source Transformed CFG . 19 Figure 3.6 sgetch.c Binary Transformed CFG . 19 Figure 3.7 signal.c Source CFG . 21 Figure 3.8 signal.c Binary CFG . 21 Figure 3.9 signal.c Source Marked CFG . 22 Figure 3.10 signal.c Binary Marked CFG . 22 Figure 3.11 signal.c Source Transformed CFG . 22 Figure 3.12 signal.c Binary Transformed CFG . 22 Figure 3.13 fputc.c Source CFG . 24 Figure 3.14 fputc.c Binary CFG . 24 Figure 3.15 fputc.c Source Transformed CFG . 24 Figure 3.16 fputc.c Binary Transformed CFG . 24 Figure 3.17 fputc.c Correct Source Transformed CFG . 24 Figure 4.1 qsort.c Source and Binary CFGs . 28 vii Figure 4.2 partition() Source CFG . 32 Figure 4.3 swap elements() Source CFG . 32 Figure 4.4 qsort.c Call Graph . 32 Figure 4.5 partition() Call Graph . 32 Figure 4.6 qsort.c Transformed CFG . 37 Figure 4.7 create.c removed conditional . 41 Figure 4.8 create.c Source and Binary.