Binary Analysis Framework
Total Page:16
File Type:pdf, Size:1020Kb
Dakota State University Beadle Scholar Masters Theses & Doctoral Dissertations Spring 5-1-2017 Binary Analysis Framework Josh Stroschein Dakota State University Follow this and additional works at: https://scholar.dsu.edu/theses Part of the Information Security Commons Recommended Citation Stroschein, Josh, "Binary Analysis Framework" (2017). Masters Theses & Doctoral Dissertations. 311. https://scholar.dsu.edu/theses/311 This Dissertation is brought to you for free and open access by Beadle Scholar. It has been accepted for inclusion in Masters Theses & Doctoral Dissertations by an authorized administrator of Beadle Scholar. For more information, please contact [email protected]. Running head: BINARY ANALYSIS FRAMEWORK i BINARY ANALYSIS FRAMEWORK A graduate project submitted to Dakota State University in partial fulfillment of the requirements for the degree of Doctor of Science in Cyber Security May 2017 By Josh Stroschein Project Committee: Dr. Wayne E. Pauli Dr. Yong Wang Dr. Matthew J. Miller BINARY ANALYSIS FRAMEWORK ii PROJECT APPROVAL FORM We certify that we have read this project and that, in our opinion, it is satisfactory in scope and quality as a project for the degree of Doctor of Science in Cyber Security. Student Name: Josh Stroschein Dissertation Project Title: Binary Analysis Framework Faculty supervisor: Date: Wayne E. Pauli, Ph. D. Committee member: Date: Yong Wang, Ph. D. Committee member: Date: Matthew J. Miller, Ph. D. Date: Mark L. Hawkes Dean for Graduate Studies and Research Dakota State University ` BINARY ANALYSIS FRAMEWORK iii ABSTRACT The binary analysis of software has become an integral activity for security researchers and attackers alike. As the value of being able to exploit a vulnerability has increased, the need to discover, fix and prevent such vulnerabilities has never been greater. This paper proposes the Binary Analysis Framework, which is intended to be used by security researchers to query and analyze information about system and third party libraries. Researchers can use the tool to evaluate and discover unknown vulnerabilities in these libraries. Furthermore, the framework can be utilized to analyze mitigation techniques implemented by operating system and third- party vendors. The Binary Analysis Framework takes a novel approach to system-level security by introducing a framework that provides for binary analysis of libraries utilizing a relational data model for permanent storage of the binary instructions, as well as providing novel ways of searching and interacting with the parsed instructions. Keywords: binary analysis, reverse engineering, vulnerability research ` BINARY ANALYSIS FRAMEWORK iv DECLARATION I hereby certify that this project constitutes my own product, that where the language of others is set forth, quotation marks so indicate, and that appropriate credit is given where I have used the language, ideas, expressions or writings of another. I declare that the project describes original work that has not previously been presented for the award of any other degree of any institution. Signed, Josh Stroschein ` BINARY ANALYSIS FRAMEWORK v TABLE OF CONTENTS PROJECT APPROVAL FORM ............................................................................................................ II ABSTRACT ........................................................................................................................................... III DECLARATION ................................................................................................................................... IV TABLE OF CONTENTS ........................................................................................................................ V LIST OF TABLES .............................................................................................................................. VIII LIST OF FIGURES ............................................................................................................................... IX INTRODUCTION .................................................................................................................................... 1 1.1 INTRODUCTION OF THE PROBLEM .................................................................................................. 2 1.2 MOTIVATION .................................................................................................................................. 3 1.2.1 Software auditing. ................................................................................................................. 3 1.2.2 Binary Analysis Framework. ................................................................................................ 4 1.3 BACKGROUND ................................................................................................................................ 5 1.3.1 Commodity computing hardware. ........................................................................................ 5 1.3.2 Internet of Things phenomenon. ........................................................................................... 6 1.3.3 Security implications. ........................................................................................................... 7 1.4 COMMON MEMORY CORRUPTION VULNERABILITIES .................................................................. 11 1.4.1 Memory layout of a process. ............................................................................................... 12 1.4.2 Stack buffer overflow. ........................................................................................................ 14 1.4.3 Heap corruption. ................................................................................................................. 15 1.4.4 Use-After-Free. ................................................................................................................... 16 1.4.5 Double-Free. ....................................................................................................................... 16 1.5 EVOLUTION OF MEMORY CORRUPTION MITIGATIONS ................................................................. 16 1.5 DISSERTATION ORGANIZATION .................................................................................................... 22 LITERATURE REVIEW ...................................................................................................................... 23 2.1 GENERATING AN EXECUTABLE PROGRAM ................................................................................... 23 2.1.1 File formats. ........................................................................................................................ 24 2.2 STATIC BINARY ANALYSIS ........................................................................................................... 24 2.3 INSTRUCTION SET ARCHITECTURES ............................................................................................. 25 2.3.1 Differentiating code and data in ISAs. ................................................................................ 25 2.4 PREVAILING METHODS OF DISASSEMBLY .................................................................................... 26 2.4.1 Disassembly complications. ................................................................................................ 27 ` BINARY ANALYSIS FRAMEWORK vi 2.5 BINARY FILE FORMATS ................................................................................................................ 28 2.5.1 File format malformations. ................................................................................................. 28 2.6 BINARY ANALYSIS FRAMEWORKS ............................................................................................... 29 2.6.1 PEV. .................................................................................................................................... 29 2.6.2 BARF. ................................................................................................................................. 29 2.6.3 Angr. ................................................................................................................................... 30 2.6.4 Limitations addressed by the Binary Analysis Framework. .............................................. 30 2.7 LITERATURE REVIEW SUMMARY ................................................................................................. 32 SYSTEM DESIGN ................................................................................................................................. 33 3.1 RESEARCH APPROACH ................................................................................................................. 33 3.2 LIMITATIONS ................................................................................................................................ 34 3.3 CODE ORGANIZATION .................................................................................................................. 35 3.4 OVERVIEW OF THE BINARY ANALYSIS FRAMEWORK COMPONENTS ............................................ 36 3.4.1 Graphical user interface. ..................................................................................................... 39 3.4.2 Input file validation. ............................................................................................................ 41 3.4.3 File parsing and disassembling. .........................................................................................