SOURCE-FREE BINARY MUTATION FOR OFFENSE AND DEFENSE by Vishwath R. Mohan APPROVED BY SUPERVISORY COMMITTEE: Kevin W. Hamlen, Chair Alvaro C´ardenas Latifur Khan Zhiqiang Lin Copyright c 2014 Vishwath R. Mohan All rights reserved Dedicated to my parents, who encouraged without question. To my wife, for lifting me far beyond where I could have flown myself. To my grandfather, more technology-aware than most PhDs I know. SOURCE-FREE BINARY MUTATION FOR OFFENSE AND DEFENSE by VISHWATH R. MOHAN, BS, MS DISSERTATION Presented to the Faculty of The University of Texas at Dallas in Partial Fulfillment of the Requirements for the Degree of DOCTOR OF PHILOSOPHY IN COMPUTER SCIENCE THE UNIVERSITY OF TEXAS AT DALLAS December 2014 ACKNOWLEDGMENTS This dissertation could not have been completed without the help of the author's advisor, Dr. Kevin Hamlen, who served as inspiration, role model and walking database of both ideas and knowledge. Richard Wartell, the author's research partner and collaborator on some of the research presented in the dissertation, deserves a huge shout out. This dissertation owes a lot to not just his invaluable research assistance, but also his continued friendship and support. Because no one should have to delve the depths of x86 machine code alone. The author wishes to thank Dr. Zhiqiang Lin, Dr. Latifur Khan, and Dr. Mehedy Masud, whose contributions and ideas, both defensive and offensive, greatly helped this dissertation achieve its goals. Special thanks should also be given to Dr. Per Larsen, who provided the seed of the idea that eventually became Opaque CFI. He also proved to be a motivating collaborator and good friend, for which the author is grateful. Finally, the author wishes to thank his wife, Sanjana Raghunath, for her patience and constant support. The research reported in this dissertation was supported in part by the Air Force Office of Scientific Research (AFOSR) under Young Investigator Program (YIP) award FA9550-08-1- 0044 and Active Defense award FA9550-10-1-0088, the National Science Foundation (NSF) under CAREER award #1054629, the Office of Naval Research (ONR) under award N00014- 14-1-0030, and an NSF Industry-University Collaborative Research Center (IUCRC) award v from Raytheon Company. All opinions, recommendations, and conclusions expressed are those of the authors and not necessarily of the AFOSR, NSF, ONR, or Raytheon. November 2014 vi PREFACE This dissertation was produced in accordance with guidelines which permit the inclusion as part of the dissertation the text of an original paper or papers submitted for publication. The dissertation must still conform to all other requirements explained in the \Guide for the Preparation of Master's Theses and Doctoral Dissertations at The University of Texas at Dallas." It must include a comprehensive abstract, a full introduction and literature review, and a final overall conclusion. Additional material (procedural and design data as well as descriptions of equipment) must be provided in sufficient detail to allow a clear and precise judgment to be made of the importance and originality of the research reported. It is acceptable for this dissertation to include as chapters authentic copies of papers already published, provided these meet type size, margin, and legibility requirements. In such cases, connecting texts which provide logical bridges between different manuscripts are mandatory. Where the student is not the sole author of a manuscript, the student is required to make an explicit statement in the introductory material to that manuscript describing the student's contribution to the work and acknowledging the contribution of the other author(s). The signatures of the Supervising Committee which precede all other material in the dissertation attest to the accuracy of this statement. vii SOURCE-FREE BINARY MUTATION FOR OFFENSE AND DEFENSE Publication No. Vishwath R. Mohan, PhD The University of Texas at Dallas, 2014 Supervising Professor: Kevin W. Hamlen The advent of advanced weaponized software over the past few years, including the Stuxnet, Duqu, and Flame viruses, is indicative of the seriousness with which advanced persistent threats (APTs) have begun to treat the cyber-realm as a potential theatre for offensive military action and espionage. This has coincided with a strong interest in creating malware obfuscations that hide their payloads for extended periods of time, even while under active search. Progress on this front threatens to render conventional software defenses obsolete, placing the world in dire need of more resilient software security solutions. This dissertation underlines the seriousness of this threat through the design and imple- mentation of two novel, next-generation malware obfuscation technologies that bypass to- day's widely deployed defenses. Unlike conventional polymorphic malware, which mutates randomly in an effort to evade detection, the presented attacks are reactively adaptive in the sense that they intelligently surveil, analyze, and adapt their obfuscation strategies in the wild to understand and defeat rival defenses. The dissertation then presents three novel software defenses that offer strengthened software security against both current and future offensive threats. Rather than attempting to detect threats statically (i.e., before viii they execute), or introducing dynamic monitors that raise compatibility and performance penalties for consumers, the new defenses implement automated, source-free, binary soft- ware transformations that preemptively transform untrusted software into safe software. Experiments show that this security retrofitting approach offers higher performance, greater security, and more flexible deployment options relative to competing approaches. Thus, binary code transformation and mutation is realized as both a powerful offensive and a potent defensive paradigm for software attacks and defenses. ix TABLE OF CONTENTS ACKNOWLEDGMENTS . v PREFACE . vii ABSTRACT . viii LIST OF FIGURES . xiv LIST OF TABLES . xvi CHAPTER 1 INTRODUCTION . 1 CHAPTER 2 BACKGROUND . 7 2.1 Malware Detection and Obfuscation . .7 2.2 Code-Reuse Attacks and Defenses . .9 2.3 Binary Rewriting and In-lined Reference Monitors . 10 2.4 Challenges with Source-Free Disassembly . 12 PART I MALWARE OFFENSE . 15 CHAPTER 3 EXPLOITING AN ANTIVIRUS INTERFACE . 16 3.1 Overview . 18 3.2 A data mining based malware detection model . 19 3.2.1 Feature extraction . 21 3.2.2 Training . 24 3.2.3 Testing . 24 3.3 Model-reversing Obfuscations . 25 3.3.1 Path Selection . 26 3.3.2 Feature Insertion . 27 3.3.3 Feature Removal . 30 3.4 Experiments . 32 3.4.1 Dataset . 32 3.4.2 Interface Exploit Experiment . 33 x 3.4.3 Model-driven Obfuscation Experiment . 34 3.5 Conclusion . 35 CHAPTER 4 FRANKENSTEIN . 38 4.1 Design . 40 4.1.1 Gadgets . 40 4.1.2 Semantic Blueprint . 42 4.1.3 Gadget Discovery . 44 4.1.4 Gadget Arrangement . 46 4.1.5 Gadget Assignment . 47 4.1.6 Executable Synthesis . 47 4.2 Implementation . 48 4.3 Experimental Results . 49 4.4 Conclusion . 53 PART II DEFENSIVE SOFTWARE TECHNOLOGIES . 54 CHAPTER 5 VERIFIED SYSTEM CALL SAFETY ENFORCEMENT . 55 5.1 Background . 57 5.1.1 Assumptions . 57 5.1.2 Threat model . 57 5.1.3 Attacks . 58 5.2 System Overview . 59 5.3 Detailed Design . 60 5.4 Implementation . 69 5.5 Evaluation . 71 5.5.1 Rewriting Effectiveness . 71 5.5.2 Performance Overhead . 72 5.5.3 Policy Enforcement Library Synthesis . 73 5.5.4 Case Studies . 75 5.6 Discussion . 78 5.6.1 Control-flow Policies . 78 xi 5.6.2 Code Conventions . 79 5.6.3 Other Future Work . 81 5.7 Conclusion . 82 CHAPTER 6 SELF-TRANSFORMING INSTRUCTION RELOCATION . 83 6.1 System Overview . 85 6.1.1 Approach Overview . 86 6.2 Detailed Design . 89 6.2.1 Static Rewriting Phase . 89 6.2.2 Load-time Stirring Phase . 92 6.2.3 An Example . 93 6.2.4 Special Cases . 94 6.3 Empirical Evaluation . 99 6.3.1 Effectiveness . 99 6.3.2 Performance Overhead . 105 6.4 Limitation and Future Work . 107 6.4.1 Randomization Entropy . 107 6.4.2 Limitations and Future Work . 108 6.5 Conclusion . 110 CHAPTER 7 OPAQUE CONTROL-FLOW INTEGRITY . 111 7.1 Threat Model . 115 7.1.1 Bypassing Coarse-Grained CFI . 115 7.1.2 Assumptions . 117 7.2 O-CFI Overview . 118 7.2.1 Bounding the Control Flow . 121 7.2.2 Opacifying Control-flow Bounds . 122 7.2.3 Tightening Control-flow Check Bounds . 124 7.2.4 Example Defense against JIT-ROP . 126 7.3 O-CFI Implementation . 128 7.3.1 Static Binary Rewriting . ..