SYSTEMATIC METHODS for MEMORY ERROR DETECTION and EXPLOITATION HU HONG (B.Eng., Huazhong University of Science and Technology) A
Total Page:16
File Type:pdf, Size:1020Kb
SYSTEMATIC METHODS FOR MEMORY ERROR DETECTION AND EXPLOITATION HU HONG (B.Eng., Huazhong University of Science and Technology) A THESIS SUBMITTED FOR THE DEGREE OF DOCTOR OF PHILOSOPHY DEPARTMENT OF COMPUTER SCIENCE NATIONAL UNIVERSITY OF SINGAPORE 2016 DECLARATION I hereby declare that this thesis is my original work and it has been written by me in its entirety. I have duly acknowledged all the sources of information which have been used in the thesis. This thesis has also not been submitted for any degree in any university previously. HU HONG 13 January 2016 ACKNOWLEDGEMENTS I am grateful to have my PhD study in National University of Singapore. During this long journey, many people provided their generous supports and suggestions. Without them, I will not harvest so much, including this thesis. First, I thank my advisor, Prof. Zhenkai Liang, who leads me to the exciting topic of binary analysis and guides me to be a security researcher with his insight and experi- ence. Along the challenging journey, he always advised me to think questions at a high level and dive to deep technical details, with great patience and perseverance. Thanks to Prof. Prateek Saxena for insightful comments on several papers that constitute this thesis. I learned a lot from the discussion and collaboration with him. Thanks to several professors who gave me valuable feedback to my thesis: Prof. Abhik Roychoudhury, Prof. Roland Yap and Prof. Heng Yin. They have expanded my research view and help me find the shortcomings of my work. I am indebted to all of my collaborators for their hard working and strong support. The ideas inside this thesis were inspired from discussions with them. The solid result of each project witnesses our creative collaboration. I enjoyed working with Adrian Sendroiu, Zheng Leong Chua, Shweta Shinde, Xinshu Dong, Xiaolei Li, Li Li, Guang- dong Bai and Yaoqi Jia. Many thanks to all my lab-mates, from whom I have learned broadly in research and daily life: Kailas Patil, Ting Dai, Mingwei Zhang, Chunwang Zhang, Sai Sathya- narayan, Bodhisatta Barman Roy, Enrico Budianto, Shruti Tople, Ziqi Yang, Loi Luu, Hung Dang and many others. Thanks to my family for their selfless support. My mother and farther have worked so hard to support my long journey. My sister carefully takes care of my parents, pro- viding me the freedom to pursuit my dreams. Finally, special thanks to my beloved fiancee,´ Dongyan Zhang, who has persistently encouraged and accompanied me for the hard times. i ii Contents Summary vii List of Tables ix List of Figures xi 1 Introduction1 1.1 Thesis Overview ............................. 4 2 Background7 2.1 Memory Error & Detection........................ 7 2.1.1 Memory Error Detection..................... 8 2.1.2 Memory Safety Enforcement................... 9 2.2 Exploits and Defenses........................... 10 2.2.1 Memory Error Exploits...................... 11 2.2.2 Defenses against Exploits .................... 13 2.3 The Need for New Systematic Solutions................. 18 3 Detecting Memory Errors in Privilege-Separated Software 19 3.1 Problem.................................. 22 3.1.1 Motivating Example ....................... 22 3.1.2 Problem Definition........................ 23 3.1.3 Memory Access Patterns to Detect DUIs ............ 24 3.2 Design................................... 25 3.2.1 Overview............................. 25 3.2.2 Suspicious Instruction Shortlisting................ 26 3.2.3 Dereference Behavior Analysis ................. 27 3.3 Implementation.............................. 30 iii 3.3.1 Taint Propagation......................... 30 3.3.2 Access Formula Generation ................... 31 3.4 Evaluation................................. 32 3.4.1 Efficacy.............................. 32 3.4.2 Performance ........................... 37 3.4.3 Security Implications....................... 37 3.5 Discussion................................. 38 3.6 Related Work............................... 39 3.7 Summary ................................. 40 4 Exploiting Memory Errors with Data-Flow Stitching 41 4.1 Problem Definition ............................ 44 4.1.1 Motivating Example ....................... 44 4.1.2 Objectives & Threat Model.................... 46 4.1.3 Problem Definition........................ 47 4.1.4 Key Technique & Challenges................... 48 4.2 Data-Flow Stitching............................ 50 4.2.1 Basic Stitching Technique .................... 51 4.2.2 Advanced Stitching Technique.................. 52 4.2.3 Challenges from ASLR...................... 56 4.3 The FLOWSTITCH System........................ 60 4.3.1 Memory Error Influence Analysis................ 61 4.3.2 Security-Sensitive Data Identification.............. 62 4.3.3 Stitching Candidate Selection .................. 63 4.3.4 Candidate Filtering........................ 64 4.4 Implementation.............................. 65 4.5 Evaluation................................. 66 4.5.1 Efficacy in Exploit Generation.................. 66 4.5.2 Reduction in Search Space.................... 68 4.5.3 Performance ........................... 69 4.5.4 Case Studies ........................... 70 4.6 Related Work............................... 72 iv 4.7 Summary ................................. 73 5 Exploiting Memory Errors with Data-Oriented Programming 75 5.1 Problem.................................. 78 5.1.1 Background: Data-Oriented Attacks............... 78 5.1.2 Example of Data-oriented Programming............. 79 5.1.3 Research Questions........................ 81 5.2 Data-Oriented Programming....................... 81 5.2.1 DOP Overview.......................... 81 5.2.2 Data-Oriented Gadgets...................... 82 5.2.3 Gadget Dispatcher ........................ 85 5.2.4 MINDOP is Turing-Complete.................. 88 5.3 DOP Attack Construction......................... 91 5.3.1 Challenges ............................ 91 5.3.2 Gadget Identification....................... 92 5.3.3 Dispatcher Identification..................... 94 5.3.4 Attack Construction ....................... 95 5.4 Evaluation................................. 96 5.4.1 Feasibility of DOP........................ 96 5.4.2 Turing-Complete Examples ...................101 5.4.3 Why are Expressive Payloads Useful? . 106 5.4.4 Immunity against Control-Oriented Defenses . 110 5.5 Discussion.................................110 5.5.1 Re-Enabling Control-Hijacking Attacks . 110 5.5.2 Example Vulnerable Programs..................111 5.5.3 Potential Defenses for DOP ...................114 5.6 Related Work...............................116 5.7 Summary .................................117 6 Conclusion 119 Bibliography 121 v vi Summary Memory errors are persistent threats to computer systems. Attacks exploiting mem- ory errors have resulted in severe damage in real-world programs. Every year more than one thousand of memory errors are detected and reported. At the same time, ex- ploit mechanisms are rapidly evolving, enabling attacks to bypass known protections. To prevent the huge damage from memory errors, it is crucial for us to detect them in advance and predict their evolving trend. In this thesis, we use systematic methods to detect memory errors and discover new attack mechanisms. We propose and implement three novel methods. In the first work, we detect memory errors shown in the privilege-based isolation. In particular, we iden- tify the arbitrary memory access vulnerability, where the unexpected memory access can cross different isolated partitions. We build a systematic way to detect the arbitrary memory access code and report its severity. Then we look into new exploit mecha- nisms that can bypass most of the known defense mechanisms – the data-oriented attack. Specifically, we study the consequence of data flow re-construction, where original data flows in a program are split into small pieces and are reconnected by memory errors. We propose a novel method, called data-flow stitching, to connect disjoint data flows to either leak information or gain extra privileges. Data-flow stitching significantly en- larges the capability of data-oriented attacks. Finally, we explore the expressiveness of data-oriented attacks regarding the ability to perform arbitrary computations. We pro- pose a new exploit construction technique, called data-oriented programming (DOP), to selectively stitch basic data-flow gadgets for a desired purpose. With DOP, we build Turing-complete data-oriented attacks resulted from common memory errors, demon- strating the strong expressiveness. vii viii List of Tables 3.1 Performance of the DUI Detector..................... 37 4.1 Deterministic memory region size .................... 57 4.2 Experiment environments and benchmarks. ............... 66 4.3 Evaluation of FLOWSTITCH ....................... 67 4.4 FLOWSTITCH performance........................ 69 5.1 Simulating a loop with DOP attack.................... 81 5.2 MINDOP language............................ 82 5.3 Example data-oriented gadget of addition operation........... 84 5.4 Example data-oriented gadget of assignment operation.......... 85 5.5 Example data-oriented gadget of dereference operation......... 85 5.6 Transition table for a Turing machine that shifts the binary input by one bit (equivalent to SHL instruction)..................... 90 5.7 Prevalence of DOP gadgets & dispatchers................ 97 5.8 Reachability and corruptibility of x86 gadgets in the presence of a spe- cific memory error ............................ 99 ix x List of Figures 3.1 Design of the DUI Detector.......................