IDENTIFYING CODE INJECTION AND REUSE PAYLOADS IN MEMORY ERROR EXPLOITS Kevin Z. Snow A dissertation submitted to the faculty of the University of North Carolina at Chapel Hill in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Department of Computer Science. Chapel Hill 2014 Approved by: Fabian Monrose Don Smith Montek Singh Michael Bailey Niels Provos ©2014 Kevin Z. Snow ALL RIGHTS RESERVED ii ABSTRACT KEVIN Z. SNOW: Identifying Code Injection and Reuse Payloads In Memory Error Exploits (Under the direction of Fabian Monrose) Today’s most widely exploited applications are the web browsers and document readers we use every day. The immediate goal of these attacks is to compromise target systems by executing a snippet of malicious code in the context of the exploited application. Technical tactics used to achieve this can be classified as either code injection – wherein malicious instructions are directly injected into the vulnerable program – or code reuse, where bits of existing program code are pieced together to form malicious logic. In this thesis, I present a new code reuse strategy that bypasses existing and up-and-coming mitigations, and two methods for detecting attacks by identifying the presence of code injection or reuse payloads. Fine-grained address space layout randomization efficiently scrambles program code, limiting one’s ability to predict the location of useful instructions to construct a code reuse payload. To expose the inadequacy of this exploit mitigation, a technique for “just-in-time” exploitation is developed. This new technique maps memory on-the-fly and compiles a code reuse payload at runtime to ensure it works in a randomized application. The attack also works in face of all other widely deployed mitigations, as demonstrated with a proof-of-concept attack against Internet Explorer 10 in Windows 8. This motivates the need for detection of such exploits rather than solely relying on prevention. Two new techniques are presented for detecting attacks by identifying the presence of a payload. Code reuse payloads are identified by first taking a memory snapshot of the target application, then statically profiling the memory for chains of code pointers that reuse code to implement malicious logic. Code injection payloads are identified with runtime heuristics by leveraging hardware virtualization for efficient sandboxed execution of all buffers in memory. Employing both detection methods together to scan program memory takes about a second and produces negligible false positives and false negatives provided that the given exploit is functional and triggered in the target application version. Compared to other strategies, such as the use of signatures, this approach iii requires relatively little effort spent on maintenance over time and is capable of detecting never before seen attacks. Moving forward, one could use these contributions to form the basis of a unique and effective network intrusion detection system (NIDS) to augment existing systems. iv To my mother, who encouraged and inspired me from day one. v ACKNOWLEDGEMENTS I have been supported and encouraged over the last five years by a great number individuals. Foremost, I would like to express sincere gratitude to my advisor, Fabian Monrose, for his guidance and mentorship. I am truly fortunate to have had the opportunity to leverage his knowledge and experience on this journey from beginning to end. I would also like to express appreciation to my committee members, Don Smith, Montek Singh, Michael Bailey and Niels Provos, for their support in thoughtful discussion, suggestions and diligent proofreading. Similarly, I thank my fellow doctoral students—Srinivas Krishnan, Teryl Taylor and Andrew White—and collaborators—Lucas Davi, Alexandra Dmitrienko, Christopher Liebchen, Nathan Otterness, Ahmad-Reza Sadeghi, Blaine Stancill and Jan Werner—for engaging conversations, constructive criticism, technical contributions and overall friendship. My research would not have been possible without the generous support of a Google Research Award and National Science Foundation grants CNS-0915364 and OCI-1127361. Finally, I thank my family for their care and encouragement over the years. Most importantly, my wife has endured countless nights with my eyes glued to a computer screen and has made innumerable sacrifices along the way. She has always given me her unwavering love and support, without which I would not have had the enthusiastic determination to move forward. I am truly appreciative. vi PREFACE This dissertation is original, unpublished, independent work by the author, Kevin Z. Snow, except where due reference is made in the text of the dissertation. vii TABLE OF CONTENTS LIST OF TABLES . xii LIST OF FIGURES . xiii LIST OF ABBREVIATIONS . xv 1 Introduction . 2 1.1 A Brief History of Exploitation and Mitigation . 4 1.2 Just-in-Time Code Reuse . 5 1.3 Detecting Code Reuse Payloads . 6 1.4 Detecting Code Injection Payloads . 7 1.5 Contributions . 7 2 Background . 10 2.1 Memory Errors. 12 2.2 Code Reuse . 20 2.3 Code Injection . 25 3 Just-in-Time Code Reuse. 28 3.1 Literature Review . 29 3.2 Assumptions and Adversarial Model . 30 3.3 Method . 32 3.3.1 Mapping Code Page Memory . 33 3.3.2 API Function Discovery . 37 3.3.3 Gadget Discovery . 38 3.3.4 Just-In-Time Compilation . 40 3.4 Implementation . 42 viii 3.5 Evaluation . 45 3.5.1 On Code Page Harvesting . 45 3.5.2 On Gadget Coverage. 47 3.5.3 On API Function Discovery . 49 3.5.4 On Runtime Performance . 49 3.6 Real-world Applications . 52 3.6.1 Drive-by Downloads . 52 3.6.2 Remotely Leaking Password Manager Credentials. 52 3.7 Discussion . 55 4 Detecting Code Reuse Payloads . 58 4.1 Literature Review . 59 4.2 ROP Payload Example . 60 4.3 Method . 61 4.3.1 Unpacking Payload Containers . 62 4.3.2 Efficient Scanning of Memory Snapshots . 64 4.3.3 Gadget Candidate Profiling . 65 4.3.4 ROP Chain Profiling . 66 4.4 Evaluation . 68 4.4.1 On Payload and Gadget Space. 68 4.4.2 On Gadgets and Runtime Performance. 69 4.4.3 On Accuracy . 70 4.5 Diagnostics . 72 4.6 Limitations in Face of a Skilled Adversary. 73 4.7 Architecture and OS Specificity . 75 4.8 Discussion and Lessons Learned . ..