UNIVERSITY OF CALIFORNIA, IRVINE Practical Run-Time Mitigations Against Data-Oriented Attacks DISSERTATION submitted in partial satisfaction of the requirements for the degree of DOCTOR OF PHILOSOPHY in Computer Science by Prabhu karthikeyan Rajasekaran Dissertation Committee: Professor Michael Franz, Chair Professor Ardalan Amiri Sani Professor Anton Burtsev 2020 Portions of Chapter 1, 4, and 5 © 2020 ACM. Reprinted, with permission, from CoDaRR: Continuous Data Space Randomization against Data-Only Attacks, Prabhu Rajasekaran, Stephen Crane, David Gens, Yeoul Na, Stijn Volckaert, and Michael Franz., in Proceedings of the 15th ACM Asia Conference on Computer and Communications Security, ASIA CCS 2020. Portions of Chapter 1, 2, and 5 © 2019 IEEE. Reprinted, with permission, from SoK: Sanitizing for Security, Dokyung Song, Julian Lettner, Prabhu Rajasekaran, Per Larsen, Stijn Volckaert, and Michael Franz., in Proceedings of the 40th IEEE Symposium on Security and Privacy, IEEE S&P 2019. Portions of Chapter 1, 3 and 5 © 2017 USENIX. Reprinted, with permission, from Venerable Variadic Vulnerabilities Vanquished, Priyam Biswas, Alessandro Di Federico, Scott A. Carr, Prabhu Rajasekaran, Stijn Volckaert, Yeoul Na, Michael Franz, and Mathias Payer, in Proceedings of the 2017 USENIX Annual Technical Conference, USENIX Security 2017. All other materials © 2020 Prabhu karthikeyan Rajasekaran TABLE OF CONTENTS Page LIST OF FIGURES v LIST OF TABLES vi ACKNOWLEDGMENTS vii VITA viii ABSTRACT OF THE DISSERTATION x 1 Introduction 1 1.1 Control-flow Attacks and Defenses . .1 1.2 Data-oriented Attacks and Defenses . .2 1.3 Contributions . .4 1.3.1 Variadic Function Sanitizer and Mitigation . .4 1.3.2 Dynamic Data Space Randomization . .4 1.3.3 Summary of Contributions . .5 2 Background 7 2.1 Data-Oriented Attacks . .8 2.1.1 Direct Data Manipulation . .8 2.1.2 Data Oriented Programming . .9 2.2 Memory Corruption Vulnerabilities . 11 2.2.1 Memory Safety Violations . 11 2.2.2 Use of Uninitialized Variables . 13 2.2.3 Pointer Type Errors . 13 2.2.4 Variadic Function Misuse . 15 2.2.5 Other Vulnerabilities . 16 2.3 Side-Channel Attacks . 17 2.3.1 Transient Execution Attacks . 17 3 A comprehensive approach to fixing variadic vulnerabilities 20 3.1 Motivation . 21 3.2 Background . 24 3.2.1 Variadic functions . 24 ii 3.2.2 Variadic functions ABI . 25 3.2.3 Variadic attack surface . 26 3.2.4 Format string exploits . 27 3.3 Threat model . 28 3.4 Design . 29 3.4.1 Analysis and Instrumentation . 29 3.4.2 Runtime support . 30 3.4.3 Challenges and Discussion . 32 3.5 Implementation . 36 3.6 Evaluation . 39 3.6.1 Case study: CFI effectiveness . 40 3.6.2 Exploit Detection . 43 3.6.3 Prevalence of variadic functions . 45 3.6.4 Firefox . 48 3.6.5 SPEC CPU2006 . 49 3.6.6 Micro-benchmarks . 49 4 Moving Beyond Data Space Randomization 51 4.1 Motivation . 51 4.2 Background . 54 4.3 Attacking DSR . 57 4.3.1 Threat Model . 58 4.3.2 Attack 1 { Direct Memory Disclosure . 59 4.3.3 Attack 2 { Transient Execution . 60 4.4 Design of a countermeasure . 63 4.4.1 Challenges . 63 4.4.2 Overview . 65 4.4.3 Compile-time Analysis and Instrumentation . 67 4.4.4 Mask Table and Tracking Live Masks . 68 4.4.5 Value Mask Mappings . 69 4.4.6 Run-time Monitor . 70 4.5 Details of our proof-of-concept implementation . 72 4.5.1 Code Pointer Fixups . 73 4.5.2 Optimizations . 75 4.6 Evaluation . 75 4.6.1 Performance . 75 4.6.2 Security Analysis . 80 4.7 Discussion . 82 4.7.1 Generating Variants . 82 4.7.2 Attack Against Heap Metadata . 82 5 Related Work 84 5.1 Dynamic Bug Finding { Sanitizers . 84 5.2 Variadic Function Attack Surface Mitigations . 86 5.3 Security Mitigations . 87 iii 5.3.1 Control-Flow Exploit Defenses . 88 5.3.2 Data-Oriented Exploit Defenses . 90 5.4 Rerandomization Defenses . 92 6 Conclusions 94 Bibliography 96 iv LIST OF FIGURES Page 3.1 Overview of our compilation pipeline. Our instrumentation passes run right after the C/C++frontend, while our runtime library, hexvasan.a, is merged into the final executable at link time. 30 3.2 Run-time overhead of HexVASAN in the SPECint CPU2006 benchmarks, compared to baseline LLVM 3.9.1 performance. 50 4.1 Direct memory disclosure attack on a reused XOR key. R: read V, unmasking with key KR, W: write V, masking with key KW ................ 59 4.2 Overview of CoDaRR's main components. 63 4.3 CoDaRR run-time rerandomization process. 67 4.4 Average throughput of Nginx when protected by CoDaRR. We plot the median transfer rate for 5M requests against different rerandomization intervals going from 500ms to 10s. The green line shows throughput for baseline DSR without rerandomization. 77 4.5 Average throughput of Thttpd using the same configuration as in Figure 4.4. 78 4.6 SPEC 2006 performance of CoDaRR instrumentation (without rerandom- ization). Each column shows the median run time of the benchmark with CoDaRR, normalized to the median run time without our tool. 79 v LIST OF TABLES Page 2.1 Simulating MINDOP Data Oriented Programming Gadgets in C . 10 3.1 Detection coverage for several types of illegal calls to variadic functions. X indicates detection, 7 indicates non-detection. 41 3.2 Statistics of Variadic Functions for Different Benchmarks. The second and third columns are variadic call sites broken into \Tot." (total) and \Ind." (indirect); % shows the percentage of variadic call sites. The fifth and sixth columns are for variadic functions. \A.T." stands for address taken. \Proto." is the number of distinct variadic function prototypes. \Ratio" indicates the function-per-prototypes ratio for variadic functions. 45 3.3 Statistics of Variadic Functions for SPEC 2006 Benchmarks. 46 3.4 Performance overhead on Firefox benchmarks. For Octane and JetStream higher is better, while for Kraken lower is better. 48 3.5 Performance overhead in micro-benchmarks. 50 4.1 Total number of static equivalence classes and number of allocations per class. 76 4.2 Average number of bytes rewritten at dynamic rerandomization time. Globals and Heap columns show bytes re-encrypted; Registers and Stack Spills are in-flight masks replaced. 76 vi ACKNOWLEDGMENTS First and foremost I would like to thank my advisor, Professor Michael Franz, for giving me an opportunity to conduct academic research in his research group, Secure Systems and Software Lab (SSLLab) . Thank you, Michael, for your unwavering support, mentorship, and guidance through the ups and downs of my professional and personal life during my time here at UCI. I am indebted to our lab alumnus Dr. Stephen Crane for taking me under his wing. Thank you, Stephen, for those countless hours of help in debugging our code and for setting lofty standards as a computer scientist and as a human being that I strive to emulate. The postdocs at SSLLab, both current and previous, I had the good fortune to work with have had a great impact on my growth as a researcher and systems engineer. I am deeply grateful to Dr. Per Larsen, Dr. Stijn Volckaert, Dr. Yeoul Na, Dr. David Gens, and Dr. Adrian Dabrowski for their advice, ideas, and research contributions. To the illustrious lab alumni Dr. Brian Belleville, Dr. Julian Lettner, Dr. Andrei Homescu and Dr. Mohaned Qunaibit, I thank you for your mentorship, support, and for letting me walk into your offices anytime to seek advice. To my fellow student researchers at the lab Paul, Nikhil, Joseph, Anil, Taemin, Alex, Dokyung, Mitchel, Matthew, Fabian, Min-Yih, and Chinmay, I thank you for inspiring me every day with your hard work, talent, and creativity. I thank the SSLLab community most of all for your camaraderie. To my summer internship mentors Kiarash Ghadianipour (Amazon 2015), Dr. David Molnar (MSR 2016), Dr. Christos Gkantsidis (MSR 2017), Dr. David Tarditi (MSR 2018), Chris Leary (Google 2019) and Robert Hundt (Google 2019), I thank you for the opportunity to work with you and for the countless hours of mentorship which has had a tremendous impact in my growth as a researcher and an engineer. I especially thank my committee members Professor Ardalan Amiri Sani, and Professor Anton Burtsev for their time, consideration, and constructive feedback. This research is based on work partially supported by the Defense Advanced Research Projects Agency (DARPA) under contracts FA8750-16-C-0260, FA8750-15-C-0124 and FA8750-15- C-0085, the National Science Foundation under awards CNS-1513783, CNS-1657711, and CNS-1619211, the United States Office of Naval Research (ONR) under contract N00014- 17-1-2782, the Intel Corporation, as well as gifts from Oracle Corporation. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the Defense Advanced Research Projects Agency (DARPA) or its Contracting Agents, the Office of Naval Research (ONR) or its Contracting Agents, the National Science Foundation (NSF), or any other agency of the U.S. Government. I would like to thank my friends Shreekanth, Vinay, Sudeeptha, Chloe, Holly, and my family for their love and support without which this journey would not have been possible. vii VITA Prabhu karthikeyan Rajasekaran EDUCATION Doctor of Philosophy in Computer Science 2020 University of California, Irvine Irvine, California Master of Science in Computer Science 2016 University of California, Irvine Irvine, California EXPERIENCE Graduate Research Assistant 2015{2020 University of California, Irvine Irvine, California Software Engineering Intern Summer 2019 Google LLC Sunnyvale, California Research Intern Summer 2018 Microsoft Corporation Redmond, Washington Research Intern Summer 2017 Microsoft Corporation Cambridge, UK Research Intern Summer 2016 Microsoft Corporation Redmond, Washington Software Engineering Intern Summer 2015 Amazon.com, Inc.