Practical Control-Flow Integrity by Ben Niu Presented to the Graduate and Research Committee of Lehigh University in Candidacy for the Degree of Doctor of Philosophy in Computer Science Lehigh University January, 2016 Copyright © 2015 Ben Niu. All rights reserved. ii Approved and recommended for acceptance as a dissertation in partial fulfillment of the re- quirements for the degree of Doctor of Philosophy. Ben Niu Practical Control-Flow Integrity Date Gang Tan, Dissertation Director Accepted Date Committee Members Gang Tan (Chair) Mooi-Choo Chuah Michael Spear Stephen McCamant iii ACKNOWLEDGEMENTS I am very grateful to my great advisor Prof. Gang Tan. I had no idea how to do research when I came to Lehigh, and he taught me what to think of, how to think, how to present my thoughts, and how to write them down. His door was always open for discussion, he replied emails during weekends, and he provided detailed guidance on any technical problem. I was constantly surprised at his breadth of knowledge and depth of understanding, and I enjoyed working with him. I also thank my thesis committee members, Prof. Mooi-Choo Chuah, Prof. Michael Spear, and Prof. Stephen McCamant for their courses, research works and insightful comments on my research. Michael’s parallel programming course introduced me lock-free algorithms and his operating system course consolidated my system programming skills. Stephen’s research on SFI provided me a foundation in my research, without which I would be still looking for decent solutions. I appreciate my colleagues at MSRC during my internship at Microsoft, especially Matt & Ken, Suha, Vishal, Swamy, Joe, Axel and Marcus. Discussion with them about the design of Control- Flow Guard helps me better understand practicality of CFI. I owe a big debt of gratitude to my apprehensive wife Yinzi, without whose company I could never finish this dissertation. I also thank her for bringing our daughter Jill to the world, who brought our family hope and joy, and I could not imagine how my life would be without her. Moreover, I am very thankful for the invaluable help and support my parents and parents-in-law provided during this adventure. Last but not least, I thank my friends Zheng Shi, Shen Liu, Qi Li, Junqi Chen, Zhongyuan Wei, Zhongliang Liang, Mengtao Sun, Dongrui Zeng, Zi Wang, Bin Zeng and Siliang Li for their generous help in my daily life. iv To my family. v Contents List of Tables x List of Figures xii Abstract 1 1 Introduction 2 1.1 Control-Flow Hijacking .................................. 2 1.2 Deployed Defenses ..................................... 4 1.3 Control-Flow Integrity ................................... 6 1.3.1 Threat Model .................................... 7 1.3.2 The Classic Implementation of CFI ........................ 8 1.3.3 Granularity of CFI ................................. 9 1.4 Practical Issues of Previous CFI .............................. 10 1.5 Challenges to CFI Practicality ............................... 12 1.6 Thesis Statement ...................................... 14 1.7 Contributions ........................................ 14 1.8 This Dissertation versus Previous Publications ..................... 14 1.9 Outline ............................................ 15 2 Modular Control-Flow Integrity 16 2.1 Overview ........................................... 16 vi 2.2 Fine-grained CFG Generation ............................... 17 2.2.1 Source-Level Semantics-based CFG Generation ................ 17 2.2.2 CFG Soundness ................................... 24 2.2.3 CFG Precision Loss ................................. 28 2.3 Modularity and Efficiency ................................. 30 2.3.1 Design of IDs and ID Tables ............................ 32 2.3.2 Memory Layout of MCFI and Protection of ID Tables ............. 34 2.3.3 CFG Check and Update Transactions ...................... 35 2.4 Interoperability ....................................... 43 2.5 Implementation ....................................... 46 2.6 Evaluation .......................................... 46 2.6.1 CFG Statistics .................................... 47 2.6.2 Performance Evaluation .............................. 52 2.7 Future Work ......................................... 54 2.8 Summary ........................................... 56 3 Per-Input Control-Flow Integrity 58 3.1 Overview ........................................... 58 3.1.1 Motivation for per-input CFGs .......................... 60 3.1.2 From edge addition to address activation .................... 63 3.2 System Design ........................................ 64 3.2.1 Secure code patching ................................ 64 3.2.2 Address activation ................................. 66 3.2.3 Compatibility issues ................................ 73 3.3 Implementation ....................................... 75 3.4 Evaluation .......................................... 75 3.4.1 ECFG Statistics ................................... 75 3.4.2 Performance Evaluation .............................. 78 3.5 Future Work ......................................... 79 vii 3.6 Summary ........................................... 81 4 RockJIT 82 4.1 Overview ........................................... 82 4.2 System Design ........................................ 83 4.2.1 Common JIT Architecture ............................. 83 4.2.2 RockJIT Architecture ................................ 85 4.2.3 RockJIT CFG Generation .............................. 85 4.3 JITted Code Manipulation ................................. 87 4.3.1 JITted Code Verification .............................. 88 4.3.2 JITted Code Installation, Deletion, and Modification .............. 90 4.4 Modification to a JIT compiler ............................... 91 4.5 Evaluation .......................................... 93 4.5.1 SCFG and ECFG Statistics ............................. 93 4.5.2 Performance Overhead .............................. 94 4.6 Future Work ......................................... 96 4.7 Summary ........................................... 97 5 Security Analysis 98 5.1 Mitigation of Advanced Attack Forms .......................... 99 5.1.1 Just-In-Time Code Reuse .............................. 99 5.1.2 JIT Spraying ..................................... 100 5.1.3 Counterfeit Object-Oriented Programming ................... 101 5.1.4 Control-Flow Bending ............................... 101 5.1.5 Control Jujutsu ................................... 102 5.1.6 Sigreturn-Oriented Programming ........................ 103 5.2 Comparison with Deployed Defenses .......................... 103 5.2.1 Stack Cookie ..................................... 103 5.2.2 ASLR ......................................... 104 5.3 Limitations of CFI and Future Research ......................... 105 viii 6 Related Work 108 6.1 Control-Flow Integrity ................................... 108 6.2 Software-based Fault Isolation ............................... 110 6.3 JIT Compiler Hardening .................................. 110 6.4 Software Transactional Memory .............................. 111 7 Conclusions 113 Bibliography 124 Vita 125 ix List of Tables 2.1 Condition violations in SPECCPU2006 C benchmarks. ................. 26 2.2 Numbers of cases for the two kinds of violations. .................... 27 2.3 Condition violations in SPECCPU2006 C++ benchmarks. ............... 28 2.4 CFG statistics for SPECCPU2006 C/C++ benchmarks. ................. 48 2.5 Edge distribution for SPECCPU2006 C/C++ benchmarks. ............... 48 2.6 CFG statistics for SPECCPU2006 C/C++ benchmarks without tail-call elimination. 49 2.7 Equivalence class loss of returns. ............................. 50 2.8 Equivalence class loss due to legalized C++ upcalls. .................. 50 2.9 Equivalence class loss due to C++ method pointers. .................. 51 2.10 CFG generation time decomposition. ........................... 52 3.1 ECFG statistics of SPECCPU2006 C/C++ benchmarks. ................ 76 3.2 ECFG statistics of Nginx .................................. 79 4.1 Equivalence classes for Google V8 JavaScript compiler. ................ 93 4.2 ECFG statistics of the Google V8 JavaScript engine. .................. 94 4.3 Performance overhead contributors to RockJIT-hardened V8. ............. 95 x List of Figures 1.1 A stack buffer overflow bug (CVE-2013-2028) in Nginx 1.4.0. ............. 3 1.2 A proof of concept example of Return-Oriented Programming (ROP) attacks. ... 5 1.3 An example of the ClassicCFI instrumentation for x64 Linux. ............. 9 2.1 A toy C++ example of virtual method call targets. ................... 18 2.2 An example about C++ function pointers. ........................ 19 2.3 Control transfers during C++ table-based exception handling. ............ 21 2.4 Metadata annotation for the assembly code of memcpy. ................. 23 2.5 Bad function pointer type cast example in C. ...................... 24 2.6 Example of CFG precision loss due to MCFI tail-call elimination. .......... 29 2.7 Example of CFG precision loss because of indirect calls. ................ 30 2.8 Example code of CFG precision loss due to C++ method pointers. .......... 31 2.9 MCFI’s ID Encoding for x64. ................................ 32 2.10 Memory layout of MCFI. .................................. 36 2.11 Pseudocode for implementing update transactions. .................. 39 2.12 Implementation of check transactions for x64 return instructions. .......... 40 2.13 A compatibility condition violation introduced by the interoperability support. 44 2.14