Control-Flow Security by William Patrick Arthur A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy (Computer Science and Engineering) in The University of Michigan 2016 Doctoral Committee: Professor Todd M. Austin, Chair Associate Professor Prabal Dutta Professor Scott Mahlke Assistant Professor Jason Mars Professor Dennis M. Sylvester c William Patrick Arthur All Rights Reserved 2016 For Allison and Marshall. ii Acknowledgments I would like to first thank my advisor Professor Todd Austin. This work would not be possible without his support and efforts. Serving as a role model, he was always personally motivating and made the work not only possible but a joy. I also thank my committee members for their insight and encouragement; Professors Scott Mahlke, Dennis Sylvester, Prabal Dutta, and Jason Mars. They have proven to be mentors who exemplify what is most admirable about academia and research. Additionally, I thank Reetuparna Das, a person of great importance to this work. Thank you for challenging me to strive for excellence. This work was not completed without the contribution and assistance of others, most notably from my collaborators. My undergraduate collaborators Ben Mehne and Sahil Madeka have proven invaluable. Additionally, my peers in graduate studies have provided invaluable insight and I am ever thankful for their time listening, challenging, and debating my ideas and this work. This includes Salessawi Ferede Yitbarek, Joseph Greathouse, Jason Clemons, Ricardo Rodriguez, Zelalem Birhanu Aweke, and too many others to mention in the space here. Most importantly, I thank my friend, colleague, and collaborator Biruk Wendimagegn Mammo. Lastly, I thank my family for their patience and support. They have provided the encouragement, affirmation, and ultimately the purpose in completing this work. iii Table of Contents Dedication ....................................... ii Acknowledgments ................................... iii List of Tables ...................................... viii List of Figures ..................................... ix Abstract ......................................... xi Chapter 1 Introduction ...............................1 1.1 The Software Attack Surface . .2 1.1.1 The Cycle of Exploitation of the Software Attack Surface . .2 1.1.2 The Evolution and Commercialization of Malware . .5 1.1.3 Breaking the Cycle of Exploitation . .5 1.1.4 Control-Flow Attacks . .6 1.2 Contemporary Control-Flow Attacks Rely on Mixing of Control with Data .7 1.3 Hardware-Software Co-Design Can Greatly Enhance Control-Data Isolation8 1.4 Latent Software Defects are Control-Flow Path Dependent . .9 1.5 Contributions of this Work . 10 Chapter 2 Control-Data Isolation .......................... 15 2.1 Introduction . 16 2.1.1 Control-Flow Attacks . 16 2.1.2 Control-Data Isolation . 17 2.1.3 Contributions of this Chapter . 18 2.2 Protecting Control Flow with Control-Data Isolation . 19 2.2.1 Threat and Trust Model . 19 2.2.2 CDI Threat Protection . 20 2.2.3 Achieving Higher Levels of Protection by Isolating Control and Data 21 2.3 Control-Data Isolation via Elimination of Indirect Control Flow . 23 2.3.1 CFG Discovery . 24 2.3.2 Indirection Elimination . 26 2.3.3 CDI Performance Optimizations . 27 iv 2.3.4 Detecting Attacks in CDI Protected Programs . 28 2.4 CDI Implementation for Shared Libraries . 29 2.4.1 Dynamic Nature of Shared Libraries . 29 2.4.2 Enforcing CDI for Shared Libraries . 30 2.5 CDI Compliant Code for Dynamic Compilation . 32 2.5.1 Dynamically Generated Code . 32 2.5.2 Threat and Trust Model for Dynamically Generated Code . 33 2.5.3 Properties of CDI Compliant Code . 33 2.5.4 CDI-compliant Code Attestation . 34 2.6 PitBull: Compiler-Based Control-Data Isolation . 35 2.7 Experimental Evaluation . 37 2.7.1 Benchmark Applications . 38 2.7.2 Performance Evaluation . 38 2.7.3 Impact of Optimization . 38 2.8 Related Work . 40 2.8.1 Software Mechanisms . 41 2.8.2 Hardware Solutions . 43 2.9 Chapter Conclusions . 43 Chapter 3 Locking Down Insecure Indirection with Hardware-Based Control- Data Isolation ................................... 44 3.1 Introduction . 45 3.1.1 Control-Flow Attacks . 45 3.1.2 Control-Data Isolation . 46 3.1.3 Advancing CDI with Architecture Support . 47 3.1.4 Contributions of this Chapter . 48 3.2 Protecting Control Flow with Control-Data Isolation . 49 3.2.1 Threat and Trust Model . 49 3.2.2 Control-Data Isolation Mechanism . 49 3.2.3 Runtime Overheads Executing CDI-Compliant Code . 51 3.3 Architectural Acceleration of CDI . 52 3.3.1 Caching Indirect Control Transitions . 52 3.3.2 Edge Cache Architecture . 53 3.3.3 Minimizing Edge Cache Storage . 55 3.3.4 Indirect Branch Prediction with the Edge Cache . 56 3.4 Methodology . 59 3.4.1 Implementation Platform . 59 3.4.2 Benchmark Applications . 59 3.4.3 Hardware/Software Co-Design . 59 3.5 Experimental Evaluation . 60 3.5.1 Runtime Performance Analysis . 60 3.5.2 Indirect Branch Prediction Analysis . 62 3.5.3 Security Analysis . 63 3.5.4 Area and Power Analysis . 64 v 3.6 Extending Control-Data Isolation to Advance Path-Based Control-Flow Security . 65 3.6.1 Path-Based Attacks . 65 3.6.2 Impossible Paths . 66 3.6.3 Advancing Control-Data Isolation with the Non-Speculative Return Address Stack . 66 3.6.4 Contributions of the Non-Speculative Return Address Stack . 67 3.6.5 The Return Address Stack . 68 3.6.6 The Non-Speculative Return Address Stack . 68 3.6.7 Eliminating Control Path Attacks with the Non-Speculative Return Address Stack . 69 3.6.8 Detecting Stack Unwinding with the Stack Pointer Address Stack . 70 3.6.9 Dynamic Enforcement of the Valid Control-Flow Graph Edge . 70 3.6.10 Methodology of the Non-Speculative Return Address Stack . 71 3.6.11 Experimental Evaluation of the Non-Speculative Return Address Stack . 72 3.7 Related Work . 72 3.7.1 Software Mechanisms . 72 3.7.2 Hardware Solutions . 75 3.7.3 Control-Data Isolation in Contrast to CFI . 75 3.7.4 Indirect Branch Prediction . 76 3.8 Chapter Conclusions . 76 Chapter 4 Scalable Profiling for Likely Security Bug Sites ............ 77 4.1 Introduction . 77 4.1.1 Contributions of this Chapter . 79 4.1.2 Dynamic Control Frontier Discovery . 81 4.1.3 Computing the Dynamic Control Frontier . 83 4.1.4 Profiling the Dynamic Control Frontier . 83 4.1.5 Leveraging the Dynamic Control Frontier . 85 4.2 Schnauzer: A Distributed DCF Profiler . 86 4.3 Experimental Evaluation and Results . 89 4.3.1 Benchmark Applications . 89 4.3.2 Experimental Framework . 89 4.3.3 Ground Truth Dynamic Control Frontier . 90 4.3.4 Analysis of DCF Sampling . 91 4.3.5 Schnauzer Profiling Scalability . 92 4.3.6 DCF Correlation with Real Vulnerabilities . 95 4.4 Introduction . 97 4.4.1 Hot Path Analysis . 97 4.4.2 Path Analysis and Distributed Sampling . 98 4.4.3 Complementary Works . 98 4.5 Chapter Conclusions . 100 Chapter 5 Conclusion ................................ 101 vi 5.1 Dissertation Summary . 101 5.2 Future Control-Flow Security Research . 102 5.3 Conclusion . 104 Bibliography ...................................... 105 vii List of Tables Table 1.1 Escalation of Attacks and Defenses . .4 2.1 Control-Data Isolation Performance . 39 2.2 CDI Related Works . 41 3.1 Storage Size of Edge Cache and Components . 61 3.2 Edge Cache Validation Rate . 62 3.3 Predictor Array Sensitivity on Predicting Indirect Branches . 62 3.4 Indirection Misprediction Rate of CDI-Compliant Code . 63 3.5 Average Indirect Target Reduction . 64 3.6 Non-Speculative RAS Performance . 73 3.7 Non-Speculative RAS Security Performance . 74 4.1 Benchmark Applications . 91 4.2 Software Vulnerabilities Sensitized by Dynamic Control Frontier Paths . 96 viii List of Figures Figure 1.1 Heartbleed Bug Vulnerable Code . .4 1.2 Elements of This Dissertation . 14 2.1 CDI Control Flow Protection . 21 2.2 CDI Compilation Flow . 23 2.3 Indirect Instruction Target Set for CFG Construction . 25 2.4 Dynamic Profiling for CDI Optimization . 28 2.5 CDI for Shared Library Control Flow Transfer . 31 2.6 CDI Code Attestation . 35 2.7 PitBull Compilation Flow . 36.