Introspective Intrusion Detection
Total Page:16
File Type:pdf, Size:1020Kb
UNIVERSITY OF CALIFORNIA, IRVINE Introspective Intrusion Detection DISSERTATION submitted in partial satisfaction of the requirements for the degree of DOCTOR OF PHILOSOPHY in Computer Engineering by Byron Hawkins Dissertation Committee: Associate Professor Brian Demsky, Chair Professor Michael Franz Associate Professor Athina Markopoulou 2017 Portions of Chapters 1, 2, 5 c 2017 IEEE Portions of Chapters 1, 2, 3 c 2016 ACM Portions of Chapters 1, 4 c 2015 IEEE All other materials c 2017 Byron Hawkins DEDICATION To the International Justice Mission for their enduring effort to protect powerless individuals from the malicious intrusions of the real world. ii TABLE OF CONTENTS Page LIST OF FIGURES vii LIST OF TABLES ix LIST OF ALGORITHMS x ACKNOWLEDGMENTS xi CURRICULUM VITAE xii ABSTRACT OF THE DISSERTATION xiv 1 Introduction 1 1.1 Timeline of Important RCE Defenses . 8 1.1.1 Bug Fixing . 9 1.1.2 Traditional Intrusion Detection . 10 1.1.3 Control Flow Integrity . 12 1.1.4 Diversification . 14 1.2 Theoretical Advantages and Challenges of IID . 16 1.3 Goals of Introspective Intrusion Detection . 24 1.4 Usage Model . 29 1.4.1 Forensics . 30 1.4.2 Advanced Persistent Threats . 32 1.4.3 Field Debugging . 35 1.4.4 Profiling . 37 1.5 Prototype IID Implementations . 38 1.5.1 BlackBox for x86 COTS Binaries . 40 1.5.2 ZenIDS for PHP Applications . 42 1.6 Contributions . 44 1.7 Organization . 45 2 Paradise: a Hypothetically Pure IID 46 2.1 Platform Snares . 47 2.2 Generic Threat Model . 51 2.3 Deployment . 52 2.3.1 Generating the Trusted Profile . 53 iii 2.3.2 Monitoring . 56 2.4 Maintaining the Trusted Profile . 59 2.4.1 Blacklisting . 59 3 BlackBox 61 3.1 Intel CET . 65 3.2 Threat Model . 67 3.3 Monitoring . 67 3.3.1 Trusted Profile . 68 3.3.2 Trusting Returns . 69 3.3.3 Trusting Self-Instrumentation . 70 3.3.4 Modular Trust . 71 3.3.5 Detecting Evasion . 72 3.4 Dynamically-Generated Code . 74 3.4.1 Permission Model . 75 3.4.2 Standalone Dynamic Routines . 78 3.5 Watchdog Mode . 82 3.5.1 Stack Spy . 82 3.5.2 Sniffer Dog . 85 3.6 Evaluation . 90 3.6.1 Filtering Log Noise . 91 3.6.2 Logging and Blacklisting Exploits . 99 3.6.3 Resource Efficiency . 101 3.6.4 Security Analysis . 105 3.6.5 Verifiability . 115 4 DynamoRIO JIT Optimization 118 4.1 Background and Related Work . 122 4.1.1 DynamoRIO . 122 4.1.2 QEMU . 125 4.1.3 Pin . 126 4.1.4 Valgrind . 126 4.1.5 Specialized Applications of Binary Translation . 127 4.2 Annotation-Based Optimization . 127 4.2.1 New Annotations Identifying Code Changes . 128 4.2.2 VM Area Isolation . 129 4.2.3 Selective Fragment Removal . 129 4.3 Annotation Implementation . 132 4.3.1 Binary Annotation Scheme . 132 4.3.2 Annotation Discussion . 134 4.3.3 Annotations in 64-bit Microsoft Visual Studio . 136 4.4 Inference-Based Optimization . 139 4.4.1 Parallel Mapping on Other Platforms . 142 4.5 Evaluation . 143 iv 5 ZenIDS 148 5.1 Threat Model . 150 5.2 Monitoring . 151 5.2.1 Trusted Profile . 151 5.2.2 Detection . 155 5.2.3 Dynamic vs. Static Analysis . 159 5.3 Supporting Website Evolution . 161 5.3.1 Code Expansion Events . 163 5.3.2 Taint Tracking . 164 5.4 Performance Optimization . 165 5.4.1 Synchronizing Evolution . 167 5.5 Experimental Evaluation . 167 5.5.1 Monitoring a Vulnerable Application . 168 5.5.2 Monitoring Live Applications . 168 5.5.3 Evolution . 173 5.5.4 Resource Efficiency . 175 5.5.5 Verifiability . 177 6 Related Work 178 6.1 Control Flow Integrity . 180 6.1.1 Equivalence Class CFI for x86 . 181 6.1.2 Equivalence Class CFI for Other Platforms . 193 6.1.3 Pointer Protection CFI . 196 6.1.4 RCE Exploits vs. x86 Compiled Binaries . 197 6.1.5 Application Compatibility . 201 6.1.6 Performance . 202 6.1.7 Summarizing IID in the field of CFI . 205 6.2 CFI in the Future Ubiquity of CET . 207 6.3 Traditional Intrusion Detection . 207 6.4 PHP Security . 211 6.4.1 RCE Exploits vs. the PHP Platform . 212 6.5 Software Improvement Techniques . 214 6.5.1 Debugging . 214 6.5.2 Automated Repair . 215 6.6 Usability . 216 6.6.1 Program Comprehension . 217 6.6.2 Symbolic Execution . 218 6.6.3 Machine Learning . 218 7 Conclusions and Future Work 220 7.1 Adoption . 221 Bibliography 224 v Appendices 239 A Index of Implementation Concerns 239 List of Advantages . 239 List of Challenges . 240 List.