UNDERSTANDING AND DETECTING NEWLY EMERGING ATTACK VECTORS IN CYBERCRIMES by Daiping Liu A dissertation defense submitted to the Faculty of the University of Delaware in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Electrical & Computer Engineering 2018 c 2018 Daiping Liu All Rights Reserved UNDERSTANDING AND DETECTING NEWLY EMERGING ATTACK VECTORS IN CYBERCRIMES by Daiping Liu Approved: Kenneth E. Barner, Ph.D. Chair of the Department of Electrical Engineering Approved: Babatunde A. Ogunnaike, Ph.D. Dean of the College of Engineering Approved: Ann L. Ardis, Ph.D. Senior Vice Provost for Graduate and Professional Education I certify that I have read this dissertation defense and that in my opinion it meets the academic and professional standard required by the University as a dissertation defense for the degree of Doctor of Philosophy. Signed: Haining Wang, Ph.D. Professor in charge of dissertation defense I certify that I have read this dissertation defense and that in my opinion it meets the academic and professional standard required by the University as a dissertation defense for the degree of Doctor of Philosophy. Signed: Chase Cotton, Ph.D. Member of dissertation defense committee I certify that I have read this dissertation defense and that in my opinion it meets the academic and professional standard required by the University as a dissertation defense for the degree of Doctor of Philosophy. Signed: Xiaoming Li, Ph.D. Member of dissertation defense committee I certify that I have read this dissertation defense and that in my opinion it meets the academic and professional standard required by the University as a dissertation defense for the degree of Doctor of Philosophy. Signed: Kun Sun, Ph.D. Member of dissertation defense committee ACKNOWLEDGEMENTS This dissertation is written with the support and help from many individuals. I would like to thank all of them. First and foremost, I would like to express my deepest appreciation to my ad- visor, Dr. Haining Wang. Without his guidance in my research, encouragement in my life, and confidence in my abilities, this dissertation would not have been possible. I would also like to thank my dissertation committee, Dr. Chase Cotton, Dr. Xiaoming Li, and Dr. Kun Sun, for serving on my Ph.D committee as well as their insightful comments. My sincere thanks also go to all members of our group past and present, Dr. Shuai Hao, Dr. Haitao Xu, Dr. Jidong Xiao, Dr. Dacuan Liu, Xing Gao, Yubao Zhang, for the stimulating discussions, constructive suggestions, generous assistance, and effective teamwork. Futhurmore, I would like to thank the faculty and staff at the Department of Electrical & Computer Engineering, University of Delaware. Special thanks to Gwen Looby and Amber Spivey for their considerate and effective assistance. Last but not the least, I would like to thank my family. Thanks to my parents, whose unwavering love and support has made me who I am today. Thanks to my wife, Ge Peng, for lighting up my life with so much love and joy. This dissertation was supported in part by the U.S. NSF grant CNS-1618117, as well as ONR grants N00014-13-1-0088 and N00014-17-1-2485. iv TABLE OF CONTENTS LIST OF TABLES :::::::::::::::::::::::::::::::: x LIST OF FIGURES ::::::::::::::::::::::::::::::: xi ABSTRACT ::::::::::::::::::::::::::::::::::: xii Chapter 1 INTRODUCTION :::::::::::::::::::::::::::::: 1 1.1 Problem Statements ::::::::::::::::::::::::::: 1 1.2 Contributions ::::::::::::::::::::::::::::::: 3 1.3 Dissertation Organization :::::::::::::::::::::::: 5 2 RELATED WORK ::::::::::::::::::::::::::::: 6 2.1 Malicious Document Detection :::::::::::::::::::::: 6 2.2 DNS Security ::::::::::::::::::::::::::::::: 8 2.3 Malicious Domain Detection ::::::::::::::::::::::: 9 2.4 Memory Safety in Software :::::::::::::::::::::::: 11 3 DETECTING MALICIOUS JAVASCRIPT IN PDF THROUGH DOCUMENT INSTRUMENTATION ::::::::::::::::: 13 3.1 Introduction :::::::::::::::::::::::::::::::: 13 3.2 System Design ::::::::::::::::::::::::::::::: 16 3.2.1 Architecture :::::::::::::::::::::::::::: 16 3.2.2 Static Features :::::::::::::::::::::::::: 17 3.2.3 Document Instrumentation :::::::::::::::::::: 19 3.2.4 Runtime Features ::::::::::::::::::::::::: 22 3.2.5 Runtime Detection and Confinement :::::::::::::: 25 v 3.2.6 De-instrumentation :::::::::::::::::::::::: 28 3.3 Security Analysis ::::::::::::::::::::::::::::: 28 3.3.1 Threat Model ::::::::::::::::::::::::::: 29 3.3.2 Potential Advanced Attacks and Countermeasures ::::::: 29 3.4 Evaluation ::::::::::::::::::::::::::::::::: 31 3.4.1 Data Collection :::::::::::::::::::::::::: 32 3.4.2 Feature Validation :::::::::::::::::::::::: 32 3.4.3 Detection Accuracy :::::::::::::::::::::::: 36 3.4.4 System Performance ::::::::::::::::::::::: 39 3.5 Conclusion ::::::::::::::::::::::::::::::::: 42 4 TOWARDS AUTOMATED DETECTION OF SHADOWED DOMAINS :::::::::::::::::::::::::::::::::: 43 4.1 Introduction :::::::::::::::::::::::::::::::: 43 4.2 Background :::::::::::::::::::::::::::::::: 46 4.2.1 Basics of Domain Name ::::::::::::::::::::: 46 4.2.2 Domain Shadowing :::::::::::::::::::::::: 47 4.2.3 Real-world Example ::::::::::::::::::::::: 50 4.3 Automatic Detection of Shadowed Domains :::::::::::::: 51 4.3.1 Overview ::::::::::::::::::::::::::::: 51 4.3.2 Dataset :::::::::::::::::::::::::::::: 54 4.3.3 Features of Domain Shadowing ::::::::::::::::: 56 4.3.3.1 Subdomain Usage ::::::::::::::::::: 58 4.3.3.2 Subdomain Hosting :::::::::::::::::: 60 4.3.3.3 Subdomain Activity :::::::::::::::::: 61 4.3.3.4 Subdomain Name :::::::::::::::::::: 63 4.4 Evaluation ::::::::::::::::::::::::::::::::: 63 4.4.1 Training and Testing Classifiers ::::::::::::::::: 64 4.4.2 Feature Analysis ::::::::::::::::::::::::: 66 4.4.3 Generality of Trained Models :::::::::::::::::: 68 4.4.4 Evaluation on Dunknown :::::::::::::::::::::: 69 vi 4.4.5 Evaluation on Dvt ::::::::::::::::::::::::: 69 4.5 Measurement and Discoveries :::::::::::::::::::::: 72 4.5.1 Case Studies :::::::::::::::::::::::::::: 76 4.6 Discussion ::::::::::::::::::::::::::::::::: 77 4.7 Conclusion ::::::::::::::::::::::::::::::::: 78 5 ALL YOUR DNS RECORDS POINT TO US: UNDERSTANDING THE SECURITY THREATS OF DANGLING DNS RECORDS :::::::::::::::::::::: 80 5.1 Introduction :::::::::::::::::::::::::::::::: 80 5.2 DNS Overview :::::::::::::::::::::::::::::: 82 5.3 Dangling DNS Records :::::::::::::::::::::::::: 84 5.3.1 Security Sensitive Dares ::::::::::::::::::::: 85 5.3.2 IP in Cloud :::::::::::::::::::::::::::: 87 5.3.3 Abandoned Third-party Services ::::::::::::::::: 90 5.3.4 Expired Domains ::::::::::::::::::::::::: 91 5.3.5 Summary ::::::::::::::::::::::::::::: 92 5.4 Measurement Methodology :::::::::::::::::::::::: 92 5.4.1 Domain Collection :::::::::::::::::::::::: 92 5.4.2 DNS Data Retrieval ::::::::::::::::::::::: 93 5.4.3 Searching for Dares :::::::::::::::::::::::: 93 5.4.3.1 Checking A Records (Lines 7 and 9) ::::::::: 94 5.4.3.2 Checking Abandoned Services (Line 15) ::::::: 96 5.4.3.3 Checking Expired Domains (Lines 12 and 19) :::: 97 5.4.4 Limitations :::::::::::::::::::::::::::: 97 5.5 Measurement Results ::::::::::::::::::::::::::: 97 5.5.1 Characterization of Dares :::::::::::::::::::: 98 5.5.2 IP in Cloud :::::::::::::::::::::::::::: 100 5.5.3 Abandoned Third-party Services ::::::::::::::::: 105 5.5.4 Expired Domains ::::::::::::::::::::::::: 106 5.5.5 Exploiting Dares ::::::::::::::::::::::::: 107 vii 5.5.6 Ethical Considerations :::::::::::::::::::::: 108 5.6 Threat Analysis :::::::::::::::::::::::::::::: 108 5.6.1 Scamming, Phishing, and More ::::::::::::::::: 108 5.6.2 Active Cookie Stealing :::::::::::::::::::::: 110 5.6.3 Email Fraud :::::::::::::::::::::::::::: 111 5.6.4 Forged SSL Certificate :::::::::::::::::::::: 111 5.7 Mitigations :::::::::::::::::::::::::::::::: 112 5.8 Conclusion ::::::::::::::::::::::::::::::::: 114 6 PRACTICAL AND ROBUST DEFENSE AGAINST USE-AFTER-FREE EXPLOITS VIA CONCURRENT POINTER SWEEPING ::::::::::::::::::::::::::::::::: 116 6.1 Introduction :::::::::::::::::::::::::::::::: 116 6.2 Background and Threat Model :::::::::::::::::::::: 118 6.3 Overview :::::::::::::::::::::::::::::::::: 119 6.3.1 High-Level Approach of pSweeper :::::::::::::::: 119 6.3.2 An Illustration Example ::::::::::::::::::::: 120 6.3.3 Architecture of pSweeper ::::::::::::::::::::: 121 6.4 System Design ::::::::::::::::::::::::::::::: 122 6.4.1 Memory Allocation Status Table :::::::::::::::: 123 6.4.2 Locating Live Pointers :::::::::::::::::::::: 123 6.4.2.1 Pointers on Data Segment ::::::::::::::: 123 6.4.2.2 Pointers on Stack :::::::::::::::::::: 124 6.4.2.3 Pointers on Heap :::::::::::::::::::: 124 6.4.3 Deferred Free ::::::::::::::::::::::::::: 126 6.4.4 Concurrent Pointer Sweeping (CPS) ::::::::::::::: 127 6.4.4.1 CPS Threads :::::::::::::::::::::: 127 6.4.4.2 Preventing Dangling Pointer Propagation :::::: 130 viii 6.4.4.3 More pSweeper Threads :::::::::::::::: 131 6.4.5 Object Origin Tracking (OOT) ::::::::::::::::: 132 6.5 Evaluation ::::::::::::::::::::::::::::::::: 133 6.5.1 Effectiveness of pSweeper :::::::::::::::::::: 134 6.5.2 Performance on SPEC CPU2006 :::::::::::::::: 134 6.5.2.1 Runtime Overhead ::::::::::::::::::: 136 6.5.2.2 Memory Overhead ::::::::::::::::::: 138 6.5.2.3 Comparison to DangSan :::::::::::::::: 139 6.5.3 Scalability on Multi-threaded Applications ::::::::::: 139 6.5.4 Macro Benchmarks :::::::::::::::::::::::: 140 6.5.4.1 Lighttpd ::::::::::::::::::::::::: 140 6.5.4.2 Mozilla Firefox ::::::::::::::::::::: 141 6.6 Discussion & Limitations :::::::::::::::::::::::::