Denial of Service in Web Domains: Building Defenses against Next-Generation Attack Behavior DUSAN STEVANOVIC A DISSERTATION SUBMITTED TO THE FACULTY OF GRADUATE STUDIES IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY GRADUATE PROGRAM IN COMPUTER SCIENCE AND ENGINEERING YORK UNIVERSITY TORONTO, CANADA MAY 2016 ©DUSAN STEVANOVIC, 2016 Abstract The existing state-of-the-art in the field of application layer Distributed Denial of Service (DDoS) protection is generally designed, and thus effective, only for static web domains. To the best of our knowledge, our work is the first that studies the problem of application layer DDoS defense in web domains of dynamic content and organization, and for next-generation bot behaviour. In the first part of this thesis, we focus on the following research tasks: 1) we identify the main weaknesses of the existing application-layer anti-DDoS solutions as proposed in research literature and in the industry, 2) we obtain a comprehensive picture of the current-day as well as the next-generation application-layer attack behaviour and 3) we propose novel techniques, based on a multidisciplinary approach that combines offline machine learning algorithms and statistical analysis, for detection of suspicious web visitors in static web domains. Then, in the second part of the thesis, we propose and evaluate a novel anti-DDoS system that detects a broad range of application-layer DDoS attacks, both in static and dynamic web domains, through the use of advanced techniques of data mining. The key advantage of our system relative to other systems that resort to the use of challenge-response tests (such as CAPTCHAs) in combating malicious bots is that our system minimizes the number of these tests that are presented to valid human visitors while succeeding in preventing most malicious attackers from accessing the web site. The results of the experimental evaluation of the proposed system demonstrate effective detection of current and future variants of application layer DDoS attacks. ii Acknowledgements First and foremost, I would like to thank Professor Natalija Vlajic, my research supervisor, for her endless guidance, encouragement and patience over the past five and half years. Professor Vlajic was the main reason for the successful completion of my thesis. She has invested immeasurable hours in helping me co- author 12 research papers, a book chapter and 4 journal articles. It has been a great privilege and an honour to work with her. I am grateful to Professor Matthew Kyan, Professor Suprakash Datta, Ashraf Matrawy, Regina Lee, and Professor Uyen Trang Nguyen for agreeing to serve on my examination committee and for providing very useful feedback on my thesis. I especially would like to thank Professor Suprakash Datta, Professor Bil Tzerpos and Professor Uyen Trang Nguyen for serving on my supervisory committee and for their valuable input during my studies. I thank the exceptional members of the Signal Processing and Communications Laboratory, who have made my experience in graduate school at York University most enjoyable. Lastly, I would like to express my deepest gratitude to my dearest mother Rada, father Mirce, brother Nikola, grandma Mira and uncle Rade for their continuous love and encouragement. I thank my wife Natalie, for her everlasting love, understanding and support. This work is dedicated to my son Milan. iii Table of Contents Abstract ......................................................................................................................................................... ii Acknowledgements ...................................................................................................................................... iii Table of Contents......................................................................................................................................... iv List of Acronyms ......................................................................................................................................... vii List of Tables .............................................................................................................................................. viii List of Figures ............................................................................................................................................ xiv Chapter 1 Introduction ............................................................................................................................ 1 1.1 Overview of the Problem and Motivation ........................................................................................ 1 1.2 Thesis Contributions ......................................................................................................................... 3 1.3 Outline of the Thesis ........................................................................................................................ 5 Chapter 2 Denial of Service - Motivation, Methodology and Defenses ................................................. 7 2.1 Denial of Service: Motivations, Impacts, Execution Mechanisms ................................................... 7 2.2 Distributed Denial of Service ......................................................................................................... 11 2.3 DDoS Malware ............................................................................................................................... 16 2.4 DDoS Defensive Strategies ............................................................................................................ 16 Chapter 3 Application Layer DDoS Attacks and Related Defenses ..................................................... 20 3.1 Overview of Various Types of Application Layer Attacks ............................................................ 21 3.2 Overview of HTTP-based DDoS Attacks ...................................................................................... 21 iv 3.3 State-of-the-Art and Limitations of HTTP-based DDoS Toolkits ................................................. 33 3.4 Overview of the State-of-the-Art in HTTP-based DDoS Defense ................................................. 35 3.5 Summary......................................................................................................................................... 40 Chapter 4 Detection of Web Crawlers and Suspicious Web Visitors in Static Web Domains ............. 42 4.1 Review of Existing Work on the Topic of Web Crawler Detection ............................................... 43 4.2 Detecting Differences in Browsing Behavior of Web Visitors with Supervised Data Mining Classifiers ....................................................................................................................................... 49 4.3 Summary......................................................................................................................................... 91 Chapter 5 Detecting Differences in Browsing Behavior of Web Visitors with Unsupervised Clustering Algorithms 93 5.1 Study on Unsupervised Clustering of Web Visitors Sessions ........................................................ 94 5.2 Detecting Differences in Browsing Behavior of Web Visitors with Non-Parametric Correlation Analysis ........................................................................................................................................ 134 5.3 Summary....................................................................................................................................... 152 Chapter 6 Next-Generation System for HTTP-based DDoS Defence ................................................ 153 6.1 System Overview .......................................................................................................................... 153 6.2 Implementation of the 3 Stages of Our System ............................................................................ 164 6.3 Evaluation of the System .............................................................................................................. 172 6.4 Summary....................................................................................................................................... 192 Chapter 7 Conclusions and Future Work ............................................................................................ 194 Bibliography...............................................................................................................................................197 v Appendix A – DDoS Malware ................................................................................................................. 210 A.1 Scanning Strategies in DDoS-executing Malware ....................................................................... 210 A.2 Recruiting Strategies in DDoS-executing Malware ..................................................................... 212 A.3 Agent Control Mechanisms in DDoS-executing Malware .......................................................... 212 Appendix B – Recall, Precision and F1 ..................................................................................................... 215 Appendix C – Nonparametric Statistical Tests ......................................................................................... 217 C.1 Mann-Whitney Rank Sum Test (also known as Wilcoxon-Mann-Whitney Rank Sum Test) ......... 217 C.2 Kolmogorov-Smirnov Test for 2 Independent