Mitigating Botnet-Based Ddos Attacks Against Web Servers

MITIGATING BOTNET-BASED DDOS ATTACKS AGAINST WEB SERVERS by Peter L. Djalaliev B.Sc., Washington and Lee University, 2005 M.Sc., University of Pittsburgh, 2009 Submitted to the Graduate Faculty of the Dietrich School of Arts and Sciences in partial fulfillment of the requirements for the degree of Doctor of Philosophy University of Pittsburgh 2013 UNIVERSITY OF PITTSBURGH DIETRICH SCHOOL OF ARTS AND SCIENCES, DEPARTMENT OF COMPUTER SCIENCE This dissertation was presented by Peter L. Djalaliev It was defended on July 15, 2013 and approved by Dr. Adam Lee, Assistant Professor, Department of Computer Science Dr. Youtao Zhang, Associate Professor, Department of Computer Science Dr. Daniel Mosse, Professor, Department of Computer Science Dr. Prashant Krishnamurthy, Associate Professor, School of Information Sciences Dissertation Director: Dr. Adam Lee, Assistant Professor, Department of Computer Science ii MITIGATING BOTNET-BASED DDOS ATTACKS AGAINST WEB SERVERS Peter L. Djalaliev, PhD University of Pittsburgh, 2013 Distributed denial-of-service (DDoS) attacks have become wide-spread on the Internet. They continuously target retail merchants, financial companies and government institutions, dis- rupting the availability of their online resources and causing millions of dollars of financial losses. Software vulnerabilities and proliferation of malware have helped create a class of application-level DDoS attacks using networks of compromised hosts (botnets). In a botnet- based DDoS attack, an attacker orders large numbers of bots to send seemingly regular HTTP and HTTPS requests to a web server, so as to deplete the server's CPU, disk, or memory capacity. Researchers have proposed client authentication mechanisms, such as CAPTCHA puzzles, to distinguish bot traffic from legitimate client activity and discard bot-originated packets. However, CAPTCHA authentication is vulnerable to denial-of-service and artificial intelligence attacks. This dissertation proposes that clients instead use hardware tokens to authenticate in a federated authentication environment. The federated authentication solu- tion must resist both man-in-the-middle and denial-of-service attacks. The proposed system architecture uses the Kerberos protocol to satisfy both requirements. This work proposes novel extensions to Kerberos to make it more suitable for generic web authentication. A server could verify client credentials and blacklist repeated offenders. Traffic from blacklisted clients, however, still traverses the server's network stack and consumes server resources. This work proposes Sentinel, a dedicated front-end network device that intercepts server-bound traffic, verifies authentication credentials and filters blacklisted traffic before it iii reaches the server. Using a front-end device also allows transparently deploying hardware ac- celeration using network co-processors. Network co-processors can discard blacklisted traffic at the hardware level before it wastes front-end host resources. We implement the proposed system architecture by integrating existing software appli- cations and libraries. We validate the system implementation by evaluating its performance under DDoS attacks consisting of floods of HTTP and HTTPS requests. iv TABLE OF CONTENTS PREFACE ......................................... xiv 1.0 INTRODUCTION .................................1 1.1 DISTRIBUTED DENIAL-OF-SERVICE ATTACKS.............2 1.2 THESIS STATEMENT AND CONTRIBUTIONS...............5 1.3 ROADMAP.................................... 11 2.0 SYSTEM MODEL AND BACKGROUND .................. 12 2.1 SYSTEM MODEL................................ 12 2.2 HYPERTEXT TRANSFER PROTOCOL................... 14 2.3 TRANSPORT LAYER SECURITY...................... 14 2.4 FEDERATED AUTHENTICATION...................... 16 2.4.1 OPENID................................. 18 2.4.2 PASSPORT................................ 19 2.4.3 SECURITY ASSERTION MARKUP LANGUAGE.......... 20 2.4.4 IDENTITY METASYSTEM...................... 21 2.4.5 KERBEROS............................... 22 2.5 CONCLUSION.................................. 24 3.0 SENTINEL ..................................... 25 3.1 MODES OF OPERATION........................... 26 3.2 EXTENSIBLE IN-NETWORK CLIENT AUTHENTICATION....... 26 3.3 COMMITTING RESOURCES TO UNAUTHENTICATED CLIENTS... 28 3.4 COUNTING UNAUTHENTICATED REQUESTS AND BLACKLISTING. 30 3.4.1 BLOOM FILTER FALSE POSITIVES................. 31 v 3.4.2 BLACKLISTING............................. 32 3.5 PACKET FILTERING, SCRUBBING, AND DEEP PACKET INSPECTION 33 3.6 IMPLEMENTATION.............................. 34 3.6.1 NETRONOME NFE-I8000....................... 34 3.6.1.1 SOFTWARE APIs....................... 35 3.6.2 CAPTCHA AUTHENTICATION MODULE............. 36 4.0 HARDWARE TOKEN AUTHENTICATION IN A FEDERATED SET- TING ......................................... 37 4.1 HARDWARE TOKEN AUTHENTICATION................. 39 4.2 KERBEROS FEDERATED AUTHENTICATION.............. 40 4.2.1 INTEGRATING HARDWARE TOKEN AUTHENTICATION INTO KERBEROS............................... 42 4.2.2 OFFLOADED KERBEROS AUTHENTICATION REQUESTS... 43 4.2.3 KERBEROS CLIENT CONFIGURATION.............. 44 4.2.4 KERBEROS CLIENT ANONYMITY................. 45 4.3 EXTENDED SENTINEL DESIGN....................... 45 4.3.1 AUTHENTICATION OF HTTP REQUESTS............. 46 4.3.2 AUTHENTICATION OF HTTPS REQUESTS............ 47 4.4 CLIENT INTERFACE: IDENTITY METASYSTEM AND INFORMATION CARDS...................................... 48 4.4.1 IDENTITY METASYSTEM...................... 48 4.4.2 INTEGRATING IDENTITY METASYSTEM WITH KERBEROS. 50 5.0 SYSTEM IMPLEMENTATION ........................ 52 5.1 INTEGRATING KERBEROS AND HARDWARE TOKEN AUTHENTICA- TION....................................... 52 5.2 OFFLOADED KERBEROS AUTHENTICATION REQUESTS....... 54 5.3 VERIFYING KERBEROS CREDENTIALS IN HTTP AND HTTPS TRAF- FIC........................................ 55 5.4 IDENTITY METASYSTEM AND INFORMATION CARD INTEGRATION 57 5.4.1 INFORMATION CARD EXTENSIONS................ 57 vi 5.4.2 SECURITY TOKEN REQUEST AND RESPONSE EXTENSIONS. 58 5.5 USER INTERFACE............................... 59 5.5.1 SENTINEL RESPONSE TO UNAUTHENTICATED HTTP REQUESTS 60 5.5.2 IDENTITY SELECTOR QUERY DIALOG WINDOW........ 61 6.0 PERFORMANCE ANALYSIS ......................... 64 6.1 EXPERIMENTAL SETUP........................... 64 6.1.1 DDOS ATTACK GENERATION WITH BONESI.......... 66 6.1.2 WEB SERVER PAGES......................... 67 6.2 PERFORMANCE EXPERIMENT DESIGN.................. 68 6.3 HTTP FLOOD ATTACKS AGAINST WEB SERVERS........... 72 6.4 HTTP FLOOD ATTACKS AND CAPTCHA AUTHENTICATION WITH DYNAMICALLY GENERATED PUZZLES.................. 74 6.5 HTTP FLOOD ATTACKS AND CAPTCHA AUTHENTICATION WITH PRE-GENERATED PUZZLES......................... 76 6.6 HTTP FLOOD ATTACKS AND HARDWARE TOKEN AUTHENTICATION 78 6.7 HTTP FLOOD ATTACKS AND BLACKLISTING.............. 80 6.8 HTTPS FLOOD ATTACKS AGAINST WEB SERVERS.......... 84 6.9 HTTPS FLOOD AND HARDWARE TOKEN AUTHENTICATION.... 86 7.0 SECURITY ANALYSIS ............................. 88 7.1 NETWORK MODEL.............................. 88 7.1.1 USAGE SCENARIO........................... 90 7.1.2 TRUST DOMAINS AND DEPENDENCIES............. 91 7.2 SYSTEM ASSETS AND ACCESS POINTS.................. 93 7.3 THREAT PROFILE............................... 93 7.3.1 REPLAYING STOLEN KERBEROS PROXY SERVICE TICKETS 93 7.3.2 KERBEROS PASSWORD AND RSA SECURID TOKEN CODE THEFT 95 7.3.3 RSA SECURID TOKEN AND TOKEN SEED THEFT....... 95 7.3.4 AVAILABILITY OF RSA AUTHENTICATION MANAGER.... 96 7.3.5 AVAILABILITY OF KERBEROS KDCS............... 97 7.3.6 DEPLETING SENTINEL'S TCAM MEMORY CAPACITY..... 97 vii 7.4 CONCLUSION.................................. 98 8.0 RELATED WORK AND ALTERNATIVE APPROACHES ....... 99 8.1 CAPTCHAs................................... 99 8.1.1 KILL-BOTS............................... 102 8.2 TRUSTED COMPUTING............................ 103 8.3 SECURE OVERLAYS.............................. 105 8.4 ROAMING HONEYPOTS........................... 107 8.5 CLIENT PUZZLES............................... 107 8.6 COMMERCIAL APPLICATION LEVEL DDOS SOLUTIONS....... 109 8.7 ALTERNATIVE APPROACHES........................ 110 8.8 CONCLUSION.................................. 112 9.0 CONCLUSION ................................... 114 9.1 THESIS STATEMENT AND CONTRIBUTIONS............... 114 9.2 FUTURE WORK................................ 116 9.2.1 LEGITIMATE AUTOMATED PROCESS ACCESS TO WEB SERVERS UNDER DDOS ATTACKS....................... 116 9.2.2 ALTERNATIVE FORMS OF HUMAN USER VERIFICATION... 117 9.2.3 KERBEROS CROSS-REALM ROUTING PROTOCOLS FOR AU- THENTICATION PATH DISCOVERY................ 117 BIBLIOGRAPHY .................................... 118 viii LIST OF TABLES 1 Variable and fixed parameters for performance experiments.......... 71 2 CAPTCHA image format comparison...................... 75 ix LIST OF FIGURES 1 Botnet-based DDoS attacks............................3 2 Overview of system architecture.........................6 3 System Model................................... 13 4 Federated identities and authentication..................... 17 5 OpenID 2.0 message exchange.......................... 18 6 Passport message exchange............................ 19 7 SAML message exchange............................. 20 8 Identity Metasystem message exchange..................... 21 9 Kerberos message exchange............................ 23 10 Overview of Sentinel................................ 25 11 SYN cookies and asynchronous client authentication.............

Load more