Detection of Botnet Command and Control Traffic in Enterprise Networks
Total Page:16
File Type:pdf, Size:1020Kb
DETECTIONOF BOTNET COMMANDAND CONTROL TRAFFIC IN ENTERPRISE NETWORKS Pieter BURGHOUWT DETECTIONOF BOTNET COMMANDAND CONTROL TRAFFIC IN ENTERPRISE NETWORKS Proefschrift ter verkrijging van de graad van doctor aan de Technische Universiteit Delft, op gezag van de Rector Magnificus prof. ir. K. C. A. M. Luyben, voorzitter van het College voor Promoties, in het openbaar te verdedigen op 2 februari 2015 om 10:00 uur door Pieter BURGHOUWT ingenieur Elektrotechniek geboren te Voorburg, Nederland. Dit proefschrift is goedgekeurd door de promotor: Prof. dr. H. J. Sips Samenstelling promotiecommissie: Rector Magnificus, voorzitter Prof. dr. ir. H.J. Sips, Delft University of Technology, promotor Dr. M.E.M. Spruit, The Hague University of Applied Sciences Prof. dr. K.G. Langendoen, Delft University of Technology Prof. dr. ir. J. van den Berg, Delft University of Technology Prof. dr. ir. H.J. Bos, VU University of Amsterdam Prof. dr. S Etalle, Technical University of Eindhoven Dr. ir. J. Henseler, Amsterdam University of Applied Sciences Prof. dr. ir. D.H.J. Epema, Delft University of Technology Dr. M.E.W. Spruijt, heeft als begeleider in belangrijke mate aan de totstandkoming van het proefschrift bijgedragen. Keywords: botnets, network intrusion detection, command and control traffic, cybersecurity Printed by: Ipskamp Drukkers B.V Copyright © 2015 by Pieter Burghouwt Cover design: Odine Burghouwt ISBN 978-94-6186-414-7 An electronic version of this dissertation is available at http://repository.tudelft.nl/. This thesis is dedicated to my wife Carmen and my children for their love and patience. Pieter Burghouwt ACKNOWLEDGMENTS This thesis is the result of hard work over the last years. To combine PhD research with my work as a lecturer, a husband and a father was often challenging. Nevertheless, with great satisfaction I can look back at a phase of my life in which I was a student again. I enjoyed all aspects of it: from satisfying my curiosity in the research, to presenting my work at conferences in other countries. This work would not have been possible without the support of many people. I express my grateful thanks to all of them. First my thanks go to my supervisors Henk Sips and Marcel Spruit for their continu- ous and high quality support during my PhD studies. By their coaching and constructive feedback they enabled me to conduct the research and develop my research skills. I thank the PhD committee for their review and their provision of high quality feed- back that improved my thesis. I also thank my employer The Hague University of Applied Sciences who offered me the unique opportunity to conduct my PhD research. My special thanks go to: Gert de Ruiter, Ineke van der Meulen, and other members of the board who enabled and pa- tiently supported my study, to Jan Dirk Schagen who encouraged me to start the PhD research, and to all my colleagues who had to work harder because of my limited avail- ability for teaching. Finally but most importantly I thank my wife Carmen and my children for their pa- tience, moral support, and their faith in me. My mental and sometimes physical ab- sence must have been difficult for them but they never complained. I could not have completed my research without the support of all these wonderful people! Pieter Burghouwt Delft, January 2015 vii CONTENTS 1 Introduction1 1.1 Research Background............................1 1.1.1 Schematic Overview of a Botnet...................1 1.1.2 Defining Botnets and Bots......................2 1.1.3 The Evolution of Botnets.......................3 1.1.4 Botnet Countermeasures......................5 1.2 The Enterprise Network...........................6 1.3 Problem Statement.............................7 1.4 Research Objectives and subsequent research activities..........8 1.4.1 Contributions............................9 1.5 Thesis Outline............................... 10 2 Countering Botnets by the Detection of C&C Traffic in Enterprise Networks 13 2.1 Network Intrusion Detection........................ 14 2.1.1 Feature Extraction.......................... 14 2.1.2 Knowledge.............................. 15 2.2 Evaluation of Intrusion detection methods................. 19 2.2.1 Detection performance....................... 19 2.2.2 Other Evaluation Criteria...................... 21 2.3 Classification of Botnet Countermeasures................. 24 2.3.1 Existing Classifications........................ 24 2.3.2 Taxonomy versus Ontology-based Faceted Classification...... 26 2.3.3 A Faceted Ontology-based Classification of Botnet Countermea- sures................................. 27 2.3.4 Motivation of the Facets "counters" and "is-implementated-in" ... 29 2.3.5 Motivation of the Facet "intervenes by" ................ 30 2.3.6 Refinement of Reactive Measures.................. 32 2.3.7 Evaluation of the Ontology-based Classification........... 33 2.4 An Overview of Existing Network-based C&C Detection.......... 35 2.4.1 Classification of Existing Misused-based Detection......... 37 2.4.2 Examples of Anomaly-based Detection............... 38 2.5 Network-based C&C Detection in an Enterprise Network.......... 40 2.5.1 Enterprise-specific Characteristics.................. 40 2.5.2 Selection of Classes in the Detection Ontology............ 41 2.5.3 New Detection Approaches..................... 42 ix x CONTENTS 3 Detection of Botnet Communication by Monitoring User Activity 45 3.1 Introduction................................ 46 3.2 Related Work................................ 47 3.3 Detection Principle............................. 48 3.3.1 Detection of Botnet Traffic to Twitter.com.............. 49 3.3.2 Empirical Estimation of Optimal Time Windows........... 51 3.3.3 Theoretical Performance of the Detector............... 51 3.3.4 Experimental Evaluation of the Detection Algorithm........ 54 3.4 Special Cases of Twitter Traffic....................... 55 3.4.1 Automatic Legal Traffic....................... 55 3.4.2 Evasion by User Synchronized Botnet Traffic............ 56 3.5 Conclusions................................. 57 4 Detection of C&C Traffic by Causal Analysis of Traffic Flows 59 4.1 Introduction................................ 60 4.2 Related Work................................ 62 4.3 Causal Analysis of Flows and Anomaly Detection.............. 63 4.3.1 The Direct Cause of a Flow...................... 63 4.3.2 Optimal Selection of the Direct Cause................ 64 4.3.3 Detection Performance....................... 65 4.4 CITRIC: Practical Implementation of TFC graph Construction and Detec- tion..................................... 66 4.4.1 Implementation Issues and Solutions................ 67 4.5 Experimental Evaluation.......................... 67 4.5.1 Empirical Determination of the Optimal Windows Sizes....... 68 4.5.2 TFC Detection of Real C&C Traffic.................. 71 4.5.3 Visualization of the TFC Graphs................... 71 4.6 Evasion and Related Improvements of TFC Detection............ 73 4.6.1 Solutions against Piggybacking................... 73 4.7 Conclusions................................. 74 5 Detection of C&C Traffic by Identification of Untrusted Destinations 75 5.1 Introduction................................ 76 5.2 UDI Detection Approach.......................... 76 5.2.1 Logical Destination Identifiers.................... 78 5.2.2 Forward Reference Extraction.................... 79 5.2.3 The UDI Detection Algorithm.................... 79 5.2.4 Detection Errors........................... 80 5.3 Detector Implementation.......................... 81 5.3.1 Partial ldi-matching......................... 82 5.3.2 Name-Based Criteria......................... 82 5.4 Experimental Evaluation.......................... 82 5.4.1 Controlled Environment....................... 83 5.4.2 Evaluation of False Positives and Stage Contribution......... 84 5.4.3 Evaluation of True Positives..................... 85 CONTENTS xi 5.5 Evasion of UDI Detection and Solutions.................. 86 5.6 Related Work................................ 88 5.6.1 Work Related with Flow Analysis in Consecutive Stages....... 88 5.6.2 Work Related with Logical Destination Referencing......... 88 5.6.3 Work Related with Human Input Evaluation............. 89 5.7 Conclusions................................. 89 6 Detection of Botnet Command and Control Channels by Anomalous Degrees of DNS Domains 91 6.1 Introduction................................ 92 6.2 Related Work................................ 92 6.3 Detection by Domain Degree........................ 93 6.3.1 Distribution of the Domain Degree................. 93 6.3.2 Detection Process.......................... 95 6.3.3 Detection Evasion.......................... 96 6.4 Experimental Evaluation.......................... 98 6.4.1 Measurement and Comparison of the Degree Distribution..... 99 6.4.2 The Effect of Filtering Popular Domains............... 100 6.5 Conclusions................................. 103 7 Concluding Remarks 105 7.1 Research Objectives, Subsequent Research Activities, and Results..... 106 7.2 Future Work................................. 110 7.2.1 Short term work........................... 110 7.2.2 Long Term work........................... 111 References 113 A Appendix: CITRIC 125 A.1 An Overview of CITRIC........................... 125 A.2 The Most Important Objects of CITRIC................... 126 A.3 The Storage of Flows, DNS Records and Events............... 127 A.4 Searching in the HTTP Payload....................... 128 A.5 Settings and Logs.............................. 130 Summary 133 Samenvatting 135 Curriculum Vitæ 137 List