Implementation of a Distributed Intrusion Detection and Reaction System
Total Page:16
File Type:pdf, Size:1020Kb
Mestrado em Engenharia Inform´atica Disserta¸c~ao Relat´orioFinal Implementation of a distributed intrusion detection and reaction system Jo~aoPedro dos Santos Soares [email protected] Orientador: Prof. Dr. Jorge Granjal Co-Orientador: Eng. Ricardo Ruivo Data: 1 de Setembro de 2016 1 Mestrado em Engenharia Inform´atica Disserta¸c~ao Relat´orioFinal Implementation of a distributed intrusion detection and reaction system Jo~aoPedro dos Santos Soares [email protected] Orientador: Prof. Dr. Jorge Granjal Co-Orientador: Eng. Ricardo Ruivo J´uriArguente: Prof. Dr. M´arioRela J´uriVogal: Prof. Dr. Paulo Sim~oes Data: 1 de Setembro de 2016 3 Abstract Security was not always an important aspect in terms of networking and hosts. Nowa- days, it is absolutely mandatory. Security measures must make an effort to evolve at the same rate, or even at a higher rate, than threats, which is proving to be the most difficult of tasks. In this report we will detail the process of the implementation of a real distributed intrusion detection and reaction system, that will be responsible for securing a core set of networks already in production, comprising of thousands of servers, users and their respec- tive confidential information. Keywords: intrusion detection, intrusion prevention, host, network, security, threat, vul- nerability, hacking 5 Acknowledgments As we step into the most important project of my student life, there's a couple of people to whom I am extremely grateful, without them, this project and my life in general wouldn't be the same. I want to start by thanking my supervisor: Professor Jorge Granjal, who I quickly grew to admire for his dedication to this project. I can't stress enough the importance of the presence of a supervisor in the development of a thesis project, from the beginning until the very end, and this was certainly the case. Even on holidays, Professor Jorge Granjal attended to every meeting and made sure I received the appropriate feedback and counseling every time. Along with Professor Jorge Granjal, comes Engineer Ricardo Ruivo, my co-supervisor and work colleague, who more than that, became my friend and mentor at work. When I started working at SIC (Servi¸code Inform´aticae Comunica¸c~oes),there was no better way I could be welcomed if not with Ricardo's good mood and professionalism. He taught me a great deal of what I know now, and is one of the main reasons I wish to continue to work at SIC in the following year. His abilities and skills at work still amaze me to this day. I must also thank my family, that even if not directly involved with this project, supported me for as long as I can remember, not ever doubting my abilities and always encouraging me to go further and to work harder. All this while providing me with all the resources and freedom that I wanted and/or needed. I'm eternally grateful and lucky to have been born surrounded by my wonderful parents and sister. I also want to thank all my close friends that contributed to my resilience throughout the year, by providing me with incredibly fun, and also sometimes crazy moments. In this group of friends I want to give a special thanks to Maxi (Jo~aoLouren¸co),who played a great part in helping me build my current personality, by constantly sharing his optimistic view with me, which was one of the major reasons that kept me going, and made me pursue the master's degree. Last but not least, I want to thank the newest addition to my life: Rita, my girlfriend, who I've known for almost a year, but only a few months ago have I started to look at her the way I already should have in the beginning. Her presence alone inspired me and encouraged me to work, and helped me surpass my sporadic laziness. It impresses me how far we've come and what we've achieved in only a couple of months, we are definitely not normal, and hopefully we stay that way for more years to come. 7 List of Figures 1 Gantt Chart for the first semester. 18 2 Gantt Chart for the second semester. 19 3 New Gantt Chart for the second semester. 20 4 Switched network versus a hubbed network. 25 5 The association (and dissociation) process of a client to a wireless access point. 28 6 Using an intermediary to communicate between wireless clients. 29 7 TCP's three-way handshake . 35 8 A normal TCP data exchange . 37 9 TCP sequence number being brute-forces . 38 10 Wireshark logs of a DNS query . 40 11 Amplification attack in practice . 41 12 A minimalist diagram of the department's network infrastructure. 63 13 Top vulnerabilities when aggregating both inbound and outbound traffic. 65 14 Top vulnerabilities exploited in attacks to the department's network. 66 15 Top targeted machines in attacks to the department's network. 66 16 Top origin countries in attacks to the department's network. 67 17 Top vulnerabilities exploited in attacks originated at the department's network (Outlier). 68 18 Top machines launching attacks originated at the department's network (Out- lier). 68 19 Top vulnerabilities exploited in attacks originated at the department's network. 69 20 Top machines launching attacks originated at the department's network. 70 21 Top targeted countries in attacks originated at the department's network. 71 22 Logical architecture diagram. 80 23 Physical architecture diagram. 83 24 Interaction between the components of our architecture. 85 25 Network diagram of the current architecture implementation . 95 26 Puppet manifest hierarchy . 97 27 Log flow diagram . 102 28 Top vulnerabilities detected by the NIDS . 104 29 Reader and crawler interaction . 105 30 Testing setup diagram. 110 9 List of Tables 1 Comparison of traffic capture techniques. 49 2 Comparison of detection strategies. 56 3 Comparison of intrusion detection/prevention systems. 61 4 Server specifications. 64 5 Availability scenario . 76 6 Modifiability scenario 1 - Updates . 76 7 Modifiability scenario 2 - Network topology changes . 76 8 Performance scenario . 77 9 Testability scenario . 77 10 Usability scenario . 78 11 Scalability scenario . 78 12 Architecture validation tests. 116 13 Reaction times of the three IDS applications. 134 14 Phase 1 results . 135 15 Phase 2 results . 136 16 Phase 3 results . 137 17 Statistics producing test. 138 18 Phase 1 results . 139 19 Phase 2 results . 139 20 Phase 3 results . 140 21 HIDS Availability test results. 140 22 Phase 1 results . 142 23 Phase 2 results . 142 24 Phase 3 results . 143 25 Detailed HIDS scalability testing. 144 26 Misuse-based detection validation. 145 27 Detailed SQL injection test results. 146 28 Detailed cross-site scripting test results. 146 29 Detailed NTP amplification attack test results. 146 30 Anomaly-based detection validation. 147 31 Detailed FTP bruteforcing test results. 147 32 Detailed TCP Syn Flood test results - Phase 1. 148 33 Detailed TCP Syn Flood test results - Phase 2. 148 34 Detailed HTTP directory scan test results. 148 35 Detailed Port scanning test results. 148 36 Detailed system availability threatening test. 149 37 Detailed OSSEC integrity checking testing. 149 38 Detailed OSSEC rootkit detection testing. 149 11 Index 1 Introduction 17 1.1 Contextualization . 17 1.2 Objectives . 17 1.3 Life cycle and Planning . 18 1.4 Report structure . 20 2 State of the Art 23 2.1 Threats and vulnerabilities . 23 2.1.1 Physical Layer . 23 2.1.2 Link Access Layer . 24 2.1.3 Network Layer . 31 2.1.4 Transport Layer . 34 2.1.5 Application Layer . 39 2.2 Detecting intrusions: Technologies and strategies . 44 2.2.1 Techniques for achieving network visibility . 44 2.2.2 Detection strategies . 49 2.2.3 Comparison of detection strategies . 56 2.2.4 Types of intrusion detection systems . 56 2.3 Preventing intrusions: Technologies and strategies . 57 2.3.1 Network intrusion prevention . 57 2.3.2 Host intrusion prevention . 57 2.4 Exploring intrusion detection/prevention systems . 58 2.4.1 Free & open-source products . 58 2.4.2 Commercial products . 60 2.4.3 Comparison of intrusion detection/prevention systems . 61 3 Preliminary Work 63 3.1 Traffic capture and analysis strategy . 64 3.1.1 Capturing the network traffic . 64 3.1.2 Analyzing network traffic . 65 3.2 Preliminary analysis results and conclusion . 65 3.2.1 Attacks against the DEI's network . 66 3.2.2 Attacks originated at the DEI's network . 67 4 System Architecture 73 4.1 Functional Requirements . 73 4.1.1 Detecting . 73 4.1.2 Reacting . 74 4.1.3 Alerting and Logging . ..