An Efficient Intrusion Detection System for Networks with Centralized Routing Paulo Filipe Canha de Andrade Dissertação para obtenção de Grau de Mestre em Engenharia Informática e de Computadores Júri Presidente: Prof. Luis Eduardo Teixeira Rodrigues Orientador: Prof. Fernando Henrique Corte Real Mira da Silva Co-orientador: Prof. Carlos Nuno da Cruz Ribeiro Vogais: Prof. Rodrigo Seromenho Miragaia Rodrigues Setembro de 2007 Abstract As Internet becomes more and more ubiquitous, security is an increasingly important topic. Furthermore, private networks are expanding and security threats from within the network have to be cautioned. For these large networks, which are generally high-speed and with several segments, Intrusion Detection System (IDS) placement usually comes down to a compromise between investment and monitoring ability. One common solution in these cases, is to use more than one IDS scattered across the network, thus raising the amount invested and administrative power to operate. Another solution is to collect data through sensors and send it to one IDS via an Ethernet hub or switch. This option normally tends to overload the hub/switch port where the IDS is connected. This document presents a new solution, for networks with a star topology, where a single IDS is coupled to the network’s core router. This solution allows the IDS to monitor every different network segment attached to the router in a round-robin fashion. Practical implementation issues and operational implications of this solution are also analyzed and discussed. Keywords: Intrusion Detection Systems,Security Analysis, High-speed Networks, Switch-based Networks. i Resumo À medida que a Internet se torna cada vez mais acessível, a segurança é cada vez mais um tópico muito importante. Para além disso, com o crescimento célere de redes privadas, as ameaças proveniente do interior da rede têm que ser acauteladas. Para estas redes de grande dimensão, que são geralmente de alta velocidade e com vários segmentos, o posicionamento de um Sistema de Detecção de Intrusões (SDI) acaba normalmente num compromisso entre o investimento e a capacidade de monitorização. Uma solução comum para estas redes, é o uso de vários SDIs espalhados pela rede. Elevando assim, o investimento necessário e os recursos administrativos necessários. Outra solução possível, é a de colher os dados da rede através de sensores e enviá-los para um SDI via um comutador de pacotes. Contudo, com esta solução, existe o risco de sobrecarregar a porta do comutador de pacotes onde está ligado o SDI. Este documento descreve uma nova solução para redes com uma topologia em estrela. Onde um único SDI está acoplado ao nó central da rede. Esta solução permite que o SDI monitorize todos os segmentos ligados a esse nó central de uma forma iterativa. Detalhes de implementação e implicações operacionais são também analisados. Palavras Chave: Sistemas de Detecção de Intrusos, Análise de Segurança, Redes de Alta Velocidade, Redes baseadas em Comutadores de Pacotes. ii Acknowledgements I would like express my gratitude to the supervising professors Fernando Mira da Silva and Carlos Ribeiro for their support, critical judgment and expertise. And from whom I learnt a lot. Fellow coworkers at CIIST, Jorge Matias, Miguel Cabeça, Cláudio Martins and André Regateiro, for their pointers and overall good humor. They allowed for a great working environment. And finally my family and girlfriend, Diana Nunes, for being there when troubles emerged and for letting me spend so much time around my work. iii Table of Contents Abstract i Resumo ii Acknowledgements iii Table of Contents v List of Figures vi List of Tables vii List of Acronyms viii 1 Introduction 1 1.1 Context . .1 1.2 Objectives . .2 1.3 Structure of this thesis . .3 2 Context/Area 4 2.1 Intrusion Detection Systems . .4 2.1.1 Terminology . .5 2.2 Different Types of Intrusion Detection Systems . .5 2.2.1 Host-based Intrusion Detection Systems . .7 2.2.2 Network-based Intrusion Detection Systems . .9 3 Network Topologies 13 3.1 IDS Placement . 13 iv 3.1.1 Basic Network Setups . 13 3.1.2 Larger Network Setups . 17 3.2 Discussion . 19 4 Port hopping 20 4.1 Networks with Centralized Routing . 20 4.2 Port hopper approach . 20 4.3 Value . 21 4.4 Requirements . 22 5 Implementation 24 5.1 Software Architecture . 25 5.1.1 IDS integration with the software . 26 5.2 IDS Deployment . 27 5.2.1 Signature Thresholds . 28 6 Evaluation 32 6.1 Design Issues . 32 6.2 Implementation Issues . 33 6.3 Monitoring Window . 33 7 Conclusion 35 7.1 Achieved Value . 35 7.2 Final Remarks . 35 7.3 Future Work . 36 Bibliography 37 Appendix One 40 .1 Example configuration file for the software . 40 .2 Example interaction with the system . 42 Appendix Two 44 .3 Packet information displayed by Basic Analysis and Security Engine (BASE) . 44 v List of Figures 1.1 Growth in number of incidents reported to the CERT/CC . .1 1.2 Attack sophistication versus intruder knowledge (reproduced from [CER06]). .2 3.1 A common network topology. 14 3.2 Using a hub/tap to copy traffic to the IDS. 15 3.3 Schematic representation of a network Tap. 16 3.4 Using the switch port mirroring capabilities to copy traffic to the IDS. 17 3.5 Consolidating the taps output with an application-switch which is, in turn, balancing the load to two IDSs. 18 4.1 Example of a network with a star topology, where the IDS is connected directly to the central node.................................................. 21 4.2 Coupling of the IDS with the central router. 22 5.1 Network topology of Instituto Superior Técnico (reproduced from [Mic06]). 24 5.2 Structural view of the software. 25 vi List of Tables 2.1 Summary of IDS properties. .6 5.1 Top three signatures at the end of the first day. 29 5.2 Top four signatures at the end of the second day. 30 vii List of Acronyms IDS Intrusion Detection System CERT/CC Computer Emergency Response Team Coordination Center DoS Denial-of-Service TCP Transmission Control Protocol UDP User Datagram Protocol OS Operating System HIDS Host-based Intrusion Detection System NIDS Network-based Intrusion Detection System AIDS Application-based Intrusion Detection System SIDS Stack-based Intrusion Detection System AIDE Advanced Intrusion Detection Environment MAC Media Access Control SPAN Switch Port Analyzer CRC Cyclic Redundancy Check VLAN Virtual Local Area Network GPL GNU General Public License XML Extensible Markup Language IP Internet Protocol PDA Personal Digital Assistant SSH Secure Shell SNMP Simple Network Management Protocol CPU Central Processing Unit OID Object Identifier MIB Management Information Base IST Instituto Superior Técnico BASE Basic Analysis and Security Engine CIIST Centro de Informática do Instituto Superior Técnico ICMP Internet Control Message Protocol DNS Domain Name System P2P Peer-to-Peer viii Chapter 1 Introduction This document presents a new solution in the area of Intrusion Detection System placement, and discusses an actual implementation done at Instituto Superior Técnico (IST). This chapter intends to provide the motivation behind this solution, as well as describe its objectives. 1.1 Context Over the past two decades, with the rapid growth of the Internet — which now counts with more than 100 million sites [Net06] — companies have been forced to change the way they do business. To keep up with new Internet-centric companies or simply to still be competitive, many companies have had to alter their business process to accommodate this new means of operation and communication. 160000 137529 140000 120000 100000 82094 80000 60000 55100 Number of Incidents 40000 31756 20000 9852 1334 2340 2412 2573 2134 3734 0 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 Years Figure 1.1: Growth in number of incidents reported to the CERT/CC 1 However, along with this growth, the number of attacks to Internet sites has also increased dramatically. As Fig. 1.1 [Las03] denotes, between 2000–2003, the number of incidents reported to the Computer Emergency Response Team Coordination Center (CERT/CC) grew around 400%. There are a few factors that contribute to this astonishing rate. First, there is the continuously publication of exploits and vulnerabilities on the Internet as they are discovered. Secondly, there is a profusion of intrusion tools and automated scripts available that duplicate known methods of attack. These two factors combined allow for practically anyone with little technical knowledge to be able to perform an attack. Consequently, the number of sophisticated attacks has been increasing. Fig. 1.2 illustrates this point [CER06]. HIGH Tools Staged “Stealth” / advanced Auto Coordinated Intruder scanning techniques Knowledge Cross site scripting Automated probes/scans Distributed attack tools Sniffers www attacks Sweepers GUI Packet spoofing denial of service Back doors Network mgmt. diagnostics Disabling audits Hijacking sessions Burglaries Exploiting known vulnerabilities Attack Password cracking Sophistication Self-replicating code Intruders Password guessing LOW 1980 1985 1990 1995 2000 Figure 1.2: Attack sophistication versus intruder knowledge (reproduced from [CER06]). 1.2 Objectives The main goal of the proposal described in this document is to provide a cost-effective solution to IDS place- ment for networks with a star topology. The goals of this proposal are the following: • Make use of existing technologies and freely available tools, as well as, require few hardware to imple- ment, thus keeping costs at a minimum. • To provide a flexible solution, such that it is feasible for all networks with a star topology and be ad- justable for new and changing environments. • To provide valuable information, namely statistics, in a format legible to any system administrator, prefer- ably through a web interface. 2 1.3 Structure of this thesis In chapter 2, an introduction to IDSs is presented and the state of the art is reviewed.