An Efficient Intrusion Detection System for Networks with Centralized Routing

Paulo Filipe Canha de Andrade

Dissertação para obtenção de Grau de Mestre em Engenharia Informática e de Computadores

Júri

Presidente: Prof. Luis Eduardo Teixeira Rodrigues Orientador: Prof. Fernando Henrique Corte Real Mira da Silva Co-orientador: Prof. Carlos Nuno da Cruz Ribeiro Vogais: Prof. Rodrigo Seromenho Miragaia Rodrigues

Setembro de 2007 Abstract

As Internet becomes more and more ubiquitous, security is an increasingly important topic. Furthermore, private networks are expanding and security threats from within the network have to be cautioned. For these large networks, which are generally high-speed and with several segments, Intrusion Detection System (IDS) placement usually comes down to a compromise between investment and monitoring ability. One common solution in these cases, is to use more than one IDS scattered across the network, thus raising the amount invested and administrative power to operate. Another solution is to collect data through sensors and send it to one IDS via an Ethernet hub or switch. This option normally tends to overload the hub/switch port where the IDS is connected. This document presents a new solution, for networks with a star topology, where a single IDS is coupled to the network’s core router. This solution allows the IDS to monitor every different network segment attached to the router in a round-robin fashion. Practical implementation issues and operational implications of this solution are also analyzed and discussed.

Keywords: Intrusion Detection Systems,Security Analysis, High-speed Networks, Switch-based Networks.

i Resumo

À medida que a Internet se torna cada vez mais acessível, a segurança é cada vez mais um tópico muito importante. Para além disso, com o crescimento célere de redes privadas, as ameaças proveniente do interior da rede têm que ser acauteladas. Para estas redes de grande dimensão, que são geralmente de alta velocidade e com vários segmentos, o posicionamento de um Sistema de Detecção de Intrusões (SDI) acaba normalmente num compromisso entre o investimento e a capacidade de monitorização. Uma solução comum para estas redes, é o uso de vários SDIs espalhados pela rede. Elevando assim, o investimento necessário e os recursos administrativos necessários. Outra solução possível, é a de colher os dados da rede através de sensores e enviá-los para um SDI via um comutador de pacotes. Contudo, com esta solução, existe o risco de sobrecarregar a porta do comutador de pacotes onde está ligado o SDI. Este documento descreve uma nova solução para redes com uma topologia em estrela. Onde um único SDI está acoplado ao nó central da rede. Esta solução permite que o SDI monitorize todos os segmentos ligados a esse nó central de uma forma iterativa. Detalhes de implementação e implicações operacionais são também analisados.

Palavras Chave: Sistemas de Detecção de Intrusos, Análise de Segurança, Redes de Alta Velocidade, Redes baseadas em Comutadores de Pacotes.

ii Acknowledgements

I would like express my gratitude to the supervising professors Fernando Mira da Silva and Carlos Ribeiro for their support, critical judgment and expertise. And from whom I learnt a lot.

Fellow coworkers at CIIST, Jorge Matias, Miguel Cabeça, Cláudio Martins and André Regateiro, for their pointers and overall good humor. They allowed for a great working environment.

And finally my family and girlfriend, Diana Nunes, for being there when troubles emerged and for letting me spend so much time around my work.

iii Table of Contents

Abstract i

Resumo ii

Acknowledgements iii

Table of Contents v

List of Figures vi

List of Tables vii

List of Acronyms viii

1 Introduction 1

1.1 Context ...... 1

1.2 Objectives ...... 2

1.3 Structure of this thesis ...... 3

2 Context/Area 4

2.1 Intrusion Detection Systems ...... 4

2.1.1 Terminology ...... 5

2.2 Different Types of Intrusion Detection Systems ...... 5

2.2.1 Host-based Intrusion Detection Systems ...... 7

2.2.2 Network-based Intrusion Detection Systems ...... 9

3 Network Topologies 13

3.1 IDS Placement ...... 13

iv 3.1.1 Basic Network Setups ...... 13

3.1.2 Larger Network Setups ...... 17

3.2 Discussion ...... 19

4 Port hopping 20

4.1 Networks with Centralized Routing ...... 20

4.2 Port hopper approach ...... 20

4.3 Value ...... 21

4.4 Requirements ...... 22

5 Implementation 24

5.1 Software Architecture ...... 25

5.1.1 IDS integration with the software ...... 26

5.2 IDS Deployment ...... 27

5.2.1 Signature Thresholds ...... 28

6 Evaluation 32

6.1 Design Issues ...... 32

6.2 Implementation Issues ...... 33

6.3 Monitoring Window ...... 33

7 Conclusion 35

7.1 Achieved Value ...... 35

7.2 Final Remarks ...... 35

7.3 Future Work ...... 36

Bibliography 37

Appendix One 40

.1 Example configuration file for the software ...... 40

.2 Example interaction with the system ...... 42

Appendix Two 44

.3 Packet information displayed by Basic Analysis and Security Engine (BASE) ...... 44

v List of Figures

1.1 Growth in number of incidents reported to the CERT/CC ...... 1

1.2 Attack sophistication versus intruder knowledge (reproduced from [CER06])...... 2

3.1 A common network topology...... 14

3.2 Using a hub/tap to copy traffic to the IDS...... 15

3.3 Schematic representation of a network Tap...... 16

3.4 Using the switch port mirroring capabilities to copy traffic to the IDS...... 17

3.5 Consolidating the taps output with an application-switch which is, in turn, balancing the load to two IDSs...... 18

4.1 Example of a network with a star topology, where the IDS is connected directly to the central node...... 21

4.2 Coupling of the IDS with the central router...... 22

5.1 Network topology of Instituto Superior Técnico (reproduced from [Mic06])...... 24

5.2 Structural view of the software...... 25

vi List of Tables

2.1 Summary of IDS properties...... 6

5.1 Top three signatures at the end of the first day...... 29

5.2 Top four signatures at the end of the second day...... 30

vii List of Acronyms

IDS Intrusion Detection System CERT/CC Computer Emergency Response Team Coordination Center DoS Denial-of-Service TCP Transmission Control Protocol UDP User Datagram Protocol OS Operating System HIDS Host-based Intrusion Detection System NIDS Network-based Intrusion Detection System AIDS Application-based Intrusion Detection System SIDS Stack-based Intrusion Detection System AIDE Advanced Intrusion Detection Environment MAC Media Access Control SPAN Switch Port Analyzer CRC Cyclic Redundancy Check VLAN Virtual Local Area Network GPL GNU General Public License XML Extensible Markup Language IP Internet Protocol PDA Personal Digital Assistant SSH Secure Shell SNMP Simple Network Management Protocol CPU Central Processing Unit OID Object Identifier MIB Management Information Base IST Instituto Superior Técnico BASE Basic Analysis and Security Engine CIIST Centro de Informática do Instituto Superior Técnico ICMP Internet Control Message Protocol DNS Domain Name System P2P Peer-to-Peer

viii Chapter 1

Introduction

This document presents a new solution in the area of Intrusion Detection System placement, and discusses an actual implementation done at Instituto Superior Técnico (IST).

This chapter intends to provide the motivation behind this solution, as well as describe its objectives.

1.1 Context

Over the past two decades, with the rapid growth of the Internet — which now counts with more than 100 million sites [Net06] — companies have been forced to change the way they do business. To keep up with new Internet-centric companies or simply to