Ph.D. in Electronic and Computer Engineering Dept. of Electrical and Electronic Engineering University of Cagliari Host and Network based Anomaly Detectors for HTTP Attacks Davide Ariu Advisor: Prof. Giorgio Giacinto Curriculum: ING-INF/05 SISTEMI DI ELABORAZIONE DELLE INFORMAZIONI XXII Cycle Marzo 2010 Ph.D. in Electronic and Computer Engineering Dept. of Electrical and Electronic Engineering University of Cagliari Host and Network based Anomaly Detectors for HTTP Attacks Davide Ariu Advisor: Prof. Giorgio Giacinto Curriculum: ING-INF/05 SISTEMI DI ELABORAZIONE DELLE INFORMAZIONI XXII Cycle Marzo 2010 Alla mia famiglia... Host and Network based Anomaly Detectors for HTTP Attacks by Davide Ariu Abstract The huge number of people that everyday connects to Internet makes web applications an attractive target for computer criminals. For example an attack against a web service might be used to quickly spread a malware or to steal ac- cess credential from the web service users. To avoid this the protection of web applications with Intrusion Detection Systems (IDS) is necessary. Unfortunately, protecting web applications is a tricky task since they are in general large, com- plex and highly customized. Traditional systems based on signatures are not ad- equate to guarantee a solid defense since they are not able to face up with zero days attacks. Anomaly-based systems represent a valid alternative to signature based ones and they offer also protection against zero days attacks. In this dissertation we propose several anomaly-based Intrusion Detection Systems for the protection of web applications. We address that of the Intrusion Detection as a Pattern Recognition problem discussing all the aspects that must be considered in realizing an anomaly-based IDS. The formulation of the prob- lem the most suitable for web applications security is the “One-class” formula- tion. This formulation builds a statistical model which is based on legitimate patterns only, thus ignoring any kind of information about the attacks. With this approach potentially any attack pattern can be detected if it is statistically anomalous respect to those within the target (that is the legitimate) class. We propose a Host-based and two Network-based IDS. Both Host and Net- work based solutions are effective even if with different scopes. Host-based IDS are more specific and tailored to protect a particular application. A Network- based IDS offers the great advantage of being able to monitor the traffic toward an entire network segment. In all of the proposed IDS we employ Multiple Clas- sifiers to increase the accuracy of the IDS and to harden the IDS against attempts of evasion. All the IDS have been tested on several datasets of attacks and nor- mal traffic both private and publicly available. Experimental results confirm the effectiveness of the proposed solutions in terms of a high detection rate and with regard to the small amount of false alarms generated. Acknowledgements Understand the things I say, Don’t turn away from me. ’Cause I’ve spent half my life out there,You wouldn’t disagree D. O’Riordan At the end of an amazing experience such as that of pursuing a Ph.D., there is always somebody to thank. I don’t want to escape this unwritten rule, since the last three years have been a school of life before being an unbelievable professional experience. There has been at least one moment in these three years where each one of the people that I’m going to mention has been very important to me. Thus, to thank them I will follow just a chronological order. The first person I would say thank is Prof. Giorgio Giacinto, as he gave to me the possi- bility of living this experience. I would thank him for his valued advices, for his helpfulness and to have been by far more than a simple advisor. The second person I would say thank is Prof. Fabio Roli, for his exhortations to live the research with passion and ambition. I would say thank to both Prof. Giacinto and Roli for gave me the opportunity to join the PRA group and that of beginning my Ph.D. with an amazing experience at the Georgia Institute of Technology. The period in Atlanta has been certainly one of the most hard periods of my Ph.D., but I will always remember it as one of the most beautiful moments of my entire life. Many people helped me in having such an enjoyable period there. Certainly without Andrea, Claudio and Roberto (in a strict alphabetical order) my American staying wouldn’t have had the same taste. I have much appreciated that nice person which is Jack A. Lang, who gave me his hospitality with courtesy and helpfulness. There are also other people who in that period showed me their affection even if I was more than six thousands of kilometers away. Thanks to everybody. A big thank to all the PRA people and, above all, to Battista and Luca. I shared with them moments of both hard work and fun, and I spent hours waiting at the University cafeteria. Finally, there is my Family. I don’t have enough words to explain how much I love them, how much I’m thankful for their kind support and infinite patience and to say how much I’m proud of them. Without them I wouldn’t certainly be who I am and for this I will be eternally grateful to them. iii Ringraziamenti Understand the things I say, Don’t turn away from me. ’Cause I’ve spent half my life out there,You wouldn’t disagree D. O’Riordan Al termine di un esperienza ricca e impegnativa come un Dottorato di Ricerca, c’é sempre qualcuno a cui dover dire grazie. Personalmente, non ho alcuna intenzione di sottrarmi a questa tacita regola, dato che gli ultimi tre anni sono stati per me una scuola di vita prima ancora che una fantastica esperienza professionale. Stabilire una gerarchia e un ordine secondo il quale ringraziare tutte le persone a cui in- tendo esprimere la mia gratitudine non sarebbe possibile e forse nemmeno giusto, dato che c’é stato almeno un momento lungo questo cammino nel quale ognuna di esse é stata fonda- mentale. Per non fare torto a nessuno, mi limiteró pertanto a seguire un ordine cronologico. La prima persona a cui devo dire grazie é il Prof. Giorgio Giacinto, il quale mi ha offerto la possibilitá di vivere quest’esperienza. Lo ringrazio per i suoi preziosi consigli, per essersi dimostrato disponibile ogni qual volta io abbia richiesto il suo aiuto, e per essere stato molto di piú di un semplice tutor. Ringrazio il Prof. Fabio Roli, per l’incitamento a vivere con passione e ambizione l’attivitá di ricerca. Un ringraziamento congiunto a Prof. Giacinto e a Prof. Roli, per avermi offerto la possibilitá di entrare a far parte del gruppo PRA e di iniziare il mio dottorato di ricerca con uno straordinario periodo al Georgia Institute of Technology. Il periodo trascorso ad Atlanta é stato senza dubbio uno dei momenti piú faticosi del mio dottorato ma lo ricorderó sempre come uno dei momenti piú belli e intensi della mia vita. Se ha potuto essere quel che é stato lo devo a diverse persone. Tra queste Andrea, Claudio e Roberto (in rigoroso ordine alfabetico) senza la cui presenza il soggiorno negli Stati Uniti non avrebbe potuto avere lo stesso sapore. Un ringraziamento a parte, lo devo a Jack Lang, il quale mi ha aperto le porte di casa propria con grande cortesia e disponibilitá. Ci sono poi tante altre persone, che non elenco esplicitamente, ma che hanno saputo starmi vicino anche a piú di 6000 chilometri di distanza. A tutte, grazie. Un grazie a tutte le persone del gruppo PRA. Tra tutte, con Battista e Luca in particolare ho avuto modo di condividere, oltre che ore di duro lavoro, anche tanti momenti divertenti, tra cui le lunghe ore di fila ai tornelli delle mense universitarie. Infine il ringraziamento piú grande, va senza dubbio alla mia Famiglia. Nonostante credo sappiano ben poco di ció di cui mi occupo, mi hanno sempre incitato e sostenuto, aiutan- domi a ritrovare l’equilibrio che ogni tanto avevo smarrito. Non ci sono parole che possano esprimere i miei sentimenti, la mia gratitudine e quanto io sia orgoglioso di ognuno di loro. Se ho potuto raggiungere questo traguardo é anche grazie alla serenitá che mi hanno sempre v insegnato ad avere anche di fronte alle difficoltá. E di questo non potró mai essere abbas- tanza grato. Contents 1 Introduction 1 1.1 Contribution of the thesis ............................... 3 1.2 Organization....................................... 4 2 An Introduction to Intrusion Detection5 2.1 Taxonomy of Intrusion Detection Systems...................... 6 2.2 Taxonomy of Attacks .................................. 8 2.3 Intrusion Detection Systems Evaluation....................... 10 2.3.1 ROC Curves ................................... 10 2.3.2 More on evaluation metrics.......................... 11 3 Pattern Recognition Algorithms for Anomaly Detection 13 3.1 One vs. Multi-class Pattern Classification ...................... 15 3.1.1 Multi-class Pattern Classification....................... 16 3.1.2 One-class Pattern Classification ....................... 19 3.2 Algorithms for pattern classification ......................... 21 3.2.1 Hidden Markov Models ............................ 21 3.2.2 One-Class SVM................................. 22 3.3 Multiple Classifier Systems............................... 23 3.3.1 Classifier Selection............................... 24 3.3.2 Classifier Fusion ................................ 25 3.3.3 Combining Multiple One-Class SVM Classifiers.............. 26 4 Web Applications Security: an Host-based solution 29 4.1 Web Applications: An overview ............................ 29 4.2 Attacks against Web Applications........................... 31 4.2.1 SQL injection .................................. 31 4.2.2 Cross Site Scripting - XSS ........................... 31 4.2.3 Remote File Inclusion - Shellcode Injection ...............