Universidade Federal de Pernambuco Centro de Informática Pós-Graduação em Ciência da Computação Daniel Araújo Melo ARCA – Alerts Root Cause Analysis Framework Dissertação de Mestrado Recife 2014 Universidade Federal de Pernambuco Centro de Informática Daniel Araújo Melo ARCA - Alerts Root Cause Analysis Framework This dissertation has been submitted to the Informat- ics Center of the Federal University of Pernambuco as a partial requirement to obtain the degree of Master in Computer Science. Orientador: Djamel F. H. Sadok Recife 2014 Catalogação na fonte Bibliotecária Jane Souto Maior, CRB4-571 M528a Melo, Daniel Araújo ARCA - Alerts root cause analysis framework / Daniel Araújo Melo. – Recife: O Autor, 2014. 122 f.: il., fig., tab. Orientador: Djamel Fawzi Hadj Sadok. Dissertação (Mestrado) – Universidade Federal de Pernam- buco. CIn, Ciência da computação, 2014. Inclui referências. 1. Redes de computadores. 2. Segurança da informação. I. Sadok, Djamel Fawzi Hadj (orientador). II. Título. 004.6 CDD (23. ed.) UFPE- MEI 2015-42 Daniel Araújo Melo ARCA - Alerts Root Cause Analysis Dissertação apresentada ao Programa de Pós-Graduação em Ciência da Computação da Universidade Federal de Pernambuco, como requisito parcial para a obtenção do tí- tulo de Mestre em Ciência da Computação. Aprovado em: 08/09/2014 BANCA EXAMINADORA __________________________________________ Prof. Dr. Stênio Flávio de Lacerda Fernandes Centro de Informática / UFPE __________________________________________ Prof. Dr. Arthur de Castro Callado Mestrado e Doutorado em Ciências da Computação / UFC ___________________________________________ Prof. Dr. Djamel Fawzi Hadj Sadok (Orientador) Centro de Informática / UFPE A minha família, esposa e filhos. Acknowledgments Initially, I would like to thank my family, especially my mother, Carmem Dolores, my wife Juliana, my son Enos Daniel and my grandmothers, Olga and Inez. They have always stood by my side even when I was absent working in this research. I would like to gratefully acknowledge the supervision of Professor Djamel Sadok. He provided me important suggestions and encouragement during the course of this work and offered the opportunity to join GPRT research team My sincere thanks also goes to Professor Judith Kelner for pulling my ears when needed and helping me when I lost the matriculation. I would not complete the aca- demic requirements without her help. I´d like to thank to my examination committee, Stenio Fernandes e Arthur Cal- lado, for suggestions that enriched this work. I cordially thank to my colleagues from GPRT for the help and revision of my presentation, and colleagues from SERPRO, especially those that always believed that this moment would come. I want to express my gratitude to Andre Tio, Lalá, Tadeu, Noemi, Iuri, Nacho, Suana, Amanda, Maíra, for the good vibrations. And finally, thanks Universe! “If you know the enemy and know yourself you need not fear the results of hundred battles.” - Sun Tzu Abstract Modern virtual plagues, or malwares, have focused on internal host infection and em- ploy evasive techniques to conceal itself from antivirus systems and users. Traditional network security mechanisms, such as Firewalls, IDS (Intrusion Detection Systems) and Antivirus Systems, have lost efficiency when fighting malware propagation. Recent researches present alternatives to detect malicious traffic and malware propagation through traffic analysis, however, the presented results are based on experiments with biased artificial traffic or traffic too specific to generalize, do not consider the existence of background traffic related with local network services or demands previous knowledge of networks infrastructure. Specifically don’t consider a well-known intru- sion detection systems problem, the high false positive rate which may be responsible for 99% of total alerts. This dissertation proposes a framework (ARCA – Alerts Root Cause Analysis) capable of guide a security engineer, or system administrator, to iden- tify alerts root causes, malicious or not, and allow the identification of malicious traffic and false positives. Moreover, describes modern malwares propagation mechanisms, presents methods to detect malwares through analysis of IDS alerts and false positives reduction. ARCA combines an aggregation method based on Relative Uncertainty with Apriori, a frequent itemset mining algorithm. Tests with 2 real datasets show an 88% reduction in the amount of alerts to be analyzed without previous knowledge of network infrastructure. Palavras-chave: Intrusion detection. Malwares. Alerts correlation. Advanced persis- tent threats. Resumo As pragas virtuais modernas focam na contaminação de estações em redes internas, e empregam técnicas evasivas para se ocultarem dos sistemas antivírus e dos usuá- rios dos sistemas. Mecanismos tradicionais de segurança de rede, como firewalls, sis- temas de detecção de intrusão (IDS – Intrusion Detection Systems) e sistemas antiví- rus, perdem eficiência no combate a propagação de malwares. Pesquisas apresentam alternativas para detectar de tráfego malicioso e propagação de malwares através da análise de tráfego, mas apresentam resultados baseados em conjuntos de dados ar- tificiais enviesados ou reais específicos demais para serem generalizados, não consi- deram a existência de tráfego de background relacionado com serviços de rede local ou exigem conhecimento prévio da infraestrutura de rede. Especificamente não con- sideram um problema bem conhecido dos IDS: a alta taxa de falsos positivos, que podem chegar a 99% do total de alertas. Esta dissertação propõe um framework (ARCA – Alerts Root Cause Analysis) capaz de auxiliar um engenheiro de segurança a identificar causas-raiz de alertas, maliciosos ou não, permitindo a identificação de tráfego malicioso e falsos positivos. Adicionalmente, descreve os mecanismos de pro- pagação de malwares modernos, propostas de detecção de malwares através da aná- lise de alertas emitidos por IDS e propostas de redução de falsos positivos. ARCA combina um mecanismo de agregação de alertas baseado na Incerteza Relativa com o algoritmo de análise de itens frequentes Apriori. Testes realizados com dados reais demonstraram uma redução em até 88% a quantidade de alertas a serem analisados sem conhecimento prévio da infraestrutura de rede Palavras-Chaves: Intrusion detection. Malware. Alerts correlation. Advanced persis- tent threats. Lista de Figuras Figure 1 Worm propagation model (ZOU et al., 2005) .............................................. 24 Figure 2 Typical bonet´s elements (SILVA et al., 2013) ........................................... 26 Figure 4 Typical botnet life-cycle proposed in (FEILY; SHAHRESTANI; RAMADASS, 2009) ......................................................................................................................... 29 Figure 5 Botnet life cycle proposed in (RODRÍGUEZ-GÓMEZ; MACIÁ-FERNÁNDEZ; GARCÍA-TEODORO, 2013) ...................................................................................... 31 Figure 6 IRC-based botnet DDOS Attack (COOKE; JAHANIAN; MCPHERSON, 2005) .................................................................................................................................. 33 Figure 7 Hybrid P2P network .................................................................................... 36 Figure 10 Gameover Zeus network topology. Dotted line indicates information flow. .................................................................................................................................. 41 Figure 11 Organizations Categories (MCAFEE, 2010) ............................................. 43 Figure 12 Victim´s Country of Origin (MCAFEE, 2010) ............................................ 44 Figure 13 Model for APT stages proposed by (GIURA; WANG, 2012). .................... 44 Figure 14 A targeted attack in action (SOOD; ENBODY, 2013) ............................... 45 Figure 15 Infected Hosts according Wan IP (FALLIERE; MURCHU; CHIEN, 2011) 48 Figure 16 Overview of Stuxnet Malware Operation .................................................. 49 Figure 17 Countries affected by Flame according to McAfee (GOSTEV, 2012b) .... 51 Figure 18 Countries affected by Flame according Symantec (SYMANTEC, 2012b) 52 Figure 19 Flame C&C Platform(ZHIOUA, 2013) ....................................................... 54 Figure 20 An example of (a) bipartite graph and (b) one-mode projection. .............. 55 Figure 21 BotHunter System by (PORRAS, 2009) .................................................. 56 Figure 22 Vulnerabilities reported do NVD (NIST, 2014). ......................................... 59 Figure 23 Incidents reported to Cert.br (CERT.BR, 2014) ........................................ 60 Figure 24 Layout of the proposed classification system in (PARIKH; CHEN, 2008). .................................................................................................................................. 68 Figure 25 A sample multi-step-attack (SOLEIMANI; GHORBANI, 2008) ................. 70 Figure 26 Generic view of alarm correlation according (HUBBALLI; SURYANARAYANAN, 2014)..................................................................................... 71 Figure 27 Generic view of graph ordering (PAO et al., 2012). .................................. 74 Figure 28 ATLANTIDES architecture (BOLZONI; CRISPO; ETALLE, 2007) ............ 75 Figure 29 Proposed Architecture (HUBBALLI; BISWAS; NANDI, 2011). ................. 76 Figure 30 Normalized SrcIp and DstIp