Detecting malicious network activity using flow data and learning automata By Christian Auby Torbjørn Skagestad Kristian Tveiten Thesis submitted in Partial Fulfillment of the requirements for the Degree Master of Technology in Information and Communication Technology Faculty of Engineering and Science University of Agder Grimstad May 2009 Table of Contents 1 Introduction.................................................................................................................9 1.1 Background and motivation...............................................................................10 1.2 Proposed solution...............................................................................................13 1.3 Limitations .........................................................................................................14 1.4 Assumptions.......................................................................................................15 2 Related work .............................................................................................................16 2.1 DDoS detection based on traffic profiles...........................................................17 2.1.1 Goal and results...........................................................................................17 2.1.2 Relevance....................................................................................................17 2.1.3 Differences..................................................................................................17 2.2 An intelligent IDS for anomaly and misuse detection in computer networks ...19 2.2.1 Goal and results...........................................................................................19 2.2.2 Relevance....................................................................................................19 2.2.3 Differences..................................................................................................19 2.3 Visualizing netflow data ....................................................................................21 2.3.1 Goal and results...........................................................................................21 2.3.2 Relevance....................................................................................................21 2.3.3 Differences..................................................................................................21 2.4 An efficient intrusion detection model based on fast inductive learning...........22 2.4.1 Goal and results...........................................................................................22 2.4.2 Relevance....................................................................................................22 2.4.3 Differences..................................................................................................22 2.5 Argus..................................................................................................................24 2.5.1 argus............................................................................................................24 2.5.2 ra .................................................................................................................24 2.5.3 rasplit...........................................................................................................24 2.5.4 racluster.......................................................................................................24 2.6 Snort...................................................................................................................25 3 Experimental setup....................................................................................................26 3.1 Malware selection ..............................................................................................27 3.2 Malware analysis ...............................................................................................29 3.2.1 Router..........................................................................................................32 3.2.2 Firewall/ bridge...........................................................................................32 3.2.3 Client network.............................................................................................32 3.2.4 Operation.....................................................................................................33 3.3 Test sets..............................................................................................................34 3.3.1 Test set 1 - Initial research..........................................................................34 3.3.2 Test set 2 - Mass infection ..........................................................................35 3.3.3 Test set 3 - Large client network.................................................................35 3.4 Automata implementation..................................................................................36 3.5 Automata testing ................................................................................................37 3.5.1 Detection.....................................................................................................38 3.5.2 Performance ................................................................................................38 4 Malware ....................................................................................................................41 4.1 BlackEnergy.......................................................................................................42 2 of 125 4.1.1 Network behavior........................................................................................42 4.1.2 Client behavior............................................................................................43 4.1.3 Spreading ....................................................................................................43 4.1.4 Lab testing...................................................................................................43 4.2 Conficker............................................................................................................44 4.2.1 Conficker.A.................................................................................................44 4.2.2 Conficker.B.................................................................................................44 4.2.3 Conficker.C.................................................................................................45 4.2.4 Network behavior........................................................................................45 4.2.5 Client behavior............................................................................................45 4.2.6 Spreading ....................................................................................................45 4.2.7 Protection features ......................................................................................46 4.2.8 Lab testing...................................................................................................46 4.3 Danmec ..............................................................................................................47 4.3.1 Network behavior........................................................................................47 4.3.2 Client behavior............................................................................................47 4.3.3 Spreading ....................................................................................................48 4.3.4 Lab testing...................................................................................................48 4.4 Waledac..............................................................................................................50 4.4.1 Network behavior........................................................................................50 4.4.2 Client behavior............................................................................................50 4.4.3 Spreading ....................................................................................................50 4.4.4 Lab testing...................................................................................................51 5 Proposed flow based LRPI and SWE IDS................................................................52 5.1 Class structure....................................................................................................53 5.1.1 Main application .........................................................................................53 5.1.2 AutomataController ....................................................................................54 5.1.3 Automaton discarding.................................................................................56 5.1.4 Automaton...................................................................................................57 5.1.5 Implement based on rules ...........................................................................59 5.2 Specific automaton.............................................................................................62 5.2.1 BlackEnergy................................................................................................62 5.2.2 Conficker.....................................................................................................63 5.2.3 Danmec .......................................................................................................63