A Multi Agent System for Flow-Based Intrusion Detection David A
Total Page:16
File Type:pdf, Size:1020Kb
Air Force Institute of Technology AFIT Scholar Theses and Dissertations Student Graduate Works 3-21-2013 A Multi Agent System for Flow-Based Intrusion Detection David A . Ryan Follow this and additional works at: https://scholar.afit.edu/etd Part of the Digital Communications and Networking Commons Recommended Citation Ryan, David A ., "A Multi Agent System for Flow-Based Intrusion Detection" (2013). Theses and Dissertations. 901. https://scholar.afit.edu/etd/901 This Thesis is brought to you for free and open access by the Student Graduate Works at AFIT Scholar. It has been accepted for inclusion in Theses and Dissertations by an authorized administrator of AFIT Scholar. For more information, please contact [email protected]. A MULTI AGENT SYSTEM FOR FLOW-BASED INTRUSION DETECTION THESIS David A. Ryan, Second Lieutenant, USAF AFIT-ENG-13-M-43 DEPARTMENT OF THE AIR FORCE AIR UNIVERSITY AIR FORCE INSTITUTE OF TECHNOLOGY Wright-Patterson Air Force Base, Ohio DISTRIBUTION STATEMENT A. APPROVED FOR PUBLIC RELEASE; DISTRIBUTION UNLIMITED The views expressed in this thesis are those of the author and do not reflect the official policy or position of the United States Air Force, the Department of Defense, or the United States Government. This material is declared a work of the U.S. Government and is not subject to copyright protection in the United States. AFIT-ENG-13-M-43 A MULTI AGENT SYSTEM FOR FLOW-BASED INTRUSION DETECTION THESIS Presented to the Faculty Department of Electrical and Computer Engineering Graduate School of Engineering and Management Air Force Institute of Technology Air University Air Education and Training Command in Partial Fulfillment of the Requirements for the Degree of Master of Science in Computer Engineering David A. Ryan, B.S.C.E. Second Lieutenant, USAF March 2013 DISTRIBUTION STATEMENT A. APPROVED FOR PUBLIC RELEASE; DISTRIBUTION UNLIMITED AFIT-ENG-13-M-43 A MULTI AGENT SYSTEM FOR FLOW-BASED INTRUSION DETECTION David A. Ryan, B.S.C.E. Second Lieutenant, USAF Approved: 5 MA-e Zv/-.5' Date Dr. Gilbert L. Peterson, PhD (Member) Date Date AFIT-ENG-13-M-43 Abstract The detection and elimination of threats to cyber security is essential for system functionality, protection of valuable information, and preventing costly destruction of assets. This thesis presents a Mobile Multi-Agent Flow-Based IDS called MFIREv3 that provides network anomaly detection of intrusions and automated defense. This version of the MFIRE system includes the development and testing of a Multi- Objective Evolutionary Algorithm (MOEA) for feature selection that provides agents with the “optimal” set of features for classifying the state of the network. Feature selection provides separable data points for the selected attacks: Worm, Distributed Denial of Service, Man-in-the-Middle, Scan, and Trojan. This investigation develops three techniques of self-organization for multiple dis- tributed agents in an intrusion detection system: Reputation, Stochastic, and Maximum Cover. These three movement models are tested for effectiveness in locating good agent vantage points within the network to classify the state of the network. MFIREv3 also introduces the design of defensive measures to limit the effects of network attacks. Defensive measures included in this research are rate-limiting and elimination of infected nodes. The results of this research provide an optimistic outlook for flow-based multi-agent systems for cyber security. The impact of this research illustrates how feature selection in cooperation with movement models for multi agent systems provides excellent attack detection and classification. iv To Rosie. v Acknowledgments I express my deepest gratitude to all those whom continue to support me and sacrificed time and effort to help me through a difficult time. Professor Lamont provided the patience, encouragement, and knowledge making this effort possible. He has dedicated extraordinary time in my education and success. I am indebted to him for my growth as a student, officer, and engineer. My gratitude goes out as well to all the professors in the Computer Engineering department for their advice and teachings. I would not have contemplated this road if not for my parents, who instilled within me a love of creative pursuits, science, and language, all of which finds a place in this thesis. To my parents, thank you. My siblings have also been great friends throughout this process. My best friend stayed in touch weekly while completing Pilot Training and deserves my recognition; as do many other friends from Ohio and the Academy. I thank my classmates for their insights both academically and professionally. I owe uncountably infinite thanks my Maker. David A. Ryan vi Table of Contents Page Abstract......................................... iv Dedication........................................v Acknowledgments.................................... vi Table of Contents.................................... vii List of Figures...................................... xi List of Tables...................................... xiv List of Acronyms.................................... xv I. Introduction.....................................1 1.1 Protecting the Network............................2 1.2 Goal and Objectives..............................5 1.3 Approach...................................6 1.4 Thesis Overview...............................8 II. Background for Flow-Based Intrusion Detection.................. 10 2.1 Network Modeling.............................. 10 2.1.1 Model Limitations.......................... 11 2.1.2 Network Topology Model...................... 12 2.1.3 Simulated Network Traffic...................... 16 2.1.4 Network Simulation Environment.................. 17 2.1.5 Visualization of Network Model................... 18 2.1.6 MASON............................... 19 2.2 Pattern Recognition.............................. 22 2.2.1 Formating Data............................ 23 2.2.2 Feature Selection........................... 24 2.2.3 Embedded Methods......................... 25 2.2.4 Wrapper Methods........................... 26 2.2.5 Filter Methods............................ 26 2.2.6 Hybrid Methods........................... 28 2.2.7 Classification and Regression.................... 29 vii Page 2.2.8 Clustering............................... 33 2.2.9 Support Vector Machines....................... 35 2.3 Intrusion Detection.............................. 44 2.4 Flow-based Intrusion Detection........................ 45 2.5 Network Attacks of Interest.......................... 48 2.5.1 Denial of Service Attacks...................... 48 2.5.2 Vulnerability Scans.......................... 50 2.5.3 Worms................................ 52 2.5.4 Man-in-the-Middle and Eavesdropping............... 53 2.5.5 Trojans and Back-doors....................... 56 2.6 Multi-Objective Optimization......................... 59 2.6.1 Multi-Objective Problems...................... 59 2.6.2 Stochastic Search........................... 60 2.7 Multiagent Systems.............................. 64 2.8 Reputation and Trust............................. 68 2.9 Defensive Measures of Network Agents................... 71 2.10 Summary................................... 75 III. MFIREv3 Design Methodology and Implementation............... 77 3.1 Intrusion Detection System Formalization.................. 77 3.2 Simulation Environment........................... 78 3.2.1 Network Design........................... 81 3.2.2 Multi Agent System Design..................... 86 3.2.3 Simulated Network Traffic...................... 94 3.2.4 Observations and Features...................... 95 3.2.5 Attack Models............................ 98 3.3 Training the Agents.............................. 105 3.3.1 Generating training data....................... 107 3.3.2 Training the Classifier........................ 108 3.3.3 Feature Selection........................... 109 3.3.4 Kernel Method Selection....................... 113 3.3.5 Testing the MAS........................... 115 3.4 Movement models............................... 116 3.4.1 Agents using a reputation model................... 116 3.4.2 Agents using a Stochastic Model................... 120 3.4.3 Deterministic Search with Maximum Cover............. 122 3.5 Defensive Measures Methodology...................... 124 3.6 Visualization................................. 126 3.7 Summary................................... 128 viii Page IV. MFIREv3 Experimentation and Analysis of Results................ 129 4.1 Software Testing............................... 129 4.2 K-Fold Cross-validation........................... 133 4.3 Feature Selection............................... 134 4.4 Kernel Functions............................... 135 4.5 Movement Models Experimental Design................... 137 4.5.1 MFIREv3 Reputation System Experimental Design......... 139 4.5.2 MFIREv3 Stochastic Search Experimental Design......... 139 4.5.3 MFIREv3 Deterministic Maximum Cover Algorithm Experimen- tal Design............................... 141 4.6 Movement Model Analysis.......................... 143 4.6.1 MFIREv3 Reputation System Performance Assessment....... 143 4.6.2 MFIREv3 Stochastic Search Performance Assessment....... 147 4.6.3 MFIREv3 Deterministic Algorithm Performance Assessment.... 151 4.7 Defense Analysis............................... 153 4.8 Summary................................... 155 V. MFIREv3 Conclusions and Future Research.................... 156 5.1 Conclusions.................................. 156 5.2 Future Research Activity..........................