Session-Based Intrusion Detection System to Map Anomalous Network Traffic
Total Page:16
File Type:pdf, Size:1020Kb
SESSION-BASED INTRUSION DETECTION SYSTEM TO MAP ANOMALOUS NETWORK TRAFFIC by BRUCE D. CAULKINS B.S. Furman University, 1986 M.S. University of Central Florida, 1995 A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Department of Modeling and Simulation in the College of Arts and Sciences at the University of Central Florida Orlando, Florida Fall Term 2005 Major Professor: Morgan C. Wang © 2005 Bruce D. Caulkins ii ABSTRACT Computer crime is a large problem (CSI, 2004; Kabay, 2001a; Kabay, 2001b). Security managers have a variety of tools at their disposal – firewalls, Intrusion Detection Systems (IDSs), encryption, authentication, and other hardware and software solutions to combat computer crime. Many IDS variants exist which allow security managers and engineers to identify attack network packets primarily through the use of signature detection; i.e., the IDS recognizes attack packets due to their well-known “fingerprints” or signatures as those packets cross the network’s gateway threshold. On the other hand, anomaly-based ID systems determine what is normal traffic within a network and reports abnormal traffic behavior. This paper will describe a methodology towards developing a more-robust Intrusion Detection System through the use of data-mining techniques and anomaly detection. These data-mining techniques will dynamically model what a normal network should look like and reduce the false positive and false negative alarm rates in the process. We will use classification-tree techniques to accurately predict probable attack sessions. Overall, our goal is to model network traffic into network sessions and identify those network sessions that have a high-probability of being an attack and can be labeled as a “suspect session.” Subsequently, we will use these techniques inclusive of signature detection methods, as they will be used in concert with known signatures and patterns in order to present a better model for detection and protection of networks and systems. iii To my wife Kim, whose love, patience, and support meant everything for my research iv ACKNOWLEDGMENTS Many thanks to the members of my committee, Drs. Morgan C. Wang, Joohan Lee, William H. Allen, Mark E. Johnson, J. Peter Kincaid, and Michael D. Proctor. Much credit goes to Morgan Wang for his guidance and mentorship. In a multi- disciplinary area like modeling and simulations, it takes a special professor who has wide- ranging knowledge and patience to make this research work well and go smoothly. Special thanks also goes to Joohan Lee – a specialist in computer security and reliable computing, he provided an incredible insight into the problems that we encountered during our research. Bill Allen at FIT also gave key assistance with his extensive expertise as well as the FIT data sets. The US Army Human Resources Command (HRC) provided support for this research through the Army’s fully funded Advanced Civil Schooling (ACS) program. I thank them as well as my boss at my current assignment at the US Army Signal Center at Fort Gordon, Georgia – Colonel Mackey. He as well as the good folks at the Army’s School of Information Technology (SIT) gave me the time and encouragement to complete my research properly and on time. Finally, I’d like to thank my family and friends, particularly my wife Kim. Much has happened to all of us since Kim and I started this journey in August 2002 and my dad’s onslaught of dementia was the absolute low point over the last three years. However, he always told me to study hard, never quit, take as many English classes as you can in college, and always put the family first. His current condition won’t allow him to remember ever saying those words but that’s fine – I’ve never forgotten that advice. v TABLE OF CONTENTS LIST OF FIGURES ....................................................................................................................... xi LIST OF TABLES.......................................................................................................................xiii LIST OF ACRONYMS/ABBREVIATIONS.............................................................................. xiv CHAPTER 1: INTRODUCTION.................................................................................................. 1 1.1 A Historical Look at Computer Security Issues .................................................................. 2 1.1.1 The 1980s and 1990s .................................................................................................... 3 1.1.1.1 Cuckoo’s Egg......................................................................................................... 3 1.1.1.2 Internet Worm and Mitnick’s Attacks ................................................................... 4 1.1.2 More Recent Malicious Events..................................................................................... 5 1.2 Security Issues of Today...................................................................................................... 7 1.2.1 CSO Magazine Survey Results..................................................................................... 8 1.2.2 CERT/CC Statistics .................................................................................................... 11 1.2.3 Trends and Attack Profiles.......................................................................................... 13 1.3 Thesis Statement ................................................................................................................ 14 CHAPTER 2: DATA MINING ................................................................................................... 16 2.1 Data Mining Basics............................................................................................................ 17 2.1.1 Data Cleaning, Selection, and Transformation........................................................... 19 2.1.2 Data Mining and Pattern Evaluation........................................................................... 19 2.2 Data Mining Patterns ........................................................................................................ 21 2.2.1 Characterization and Discrimination .......................................................................... 21 2.2.2 Association.................................................................................................................. 22 2.2.3 Clustering.................................................................................................................... 23 vi 2.2.4 Outliers........................................................................................................................ 24 2.2.5 Evolution..................................................................................................................... 24 2.2.6 Classification and Prediction ...................................................................................... 25 2.3 A Decision Tree Analysis ................................................................................................. 25 2.3.1 Decision Trees and Large Data Sets ........................................................................... 26 2.3.2 The Model Building Procedure................................................................................... 27 2.3.3 Consequences of Using Decision Trees...................................................................... 29 2.4 Data Mining In Our Research........................................................................................... 30 CHAPTER 3: INTRUSION DETECTION SYSTEMS .............................................................. 32 3.1 Network Intrusion Detection Systems ............................................................................... 32 3.2 Misuse Detection and Anomaly Detection ........................................................................ 33 3.3 Why IDS Programs are Needed......................................................................................... 34 3.4 Anomaly Detectors of Today............................................................................................. 35 3.4.1 CLAD and LERAD..................................................................................................... 35 3.4.2 EFSA........................................................................................................................... 36 3.4.3 Distributed Data Mining and Database Mining.......................................................... 37 3.4.4 IDEVAL...................................................................................................................... 37 3.4.5 Statistical Analysis on Network Packets..................................................................... 38 CHAPTER 4: SESSION-BASED INTRUSION DETECTION.................................................. 40 4.1 Packet-Based IDS System.................................................................................................. 41 4.2 Session-Based IDS Tools................................................................................................... 42 4.2.1 Tcpdump and Editcap ................................................................................................. 42 4.2.2 Snort IDS Freeware..................................................................................................... 43 vii 4.2.3 Cap2Txt2 Java Program.............................................................................................