INTRUSION DETECTION FOR 0-DAY VULNERABILITIES A thesis submitted to Kent State University in partial fulfillment of the requirements for the degree of Master of Science By Nathan Daniel Truhan August, 2011 Thesis written by Nathan Daniel Truhan B.S., Youngstown State University, 2000 M.S., Kent State University, 2011 Approved by Michael Rothstein, Advisor John Stalvey, Chair, Department of Computer Science Timothy Moerland, Dean, College of Arts and Sciences ii TABLE OF CONTENTS CHAPTER 1 BASICS OF INTRUSION DETECTION SYSTEMS ............................ 1 1.1 Hackers ...................................................................................................................... 1 1.2 Zero-Day Vulnerabilities ........................................................................................... 2 1.3 What is an Intrusion Detection System ..................................................................... 3 1.4 Snort .......................................................................................................................... 3 1.5 Operating Systems ..................................................................................................... 4 1.6 Network Layer ........................................................................................................... 4 1.7 Data Collection Network Setup ................................................................................. 5 1.8 Common Setup Parameters ....................................................................................... 5 1.9 First Possible Network Configuration ....................................................................... 6 1.10 Primary Network Configuration ................................................................................ 8 1.11 Setup Summary ....................................................................................................... 10 CHAPTER 2 HONEYPOT AND SNORT IDS DATA COLLECTION ANALYSIS SERVER CONFIGURATION .......................................................................... 11 2.1 Installation of openSUSE Linux 11.x ..................................................................... 11 2.2 Configuring Additional Components ...................................................................... 12 2.3 Basic Analysis and Security Engine ........................................................................ 13 CHAPTER 3 SNORT INTRUSION DETECTION SERVER CONFIGURATION 14 3.1 Choosing the Operating System .............................................................................. 14 3.2 Installing openSUSE Linux 11.x for an IDS ........................................................... 15 3.3 Configuring the Network ........................................................................................ 16 iii 3.4 Installing Snort ........................................................................................................ 17 3.5 Loading the latest Snort rules with Oinkmaster ...................................................... 18 CHAPTER 4 HONEYPOT DECOY SERVER ........................................................... 19 4.1 Why implement a honeypot .................................................................................... 19 4.2 Selecting a Honeypot .............................................................................................. 20 4.3 Argos and QEMU .................................................................................................... 21 4.4 The Ubuntu 8.04 LTS Honeypot ............................................................................. 22 4.5 Alternate Microsoft Windows XP Honeypot .......................................................... 23 4.6 Configuring Snort to detect a compromised system ............................................... 23 4.7 Testing the honeypot setup ...................................................................................... 24 CHAPTER 5 SUMMARY .............................................................................................. 25 5.1 What this thesis has provided .................................................................................. 25 5.2 Detecting vulnerabilities ......................................................................................... 25 5.3 Results ..................................................................................................................... 26 5.4 Contributions ........................................................................................................... 28 5.5 Future work ............................................................................................................. 28 APPENDIX A INSTALLATION OF OPENSUSE 11.X ............................................. 30 AA Partitioning Options ................................................................................................ 32 AB User Authentication ................................................................................................. 36 AC Selecting Additional Software ................................................................................ 37 AD Completing the Installation ..................................................................................... 38 AE Configuring the Network ........................................................................................ 41 iv AF Configuring the Firewall ......................................................................................... 46 APPENDIX B INSTALLING THE MYSQL 5.5 COMMUNITY EDITION DATABASE SERVER........................................................................................ 51 BA Downloading the MySQL Server ............................................................................ 51 BB Configuring the MySQL Server databases .............................................................. 53 APPENDIX C INSTALLING THE APACHE HTTP SERVER ............................... 56 CA Preparing the Apache HTTP Server ........................................................................ 57 APPENDIX D CONFIGURING THE SNORT NETWORK ..................................... 60 DA LIBPCAP and TCPDUMP ...................................................................................... 62 DB Physical Network Connections ............................................................................... 63 DC Testing the network configuration .......................................................................... 64 APPENDIX E INSTALLING THE PHP HYPERTEXT PREPROCESSOR .......... 66 APPENDIX F INSTALLING BASIC ANALYSIS AND SECURITY ENGINE (BASE) ................................................................................................................. 68 FA Installing ADOdb .................................................................................................... 68 FB Installing Perl Compatible Regular Expressions ..................................................... 69 FC Installing PEAR::Image_Canvas and PEAR::Image_Graph .................................. 69 FD Installing PEAR::Mail and PEAR::Mail_Mime ..................................................... 70 FE Basic Analysis and Security Engine Configuration ................................................ 70 APPENDIX G INSTALLING SNORT PREREQUISITES ....................................... 73 GA Installing Perl Compatible Regular Expressions ..................................................... 73 GB Installing the libdnet networking library ................................................................. 73 v GC Installing MySQL 5.5 Client Libraries ................................................................... 74 GD Installing the Snort Data AcQuisition library.......................................................... 75 APPENDIX H INSTALLING SNORT ......................................................................... 77 HA Configuring Snort .................................................................................................... 78 HB Preparing the snort.conf file .................................................................................... 79 HC Installing the Oinkmaster Snort rule manager ......................................................... 83 HD Disabling Unwanted Rules ...................................................................................... 85 HE Snort and Network Startup Script ........................................................................... 86 APPENDIX I INSTALLING DAMN SMALL LINUX ............................................... 89 IA Additional Software Dependencies ......................................................................... 92 IB Installing a new Linux kernel .................................................................................. 93 IC Installing the Simple Directmedia Layer ................................................................ 96 ID Installing Autoconf .................................................................................................. 97 IE Installing Bridged Networking ................................................................................ 97 IF Configuring the honeypot server network ............................................................. 100 APPENDIX J INSTALLING ARGOS AND QEMU ................................................ 102 JA Installing QEMU ................................................................................................... 102 JB Installing the KQEMU module ............................................................................