The Pennsylvania State University The Graduate School College of Engineering PACKET INSPECTION FOR APPLICATION CLASSIFICATION AND INTRUSION DETECTION A Dissertation in Electrical Engineering by Jisheng Wang © 2008 Jisheng Wang Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy May 2008 ii The dissertation of Jisheng Wang was reviewed and approved* by the following: David J. Miller Associate Professor of Electrical Engineering Dissertation Co-Adviser Co-Chair of Committee George Kesidis Professor of Computer Science and Engineering Professor of Electrical Engineering Dissertation Co-Adviser Co-Chair of Committee Nirmal K. Bose HRB-Systems Professor of Electrical Engineering Prasenjit Mitra Assistant Professor of the School of Information Sciences and Technology Assistant Professor of Computer Science and Engineering Assistant Professor of Industrial and Manufacturing Engineering Kenneth W. Jenkins Professor of Electrical Engineering Head of the Department of Electrical Engineering *Signatures are on file in the Graduate School iii Abstract Current computer networks remain vulnerable to a variety of families of attacks including scanning worms, distributed denial-of-service (DDoS) attacks targeting resources associated with end-systems or critical network protocols, and hit-list worms. These kinds of attacks remain significant direct and indirect threats to the network’s infrastructure and its end-systems. Despite past developments, anomaly detection and response targeting zero-day attacks (as not yet seen) remains an open research problem. This dissertation presents the complete structure of an automated payload-based network intrusion detection system, which includes three main components: network traffic mining, network anomaly identification, and worm signature extraction. Estan et al.’s multidimensional digesting algorithm is introduced to mine significant flows – either worm flows or dominant normal flows – among entire network traffic, and several techniques are proposed for improving its efficiency. Based on the mining results, a new entropy-based criterion is presented to correctly identify anomaly network traffic, including the Slammer and Code-Red worms and the DDoS attacks. Moreover, a Generalized Suffix Tree-based approach is proposed for efficiently extracting signatures of polymorphic worms. Therefore, the proposed intrusion detection system can iv automatically generate signatures of zero-day attacks/worms which can be used to contain their spread in the future. Meanwhile, with the increasing flexibility in current networks, tons of new applications appear and begin to dominate the Internet. The newly emerging peer-to-peer applications, such as Bitcomet and Skype, can be responsible for more than 80% of the total traffic volume in the Internet. Therefore, it is essential for Internet service providers to correctly identify these new applications. This dissertation presents an efficient approach to identify Skype voice over IP (VoIP) traffic by using reliable statistical information. Because of its efficiency in both computational complexity and memory consumption, the new approach can be implemented on network backbone routers to identify Skype VoIP traffic in real-time. v Table of Contents List of Acronyms.................................................................................................... ix List of Figures........................................................................................................ xi List of Tables........................................................................................................xiii Acknowledgments................................................................................................ xiv Chapter 1. Introduction........................................................................................ 1 1.1 Background........................................................................................ 1 1.1.1 Network Traffic Management ................................................. 2 1.1.2 Network Intrusion Detection................................................... 3 1.1.3 “Lawful Interception” of IP Data Traffic ................................ 4 1.2 Contributions...................................................................................... 5 1.3 Organization....................................................................................... 6 Chapter 2. Multidimensional Network Traffic Digesting.................................... 8 2.1 Introduction........................................................................................ 8 2.2 Multidimensional, Hierarchical Flow Mining of Network Traffic .. 14 2.2.1 Identifying Significant Unidimensional Flows ..................... 17 2.2.2 Identifying Significant Multidimensional Flows .................. 21 2.2.3 Improving the Efficiency of Multidimensional Flow Mining23 2.2.4 Implementation Considerations............................................. 31 vi 2.3 Experiments Comparing Computational Efficiency........................ 33 2.4 Conclusion ....................................................................................... 39 Chapter 3. Network Intrusion Detection Systems ............................................. 40 3.1 Introduction of Network Attacks ..................................................... 40 3.2 Review of Network Intrusion Detection Systems............................ 45 3.2.1 Host/Operation System-Based Intrusion Detection .............. 45 3.2.2 Network-Based Intrusion Detection...................................... 47 3.2.3 Packet Payload-Based Intrusion Detection ........................... 49 3.3 Comprehensive Intrusion Defense System ...................................... 51 3.4 White-Listing in Payload-Based Detection ..................................... 53 3.5 Covert Malware Modeling that Exploits White-Listing .................. 55 3.6 Port-80 Data Traffic and Peer-to-Peer Traffic.................................. 59 3.7 Conclusion ....................................................................................... 61 Chapter 4. Multidimensional Mining-Based Network Anomaly Identification 62 4.1 Introduction...................................................................................... 62 4.2 Criterion for Anomaly Identification ............................................... 64 4.2.1 Leaf and Internal Node Clusters............................................ 66 4.3 Attack Identification Results............................................................ 67 4.3.1 DARPA Trace ........................................................................ 68 4.3.2 Sapphire/Slammer Trace ....................................................... 71 4.3.3 Code-Red version 2 Trace ..................................................... 72 vii 4.4 Discussion and Relation to Prior Work............................................ 79 4.5 Conclusion ....................................................................................... 83 Chapter 5. Generalized Suffix Tree-Based Worm Signature Extraction ........... 85 5.1 Introduction...................................................................................... 85 5.2 Prior Work on Worm Signature Extraction...................................... 88 5.3 New Polymorphic Worm IDS.......................................................... 93 5.3.1 Directly Mining Suspicious Clusters..................................... 93 5.3.2 Worm Signature Extraction ................................................... 97 5.4 Experimental Methodology ........................................................... 100 5.4.1 Polymorphism via Encryption Schemes.............................. 100 5.4.2 Issues in Salting Background with Worm Traffic ............... 101 5.5 Experimental Results and Discussion............................................ 102 5.6 Conclusion ..................................................................................... 107 Chapter 6. Identifying VoIP Traffic by Using Reliable Statistical Signatures. 108 6.1 Introduction and Motivation .......................................................... 108 6.2 Skype Transmission Mechanism.....................................................112 6.2.1 Peer-to-Peer Structure...........................................................112 6.2.2 Obfuscation Played by Skype...............................................114 6.3 Related Work...................................................................................117 6.4 Efficient Statistical Method for Identifying VoIP Traffic .............. 123 6.4.1 Statistical Feature Selection ................................................ 123 viii 6.4.2 Implementation Considerations........................................... 126 6.5 Statistical Analysis of Skype VoIP Traffic ..................................... 131 6.5.1 Skype Video......................................................................... 133 6.5.2 Skype Voice ......................................................................... 136 6.5.3 Skype Phone........................................................................ 139 6.5.4 Growing Window versus Sliding Window.......................... 142 6.6 Experimental Results ..................................................................... 148 6.6.1 Training Data....................................................................... 149 6.6.2 Performance Evaluation ...................................................... 149 6.7 Conclusion