A Proactive Network Surveillance Framework
Total Page：16
File Type：pdf, Size：1020Kb
A Proactive Network Surveillance Framework A Thesis submitted for the award of the degree of DOCTOR OF PHILOSOPHY by Maninder Singh under the Guidance of Dr. S. C. Saxena Director, Indian Institute of Technology, Roorkee Uttaranchal, INDIA -247667 & Dr. Seema Bawa Professor, Department of Computer Science & Engineering Thapar Institute of Engineering & Technology, Patiala Punjab, INDIA -147004 Computer Science and Engineering Department Thapar Institute of Engineering & Technology A Deemed University, Patiala – 147 004, INDIA January, 2007 dedicated to One Universal Creator God. The Name Is Truth. Creative Being Personified. No Fear. No Hatred. Image Of The Undying, Beyond Birth, Self-Existent. By Guru’s Grace Chant And Meditate True In The Primal Beginning. True Throughout The Ages. True Here And Now. O Nanak, Forever And Ever True ||1||1 1Sri Guru Granth Sahib Ji Contents List of Figures . v List of Tables . viii Certificate . ix Acknowledgement . x Abstract . xii 1 Introduction 1 1.1 Primary Network stakeholders: Hosts and Domains . 2 1.1.1 World Internet Usage and Population Statistics . 4 1.2 A preamble to Network Security . 5 1.3 Organization of the Thesis . 9 2 Literature Review 12 2.1 OSI Model and Various Protocols . 13 2.1.1 Open System Interconnection Model . 13 2.1.2 Internet Protocol (IP) . 16 2.1.3 Transport Control Protocol (TCP) . 18 2.1.4 Address Resolution Protocol (ARP) . 21 2.1.5 Internet Control Message Protocol (ICMP) . 22 2.2 Network Security Expatiation . 23 2.2.1 Reactive Security . 38 2.2.2 Proactive Security . 44 2.3 Problem Formulation . 50 3 Exploits and their Detrimental Effects 52 ii 3.1 Life Cycle of a Vulnerability and an Exploit . 52 3.2 Stack Overruns . 54 3.2.1 Exploit Development Process . 58 3.2.2 Life Cycle of a Typical Windows Exploit . 59 3.2.3 Finding the Attack Vector . 60 3.2.4 Selecting a Control Vector . 64 3.2.5 Finding and Using a Return Address . 65 3.2.6 Finding Bad Characters . 66 3.2.7 Choosing a Payload and Encoder . 67 3.2.8 Suggested Overflow Defense Methods . 68 3.3 Denial of Service Attacks (DoS) . 71 3.3.1 Ping of Death . 72 3.3.2 Land Exploit . 75 3.3.3 Smurf . 76 3.3.4 SYN Flood . 78 3.4 A Live walk-through into System Hacking . 80 3.4.1 Reconnaissance and Enumeration Phase . 81 3.4.2 Attack Phase . 85 3.4.3 Analysis of Hacked Data . 86 3.5 Conclusions . 87 4 Analysis of Opensource Honeypots 88 4.1 Use of Honeypots in Network Security . 88 4.1.1 Using Honeypots in the DeMilitarized Zone (DMZ) . 89 4.2 Tradeoffs between Levels of Interaction . 90 4.2.1 Low Level of Interaction . 90 4.2.2 Medium Level of Interaction . 91 4.2.3 High Level of Interaction . 92 4.3 Honeypots: Exploration and Analysis . 92 4.3.1 BackOfficer Friendly (BOF) . 93 4.3.2 Honeyd . 95 iii 4.3.3 Honeynets . 100 4.4 Conclusions . 106 5 Proposed Proactive Network Surveillance Framework: Design and Im- plementation Details 107 5.1 Core Security Layer . 108 5.1.1 Implementation Details: Filesystem Level . 108 5.1.2 Implementation Details: SSH login Key-pair . 115 5.2 Routing and Traffic Control Layer . 119 5.2.1 Implementation Details . 120 5.3 Security Information System Layer . 128 5.3.1 Implementation Details . 129 5.4 Perimeter Security Layer . 137 5.4.1 Implementation Details: Firewall . 137 5.4.2 Implementation Details: IDS . 141 5.5 Learn and Monitor the Unknown Layer . 144 5.5.1 Implementation Details: Hardening Steps . 144 5.5.2 Implementation Details: Honeypot . 148 5.6 Conclusions . 150 6 Deployment, Testing of the Framework and Results Obtained 151 6.1 Framework Evaluation and Results: Case I . 151 6.2 Framework Evaluation and Results: Case II . 157 6.3 Conclusions . 164 7 Conclusions and Future Scope of the Work 175 7.1 Conclusions . 175 7.2 Future Scope of Work . 178 References 180 List of Publications 191 iv List of Figures 1.1 Internet Domain Survey Host Count . 5 1.2 Crime Loss Statistics . 7 1.3 Dial-up Connections bypassing Network Periphery Security. 7 2.1 Headers and Data can be Encapsulated during Information Exchange . 15 2.2 IP Packet Format . 17 2.3 TCP Packet Format . 19 2.4 A Transport Control Protocol [TCP] Session. 20 2.5 The hierarchy of Security specializations. 24 2.6 Layout showing the major ISPs [Che99] . 27 2.7 Cyberspace and the Physical World . 28 2.8 Attack Sophistication and required Intruder Knowledge . 29 2.9 Number of Intruders able to Execute Attack . 30 2.10 Security Attacks . 31 2.11 Schematic of a Firewall . 39 2.12 Generation I Honeynet . 49 2.13 Generation II Honeynet . 50 3.1 Vulnerability Life Cycle . 53 3.2 Exploit Categorization . 54 3.3 Memory layout of a Linux process . 56 3.4 View of a Stack . 57 3.5 Exploit Analysis Test-Bed . 60 3.6 A Banner Grab of IIS version 5.0 . 62 3.7 A Banner Grab of IIS version 5.1 . 62 v 3.8 First Attack String after appropriate buffer calculations . 63 3.9 IIS 5.0 attached to OllyDbg for debugging . 64 3.10 Successful Exploitation of Windows2000 Sp0 . 68 3.11 Distributed Denial of Service Attack . 71 3.12 Smurf Attack Signatures . 77 3.13 SYN Flood Attack Signatures . 80 3.14 Active Reconnaissance: Probing the system with a tool . 82 3.15 Enumeration Phase of the Victim Machine . 83 3.16 Using DCOM-RPC Exploit . 85 3.17 Importing LM and NT hashes for password Cracking . 86 4.1 Honeypot Deployment in DeMilitarized Zone (DMZ) . 90 4.2 Test Bed for Honeypots Exploration and Analysis . 93 4.3 Back Officer Friendly Honeypot Running and Emulating services . 94 4.4 BOF Honeypot Telnet fake replies . 95 4.5 Nmap output: Attacker scanning Honeyd default host . 96 4.6 Basic Architecture of Honeynet . 102 4.7 Tcpdump network traffic log analyzed using Wireshark . 105 4.8 Flow graph statistics of an attack . 105 5.1 Proposed Framework . 109 5.2 Packet’s Journey through Kernel Tables and qdiscs . 123 5.3 Security Information System Layer . 129 5.4 Firewall placement inside network security hierarchy. 137 5.5 Firewall flow chart based on IPTables. 139 5.6 Intrusion Detection System (IDS) placement inside network security hier- archy. ..................................... 142 5.7 Virtual Honeypots Setup . 149 6.1 Test Bed for Testing the Proposed Framework . 152 6.2 Protocol Hierarchy Statistics . 154 6.3 IO Flow Graphs . 155 vi 6.4 Testing with Live Network at Thapar Institute of Engineering and Tech- nology . ..