Firewalls and Virtual Private Networks A thesis submitted in partial fulfilment of the requirements for the Degree of Master of Commerce in Computer Science in the University of Canterbury by B. A. Harris University of Canterbury 1998 YSICAL 'l"NCj:S Firewalls and Virtual Private Networks Acknowledgements Acknowledgements There are a great number of people that I wish to thank for helping me undertake this thesis. Perhaps I should start with my supervisor and friend, Ray Hunt, who has provided me with unwavering support, encouragement, and of course much needed criticism. I believe that without your help and enthusiasm this work would probably have never seen the light of day. Thank you. I am equally indebted to my fiancee Sharalie, who has stoically endured many months of separation while I pursued my dream. Your love and encouragement were given without hesitation, and for that I love you even more. I cannot go any further without acknowledging all of the support and encouragement given by my parents, sister, and future parents-in-law. Each of you have contributed in your own special way, and helped smooth a path that at times threatened to become very difficult indeed. There are a number of friends that I wish to thank because they have contributed significantly, in a variety of ways, to the completion of my thesis. So please bear with me Matthew Brady for commanding and conquering my battles; Greg Slui and Rebecca Gilmore for being great friends and providing me with somewhere to go; Mark Snelling for inspiration and cryptographic support; Stephen Harvey for entertaining discussion and helping me understand public-key infrastructures; and Sylvester and Scooby-Doo for providing essential relaxation and escapism. Finally, a very special mention is deserved for Malcolm Shore of the Government Communications Security Bureau. Malcolm has provided a great deal of technical direction and support, and was instrumental in arranging for me to take leave from work to return to university and complete my thesis. Thank: you. Funding for this thesis was provided by the Government Communications Security Bureau. Brendon Harris 1998 vii Firewalls and Virtual Private Networks Table of Contents Table of Contents LIST OF FIGURES xiii LIST OF TABLES xv CHAPTER 1. INTERNET BEGINNINGS 1 1.1 INTRODUCTION 1 1.2 THE ARPANET 1 1.3 NEW ZEALAND AND THE INTERNET 2 1.4 INTERNET GROWTH 4 1.5 SUMMARY 8 CHAPTER 2. OVERVIEW OF TCP/IP 9 2.1 INTRODUCTION 9 2.2 THE TCP/IP PROTOCOL SUITE 9 2.3 INTERNET PROTOCOL (IP) 11 2.3.1 lP ADDRESSES 13 2.3.2 lP SECURITY LABELS 14 2.4 INTERNET CONTROL MESSAGE PROTOCOL (ICMP) 14 2.5 ADDRESS RESOLUTION PROTOCOL (ARP) 16 2.6 TRANSMISSION CONTROL PROTOCOL (TCP) 16 2.7 USER DATAGRAM PROTOCOL (UDP) 19 2.8 DOMAIN NAME SERVICE 19 2.9 SUMMARY 21 CHAPTER 3. THREATS FROM THE INTERNET 23 3.1 INTRODUCTION 23 3.2 THREATS TO THE TCP!IP PROTOCOL 26 3.2.1 SYN FLOODING 26 3.2.2 lP SPOOFING, TCP SEQUENCE NUMBER PREDICTION, AND TCP SESSION HIJACK 28 3.2.3 RST AND FIN ATTACK 33 3.2.4 PING 0' DEATH 34 3.3 THREATS TO STANDARD TCP!IP SERVICES 35 3.3.1 SIMPLE MAIL TRANSPORTPROTOCOL(SMTP) 36 3.3.2 TELNET 37 3.3.3 NETWORK TIME PROTOCOL (NTP) 37 3.3.4 FINGER AND WHOIS 38 3.3.5 NETWORK FILE SYSTEM (NFS) 39 3.3.6 FILE TRANSFER PROTOCOL (FTP) 39 3.3.7 WORLD WIDE WEB (WWW) 40 3.3.8 X-WINDOW SYSTEM 41 3.4 SUMMARY 41 ix Table of Contents Firewalls and Virtual Private Networks CHAPTER 4. FIREWALL TECHNOLOGY 43 4.1 INTRODUCTION 43 4.2 FIREWALL TERMINOLOGY 43 4.3 THE OSI MODEL 45 4.4 DEFINING BOUNDARIES 47 4.5 THE ROLE OF A SECURITY POLICY 48 4.5.1 NETWORK SERVICE ACCESS POLICY (NSAP) 49 4.5 .2 FIREWALL DESIGN POLICY (FDP) 50 4.5.3 SAMPLE POLICIES 52 4.5.4 POLICY EVOLUTION 52 4.6 SUMMARY 53 CHAPTER 5. FIREWALL ARCHITECTURES 55 5.1 INTRODUCTION 55 5.2 SCREENING-ROUTER 55 5.3 DUAL-HOMED GATEWAY 57 5.4 SCREENED-HOST GATEWAY 59 5.5 SCREENED-SUBNET 61 5.6 HYBRID GATEWAYS 62 5.7 FIREWALL LIMITATIONS 63 5.8 SUMMARY 64 CHAPTER 6. CERTIFICATION OF FIREWALL TECHNOLOGY 65 6.1 INTRODUCTION 65 6.2 PROBLEMS WITH FIREWALL EVALUATION 66 6.3 GOVERNMENT CERTIFICATION 66 6.3.1 DEVELOPMENT OF INFORMATION TECHNOLOGY SECURITY EVALUATION CRITERIA 66 6.3.2 OVERVIEW OF THE ITSEC 67 6.3.3 OVERVIEW OF THE AUSTRALIAN INFORMATION SECURITY EVALUATION PROGRAMME69 6.4 COMMERCIAL CERTIFICATION 72 6.4.1 ICSA FIREWALL CERTIFICATION 72 6.4.2 ICSA FIREWALL TESTING 74 6.5 SUMMARY 75 CHAPTER 7. VIRTUAL PRIVATE NETWORKS 77 7.1 WHAT IS A VIRTUAL PRIVATE NETWORK? 77 7.2 VIRTUAL PRIVATE NETWORKS AND THE INTERNET 78 7.3 OVERVIEW OF CRYPTOGRAPHY 80 7.3.1 SECRET-KEY (SYMMETRIC) CRYPTOGRAPHY 80 7.3.2 PUBLIC-KEY (ASYMMETRIC) CRYPTOGRAPHY 82 7 .3.3 DIGITAL SIGNATURES 84 7.3.4 CERTIFICATE AUTHORITIES 85 7.4 VIRTUAL PRIVATE NETWORK TECHNOLOGY 87 7.5 POINT-TO-POINT TUNNELLING PROTOCOL 90 7.5 .1 PPTP SECURITY 92 7.6 IP SECURITY (IPSEC) 93 7.6.1 SECURITY ASSOCIATION 94 X Firewalls and Virtual Private Networks Table of Contents 7.6.2 AUTHENTICATION 95 7 .6.3 CONFIDENTIALITY 96 7.6.4 KEY MANAGEMENT 99 7.7 SECURE SOCKETS LAYER 100 7. 7.1 SESSION ESTABLISHMENT 102 7.7.2 DATA TRANSFER 103 7. 7.3 SSL AND PROXIES 104 7.8SUMMARY 105 CHAPTER 8. CONCLUSIONS 107 8.1 THE FUTURE OF INTERNET SECURITY 107 8.2 PROBLEMS AND FUTURE RESEARCH 108 APPENDIX A ITSEC TARGET EVALUATION LEVELS 111 APPENDIX B NCSA FWPD CRITERIA 113 APPENDIX C PRIVILEGED PORT NUMBERS 117 REFERENCES 123 xi Firewalls and Virtual Private Networks List of Figures List of Figures FIGURE 1-1 THE LOCATION OF EACH UNIVERSITY INNEW ZEALAND, INCLUDING THE LOGICAL TOPOLOGY OF THE KAWAIHOKI NETWORK(SEE INSET) ....................................................................................... 3 FIGURE 1-2 GROWTH OF HOST MACHINES ON THE INTERNET ................................................................... 5 FIGURE 1-3 COMPARISON OFWWW-BROWSER USE BY US AND EUROPEAN USERs: ............................... 5 FIGURE 1-4 COMPARISON OF INTERNET TECHNOLOGIES USED BYUS AND EUROPEAN USERS .................. 6 FIGURE 1-5 SERVICES USED BY INTERNET USERS IN THE UK. .................................................................. 7 FIGURE 2-1 THE FOUR LAYERS OF THE TCPIIP PROTOCOL STACK. .......................................................... 9 FIGURE 2-2 RELATIONSHIP OF PROTOCOLS IN THETCP/IP PROTOCOL SUITE. ......................................... 10 FIGURE2-3 lPDATAGRAMFORMAT....................................................................................................... 11 FIGURE2-4 ICMP MESSAGESTRUCTUREINRELATIONTOlP DATAGRAM ENCAPSULATION .................. 14 FIGURE 2-5 TCP SEGMENT FORMAT...................................................................................................... 16 FIGURE 2-6 TCP 3-wAy HANDSHAKE. ................................................................................................... 19 FIGURE 2-7 HIERARCHICAL ORGANISATION OF THFDNS ....................................................................... 21 FIGURE 3-1 TCP SYN FLOOD ATTACK. .................................................................................................. 27 FIGURE3-2 TCP 3-WAYHANDSHAKEANDDATA TRANSFER ................................................................. 29 FIGURE 3-3 EXAMPLE OF A BLIND SPOOFING ATTACK ............................................................................ 31 FIGURE 3-4 EXAMPLE OF A TCP SESSION HIJACK. .................................................................................. 32 FIGURE 4-1 COMPARISON OF OSI AND TCPIIP COMMUNICATIONS ARCHITECTURES. ............................ 46 FIGURE 4-2 OSI MODEL IN RELATION TO THE VARIOUS FIREWALL ARCHITECTURES .............................. 47 FIGURE4-3 THE ZONE-OF-RISK FOR AN ORGANISATIONAL NETWORK CONNECTED TO THEINTERNET WITHOUT A FIREWALL ARCHITECTURE ............................................................................................ 47 FIGURE 4-4 THE ZONE-OF-RISK WITH FIREWALL ARCHITECTURE IN PLACE ............................................ 48 FIGURE 5-1 THE COST OF FIREWALL ARCHITECTURES IN COMPARISON TO THE LEVEL OF SECURITY THEY PROVIDE.......................................................................................................................................... 55 FIGURE 5-2 THE OSI LAYERS AT WHICH THE SCREENINGROUTER FUNCTIONS ...................................... 56 FIGURE 5-3 TYPICAL SCREENING ROUTER BASED FIREWALL ARCHITECTURE ........................................ 56 FIGURE 5-4 THE OSI LAYERS AT WHICH THE DUAlrHOMED GATEWAY FUNCTIONS. ............................... 58 FIGURE 5-5 TYPICAL DUAL-HOMED GATEWAY....................................................................................... 58 FIGURE 5-6 THE OSI LAYERS AT WHICH THE SCREENED-HOST FIREWALL ARCHITECTURE FUNCTIONS .. 60 FIGURE 5-7 TYPICAL SCREENED-HOST FIREWALL ARCHITECTURE ......................................................... 60 FIGURE 5-8 THE OSI LAYERS AT WHICH THE SCREENED-SUBNET FIREWALL ARCHITECTURE FUNCTIONS61 FIGURE 5-9 TYPICAL SCREENED-SUB NET FIREWALL ARCHITECTURE ..................................................... 62 FIGURE 6-1 ITSEC ASSURANCE LEVELS ............................................................................................... 68 FIGURE 6-2 COMPARISON OFCC AND ITSEC ASSURANCE LEVELS ........................................................ 69 FIGURE 7-1 EXTRANET SECURED BY A VPN.......................................................................................... 79 FIGURE7-2 SYMMETRIC ALGORITHM ENCRYPTION AND DECRYPTION ................................................... 81 FIGURE 7-3 ASYMMETRIC ALGORITHM