Formal Analysis of Network Protocol Security DISSERTATION Presented in Partial Fulfillment of the Requirements for the Degree Do
Total Page:16
File Type:pdf, Size:1020Kb
Formal Analysis of Network Protocol Security DISSERTATION Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy in the Graduate School of The Ohio State University By Yating Hsu Graduate Program in Computer Science and Engineering The Ohio State University 2011 Dissertation Committee: Dr. David Lee, Advisor, Dr. Dong Xuan, Dr. Ten H. Lai Copyright by Yating Hsu 2011 Abstract As computer networks turn into an indispensable part of technology and entertainment of human life today, security and reliability become the forefront issue of network protocol design and implementation. Any security or reliability flaws in network protocol operation can lead to great loss in private data, business opportunity and reputation. As a result, methods and tools for preventing and detecting such security breach in network protocol design and implementation are urgently needed. Formal methods have proven to be successful in system verification and testing. This thesis is devoted to formally design algorithm and tools to efficiently and effectively investigate security of both protocol specification and implementation. For protocol specification, we first model a protocol as a state machine. To verify certain security property against the state machine, we need to explore all reachable states in the state machine. However, traditional reachability problem suffers from the classic state space explosion problem. We propose two techniques to tackle this problem: trace inclusion transformation and online minimization. We first transform the original state machine to a simplified machine which is a trace inclusion transformation of the original machine by removing certain system parameters. We apply online minimization to construct a minimal reachable graph of manageable size that is bisimulation equivalent to the reachability graph of the simplified machine. Then we can verify the desired security ii property against the minimal reachable graph. However, we may introduce false positive and false negative since we have simplified the original machine. In the second phase of analysis, we show that there is no false negative if the simplified machine is a trace inclusion transformation of the original machine. We also design algorithm to eliminate false positive. Then we can conclude if the state machine (the protocol design) is security against the checked security property. However, even with perfect design, fault can still be introduced during the implementation and integration phase. Due to the black-box nature of protocol implementation and often unavailability of specification, traditional verification techniques do not apply here. In this thesis, we first automatically synthesize a formal model to represent the behavior of the implementation and then use the synthesized model to assist different security flaw detection applications. We study both active and passive synthesize approaches and compare their advantages and disadvantages. We incorporate the formal model to two different security flaw detection applications. In the first application, we build a model based fuzz testing framework in which the formal model is used to guide test input selection and served as test coverage criteria. Our framework significantly improves existing black-box protocol security testing techniques. In the second application, we examine the synthesized model to determine if there is any malicious function hidden inside a protocol implementation and determine the behavior of the hidden function. We show the proposed approaches are effective with extensive case studies for various scenarios and applications. Nevertheless, the techniques and framework proposed in this iii thesis is generic and could be easily adapted for different problem domains and applications. iv Dedication Dedicated to my parents, Mr. James Hsu and Mrs. Ming-Hsiu Chao v Acknowledgments I would like to extend my gratitude to my advisor, Prof. David Lee, for his continuous guidance, tolerance and financial support during my graduate study. Appreciation also goes to Prof. Steve Lai for his help during my last year of study. I am also deeply indebted to my colleague Dr. Guoqiang Shu and many others from the CSE department. Their help and encouragement made my study smooth and with less detour. Finally, I would like to thank my parents and family in Taiwan for their unconditional love and support. vi Vita Aug 31, 1984 ..................................................Born in Taipei Taiwan 2002-2006 ......................................................B.S. Computer Science and Information Engineering, National Taiwan University, Taiwan 2006 to present ..............................................OSU Fellow, Graduate Teaching and Research Assistant, The Ohio State University Publications Yating Hsu and David Lee, “Machine Learning for Implanted Malicious Code Detection with Incompletely Specified System Implementations,” in Proc. 2nd International Workshop on Security and Trust in Future Internet (FIST) 2011. Yating Hsu and David Lee, “Authentication and Authorization Protocol Security Property Analysis with Trace Inclusion Transformation and Online Minimization,” in Proc. IEEE International Conference on Network Protocols (ICNP) 2010. Yating Hsu, Guoqiang Shu and David Lee, “A Model-based Approach to Security Flaw Detection of Network Protocol Implementation,” in Proc. IEEE International Conference on Network Protocols (ICNP) 2008. vii Guoqiang Shu, Yating Hsu and David Lee, “Detecting Communication Protocol Security Flaws by Formal Fuzz Testing and Machine Learning,” in Proc. 28th IFIP International Conference on Formal Techniques for Networked and Distributed Systems (FORTE), 2008. Fields of Study Major Field: Computer Science and Engineering viii Table of Contents Abstract ............................................................................................................................... ii Dedication ........................................................................................................................... v Acknowledgments.............................................................................................................. vi Vita .................................................................................................................................... vii List of Tables ................................................................................................................... xiii List of Figures .................................................................................................................. xiv Chapter 1: Introduction ....................................................................................................... 1 1.1 Security Analysis of Protocol Specifications ............................................................ 2 1.1.1 Authentication and Authorization ...................................................................... 4 1.2 Security Flaw Detection of Protocol Implementations ............................................. 4 1.2.1 Fuzz Testing ....................................................................................................... 6 1.2.2 Implanted Malicious Code Detection ................................................................. 7 1.3 Overview of the Dissertation..................................................................................... 8 Chapter 2: Related Works ................................................................................................... 9 2.1 Protocol Security Analysis ........................................................................................ 9 ix 2.2 API Authentication Protocols and OAuth ............................................................... 11 2.2.1 OAuth ............................................................................................................... 12 2.3 Mathematical Model Synthesis ............................................................................... 13 2.3.1 Angluin’s Learning Algorithm ......................................................................... 14 2.4 Protocol Input Format Reverse Engineering ........................................................... 15 2.5 Fuzz Testing ............................................................................................................ 16 Chapter 3: Formal Model .................................................................................................. 17 3.1 Finite State Machine................................................................................................ 17 3.2 Attacker Model ........................................................................................................ 20 Chapter 4: Security Property Analysis of Protocol Specification ..................................... 22 4.1 Model of Communication Protocols ....................................................................... 23 4.2 Authentication and Authorization Properties .......................................................... 23 4.3 Trace Inclusion Transformation .............................................................................. 26 4.4 Online Minimization and Minimal Reachable Graph ............................................. 31 4.5 Security Property Analysis - False Negatives and False Positives ......................... 36 4.5.1 False Negative .................................................................................................. 36 4.5.2 False Positive ...................................................................................................