Formal Analysis of Network Protocol Security
DISSERTATION
Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy in the Graduate School of The Ohio State University
By
Yating Hsu
Graduate Program in Computer Science and Engineering
The Ohio State University
2011
Dissertation Committee:
Dr. David Lee, Advisor, Dr. Dong Xuan, Dr. Ten H. Lai
Copyright by
Yating Hsu
2011
Abstract
As computer networks turn into an indispensable part of technology and entertainment of human life today, security and reliability become the forefront issue of network protocol design and implementation. Any security or reliability flaws in network protocol operation can lead to great loss in private data, business opportunity and reputation. As a result, methods and tools for preventing and detecting such security breach in network protocol design and implementation are urgently needed. Formal methods have proven to be successful in system verification and testing. This thesis is devoted to formally design algorithm and tools to efficiently and effectively investigate security of both protocol specification and implementation.
For protocol specification, we first model a protocol as a state machine. To verify certain security property against the state machine, we need to explore all reachable states in the state machine. However, traditional reachability problem suffers from the classic state space explosion problem. We propose two techniques to tackle this problem: trace inclusion transformation and online minimization. We first transform the original state machine to a simplified machine which is a trace inclusion transformation of the original machine by removing certain system parameters. We apply online minimization to construct a minimal reachable graph of manageable size that is bisimulation equivalent to the reachability graph of the simplified machine. Then we can verify the desired security
ii property against the minimal reachable graph. However, we may introduce false positive and false negative since we have simplified the original machine. In the second phase of analysis, we show that there is no false negative if the simplified machine is a trace inclusion transformation of the original machine. We also design algorithm to eliminate false positive. Then we can conclude if the state machine (the protocol design) is security against the checked security property.
However, even with perfect design, fault can still be introduced during the implementation and integration phase. Due to the black-box nature of protocol implementation and often unavailability of specification, traditional verification techniques do not apply here. In this thesis, we first automatically synthesize a formal model to represent the behavior of the implementation and then use the synthesized model to assist different security flaw detection applications. We study both active and passive synthesize approaches and compare their advantages and disadvantages. We incorporate the formal model to two different security flaw detection applications. In the first application, we build a model based fuzz testing framework in which the formal model is used to guide test input selection and served as test coverage criteria. Our framework significantly improves existing black-box protocol security testing techniques.
In the second application, we examine the synthesized model to determine if there is any malicious function hidden inside a protocol implementation and determine the behavior of the hidden function.
We show the proposed approaches are effective with extensive case studies for various scenarios and applications. Nevertheless, the techniques and framework proposed in this
iii thesis is generic and could be easily adapted for different problem domains and applications.
iv
Dedication
Dedicated to my parents, Mr. James Hsu and Mrs. Ming-Hsiu Chao
v
Acknowledgments
I would like to extend my gratitude to my advisor, Prof. David Lee, for his continuous guidance, tolerance and financial support during my graduate study. Appreciation also goes to Prof. Steve Lai for his help during my last year of study.
I am also deeply indebted to my colleague Dr. Guoqiang Shu and many others from the
CSE department. Their help and encouragement made my study smooth and with less detour.
Finally, I would like to thank my parents and family in Taiwan for their unconditional love and support.
vi
Vita
Aug 31, 1984 ...... Born in Taipei Taiwan
2002-2006 ...... B.S. Computer Science and Information Engineering, National Taiwan University, Taiwan
2006 to present ...... OSU Fellow, Graduate Teaching and Research Assistant, The Ohio State University
Publications
Yating Hsu and David Lee, “Machine Learning for Implanted Malicious Code Detection with Incompletely Specified System Implementations,” in Proc. 2nd International Workshop on Security and Trust in Future Internet (FIST) 2011.
Yating Hsu and David Lee, “Authentication and Authorization Protocol Security Property Analysis with Trace Inclusion Transformation and Online Minimization,” in Proc. IEEE International Conference on Network Protocols (ICNP) 2010.
Yating Hsu, Guoqiang Shu and David Lee, “A Model-based Approach to Security Flaw Detection of Network Protocol Implementation,” in Proc. IEEE International Conference on Network Protocols (ICNP) 2008.
vii
Guoqiang Shu, Yating Hsu and David Lee, “Detecting Communication Protocol Security Flaws by Formal Fuzz Testing and Machine Learning,” in Proc. 28th IFIP International Conference on Formal Techniques for Networked and Distributed Systems (FORTE), 2008.
Fields of Study
Major Field: Computer Science and Engineering
viii
Table of Contents
Abstract ...... ii
Dedication ...... v
Acknowledgments...... vi
Vita ...... vii
List of Tables ...... xiii
List of Figures ...... xiv
Chapter 1: Introduction ...... 1
1.1 Security Analysis of Protocol Specifications ...... 2
1.1.1 Authentication and Authorization ...... 4
1.2 Security Flaw Detection of Protocol Implementations ...... 4
1.2.1 Fuzz Testing ...... 6
1.2.2 Implanted Malicious Code Detection ...... 7
1.3 Overview of the Dissertation...... 8
Chapter 2: Related Works ...... 9
2.1 Protocol Security Analysis ...... 9
ix
2.2 API Authentication Protocols and OAuth ...... 11
2.2.1 OAuth ...... 12
2.3 Mathematical Model Synthesis ...... 13
2.3.1 Angluin’s Learning Algorithm ...... 14
2.4 Protocol Input Format Reverse Engineering ...... 15
2.5 Fuzz Testing ...... 16
Chapter 3: Formal Model ...... 17
3.1 Finite State Machine...... 17
3.2 Attacker Model ...... 20
Chapter 4: Security Property Analysis of Protocol Specification ...... 22
4.1 Model of Communication Protocols ...... 23
4.2 Authentication and Authorization Properties ...... 23
4.3 Trace Inclusion Transformation ...... 26
4.4 Online Minimization and Minimal Reachable Graph ...... 31
4.5 Security Property Analysis - False Negatives and False Positives ...... 36
4.5.1 False Negative ...... 36
4.5.2 False Positive ...... 38
4.5.3 Simultaneous Execution Traces ...... 40
4.6 Summary of Our Approach ...... 40
x
4.7 Case Study: OAuth ...... 41
4.7.1 A Formal Model of OAuth ...... 42
4.7.2 Security Properties of OAuth ...... 44
4.7.3 Trace Inclusion Transformation ...... 46
4.7.4 Minimal Reachable Graph ...... 47
4.7.5 Results ...... 47
4.7.6 Insecure Interfaces ...... 48
Chapter 5: Reliability and Security Testing of Protocol Implementation ...... 50
5.1 Protocol Implementation Model Synthesis: Active Learning: Lisfsm* ...... 50
5.1.1 Observation Table ...... 52
5.1.2 Supervised Learning Algorithm Lisfsm* ...... 57
5.2 Protocol Implementation Model Synthesis: Passive Synthesis with Partial FSM
Reduction ...... 60
5.2.1 Loop Free Tree FSM Reduction Algorithm ...... 64
5.2.2 A Heuristic Reduction Algorithm ...... 69
5.3 Comparisons of Active and Passive Synthesis ...... 74
5.4 Case Study 1: Model Based Fuzz Testing ...... 76
5.4.1 Overview of Our Tool ...... 76
5.4.2 MSN Instant Messaging Protocol ...... 78
xi
5.4.3 Fuzz Testing Setup ...... 79
5.4.4 Synthesized FSM Model ...... 81
5.4.5 Fuzzing Goal and Operators ...... 82
5.4.6 Result and Analysis ...... 84
5.5 Case Study 2: Implanted Malicious Code Detection ...... 90
5.5.1 UnrealIRCd ...... 92
5.5.2 MSN Client with Message flooder ...... 94
Chapter 6: Conclusion...... 96
References ...... 98
xii
List of Tables
Table 5.1 Partial Assignments by Algorithm 5.4 ...... 73
Table 5.2 Constraints ...... 73
xiii
List of Figures
Figure 3.1. An Attacher Machine MA ...... 21
Figure 4.1. EFSMs of OAuth 1.0 (a) Service Provider (b) Consumer and (c) User ...... 42
Figure 5.1. Example of Algorithm 5.3 ...... 68
Figure 5.2. Example of Algorithm 5.4 ...... 73
Figure 5.3. Workflow of Model Based Fuzz Testing ...... 77
Figure 5.4. Software Architecture of a MSN Client Fuzz Testing Tool ...... 80
Figure 5.5. Synthesize FSM Model of MSNIM Protocol Client ...... 81
Figure 5.6. Number of Crash Instances from Fuzzing ...... 85
Figure 5.7. Number of Crash Instances and Transition Coverage ...... 86
Figure 5.8. Maximum, Expected and Actual Number of Crash Instances in Gaim
(Windows) under Complete Message Type Coverage ...... 89
Figure 5.9. Probability of Detecting Maximal Number of Crash Instance from Message
Coverage ...... 89
Figure 5.10. ISFSM model for IRC server (a) with backdoor (b) without backdoor...... 93
Figure 5.11. ISFSM model for the MSN client (a) with message flooder, (b) without message flooder...... 95
xiv
CHAPTER 1: INTRODUCTION
With the surge advancement of Internet technology and prosperous online business opportunities, reliability and security has become one of the main goals of network protocol design and implementation. Any system failure or security flaw could lead to significant financial or credibility loses. According to US-CERT [63] in 2006 alone a total of 600,000 messages reported security incidents. Among the 2,453 computer vulnerabilities in the database, around 1,000 are related to network protocols.
It is of great importance and also a vital challenge to develop methods and tools for detecting security and reliability flaws in protocol design and implementation. Formal methods have been applied to various applications and problem domains to rigorously investigate system properties. In this thesis, we want to apply formal methods to design methods and tools for security analysis of both protocol specification and implementation.
1
1.1 SECURITY ANALYSIS OF PROTOCOL SPECIFICATIONS
With applications and services moving from local host to the Internet, flawless design of network protocols is crucial to the correct function of the Internet and the development of online business. At the same time, many online applications and business models that have never been seen before are launched at an unprecedented rate. Many of the new protocols are proposed and deployed without first carefully examining its security. OAuth
[50] protocol released its first version before a major security flaws was discovered thirteen months later. Researchers discovered security vulnerability in Google’s
ClientLogin Protocol [57] that is primarily used on Android but which also exists for any apps and desktop applications. The vulnerability in Google’s ClientLogin Protocol would allow attackers to eavesdrop any transmitted data.
With security becoming one of the foremost priorities and concerns of developers, it is of great interest to develop efficient algorithms and tools to verify desirable security properties against a protocol design. There are many different desirable security properties specified either explicitly or implicitly in the protocol design. In this thesis, we focus on temporal properties, specifically authentication and authorization. To verify such temporal property, one needs to check all possible execution sequences of the protocol.
It is commonly acknowledged that the design and analysis of security protocols are notoriously difficult and prone to errors which are extremely hard to detect [44]. A major hurdle of formal analysis of protocol security properties is the well-known state explosion
2
– a protocol system usually contains infinitely many or a formidable number of states.
Therefore, most of the analysis resorts to heuristics, such as state space pruning, limiting the number of protocol runs, or ignoring certain variables and parameters. In this thesis, we propose two efficient algorithms to significantly cut down the state space while still preserving all the protocol execution traces: trace inclusion transformation and online minimization [29]. Specifically, given a transition system M with a set of states and transitions among the states, we want to verify if M satisfies certain temporal property.
We first conduct a trace inclusion transform on M by simplifying certain system parameters and variables and obtain that preserves all the execution traces of M.
However, the state space of may still be infinite or too large to construct. We further apply online minimization to reduce the state space to a manageable size by merging states, which are bisimulation equivalent, and hence the execution traces of the original transition system are still preserved.
However, the two state space reduction procedures may result in false negatives and false positives. We show that our trace inclusion transformation and online minimization do not introduce any false negative and we design an efficient algorithm for ruling out all of the possible false positives. If there is any execution trace that leads to the violation of the checked property after we eliminate false positive, then one can conclude there is a security breach in the protocol design. Otherwise, the protocol design is secure against the checked property.
3
1.1.1 Authentication and Authorization
Some protocols have explicit security property requirements, such as authentication, authorization, secrecy or anonymity. On the other hand, there are security properties that are desirable or substantial to the correct functioning of the protocol systems even though they are not explicitly stated in the protocol specification, such fairness, availability or resilience to deny of service attack. In this thesis, we focus on authentication and authorization properties.
Authentication is the act or service to provide confidence the identity of the communication peer. Authorization is the ability to limit or grant access right of certain resource to an entity. We formally model authentication and authorization as temporal property and verify the temporal statement against a formal protocol model.
Our approach for formal analysis of protocol security properties is rather general and is applicable to a range of properties, including authentication and authorization, so long as they can be formulated as a temporal property.
1.2 SECURITY FLAW DETECTION OF PROTOCOL IMPLEMENTATIONS
Even after extensively study of protocol specification to discover problems in the design, flaws may still be introduced to the implementation during the phase of coding, deployment and integration due to discrepancy between specification and implementation 4 or ambiguity in the specification. for example, Twitter’s unfaithful implementation of
OAuth protocol has led to compromise of consumer secret and key [54].
CSI/FBI reported [24] that among 597 companies surveyed 66% of them are currently using penetration testing techniques and automated tools with the hope of finding security flaws early. However, the prevalent security testing tool for detecting protocol implementation faults is rather limited due to the following two major hurdles:
Black-box Implementation: The black-box nature of protocol implementation poses a challenge. Software security testing methods [22] [23] often look for suspicious operations in source code and derive input data to reach them. However, for protocol implementations usually only very limited knowledge is available especially at post deployment and integration stage.
Lack of Formal Specification A complete and machine understandable protocol specification is very rarely available, which makes formal protocol testing [38] [40] infeasible. Even when a specification is manually derived there is no guarantee that the corresponding implementation conforms to the specification since it might contain unspecified engineering details [61] or discrepancy from system evolvement.
Due to the two challenges above, the first step in the proposed approach for protocol implementation flaw detection is to construct a formal behavior model of the protocol implementation under test. Synthesized mathematical models have been used by many researchers. This thesis studies both active and passive approaches. The synthesized model is then used for different fault detection applications. In this thesis, two different
5 fault detection problems are studied: fuzz testing [30][60] and implanted malicious code detection.
1.2.1 Fuzz Testing
A major cause of flaws in protocol implementation is improper handling (e.g. incorrect assumption) of input data. The resulting security exploits range from the well understood buffer overflow attack to more sophisticated protocol level attacks such as enforcing a lower (unsafe) version of handshaking protocol during version negotiation and soliciting confidential content using illegitimate queries.
Fuzz testing is an efficient testing technique working by mutating a portion of the normal input data at the ingress interface of a protocol component in order to reveal unwanted behaviors [51]. It becomes increasingly popular in recent years as a means of security flaw detection due to extremely low cost and proven effectiveness. However, existing black-box protocol fuzzing tools are designed in a protocol specific and ad-hoc manner. The selection of input messages from a session to mutate is done either randomly or manually. The unstructured selection of input makes it impossible to measure the comprehensiveness (i.e. how much is covered) of a test suite, and relying on human interfaces usually makes such testing tools hard to automate.
In this thesis, we propose model-based fuzz testing [30]. Our approach first synthesizes an abstract behavioral model from the protocol implementation under test and then uses it 6 to guide the fuzz testing process, including test input selection and test coverage measurement. We show that our tool for security flaw detection of protocol implementation is protocol independent and provides a high fault coverage, measurability, and automation. Our experiment results also show that our approach is more effective than the existing syntax based security testing methods.
1.2.2 Implanted Malicious Code Detection
UnrealIRCd [64] is an open source IRC (Internet Relay Chat) server. In November
2009, UnrealIRCd 3.2.8.1 was replaced by a new version with a backdoor in it at some of its mirror sites. The backdoor allows a remote attacker to execute arbitrary system commands with the privileges of the user running the IRC server. The malicious code was not detected until seven months later. In June 2011, vsftpd [67], which is considered as one of the most secure and fastest FTP (File Transfer Protocol) servers for Unix-like systems, version 2.3.4 on the master site was replaced with a compromised version with a backdoor. The backdoor in vsftpd listens on port 6200 and launches a TCP callback shell when a user logs in with the name “:)”.
It is of vital importance and also a challenge to detect implanted malicious code in newly developed systems before their deployment. Before we deploy any protocol implementations, we want to examine not only if there is any malicious code hidden in it but also what the purpose and function of the malicious behavior are. 7
In this thesis, given the protocol implementation (possibly) with implanted malicious code, we first synthesize a model that describes the behavior of the protocol implementation. Then examining the synthesized model may reveal the hidden malicious function, which should be part of the whole behavior model.
1.3 OVERVIEW OF THE DISSERTATION
Chapter 2 summarizes the related works. Chapter 3 introduces the formal model for representing protocol specification and implementation. Chapter 4 discusses security property verification against protocol specification with OAuth as a case study in 4.7.
Chapter 5 is devoted to the problem of security flaw detection of protocol implementation, including active and passive protocol model synthesis in 5.1 and 5.2, and the application of the synthesized model to two specific problems: model-based fuzz testing in 5.4 and implanted malicious code detection in 5.5. Chapter 6 concludes the thesis.
8
CHAPTER 2: RELATED WORKS
2.1 PROTOCOL SECURITY ANALYSIS
It is commonly acknowledged that the design and analysis of security protocols are notoriously difficult and prone to errors that are extremely hard to detect [44]. A lot of research efforts have been devoted to formally analyzing security protocols. Various theorem proving [10][48] and state space based [35] approaches were developed to facilitate analysis of security protocols. Theorem proving is based on formulating the property as a mathematical theorem which would be proved with invariants and variants
[66]. Works in [11][14] use language MSR to formally specify Kerberos as first order atomic formulas and verify various security properties. However, theorem proving approaches are often with daunting and tedious syntax and rules. Manual proving of theorems requires highly skilled person and is prone to human errors [66]. Tools are developed to help automate this process; however lots of human assistance is often needed and the invariant and variant must be provided by human. Thus such tool is seldom fully automated.
State space methods aim at more automatic analysis and verification. Lowe [44] models Needham-Schroeder protocol with CSP and FDR along with a model checker for
9 finding an attack in the protocol. Mur [46][47] is a protocol verification tool that checks a specification against a desired property by explicit state enumeration and is thus hampered by the well known state space explosion [46][47] problem. Interrogator [35] and NRL Protocol Analyzer [35] use backward search to avoid explicitly constructing the whole state space.
State space approaches suffer from one big and fundamental problem: state explosion
[40]. The number of states tends to grow exponentially in the parameters of the system
[66]. State minimization is a natural approach to deal with state space explosion.
However, one cannot afford to construct the whole state space of a transition system and then minimize it. Online minimization [39][69][8][9][16] is to perform reachability analysis and minimization at the same time without constructing the whole state space.
Protocol model simplification and transformation have also been studied. Work in [31] defines the notion of safe transformation on CSP protocol models that has the property to preserve insecurity after transformation. [45] provides sufficient conditions under which checking simplified systems is enough. Their transformation is applicable to CSP models whereas our transformation and minimization are on general transition systems with protocol data portions. Data independence techniques are explored [58] to find
“threshold” for variables in a protocol model to simplify the analysis.
10
2.2 API AUTHENTICATION PROTOCOLS AND OAUTH
With the advancement of Internet technologies and business opportunities web applications have led to unprecedented integrated services. A web application may create user specific services by acquiring user data from an external data server. For the web application to access user data at the data server, it requires the web application to sign in at the data server with user’s login credential, usually username and password.
Mashup [26][37] is designed to meet this goal. The applications require that user reveals his credentials. Various frameworks, such as MashupOS [28], SMash [36] and
OMash [17], have been proposed to address the security issues of Mashup.
However, it is always undesirable for the user to reveal his username and password.
From this and other applications’ needs, API (Application Programming Interface) authentication and authorization protocols are developed for web application to obtain user data without user credential.
Various vendors offer their own API authentication and authorization protocols, such as AOL OpenAuth [53], Yahoo! BBAuth [6], and Flickr Authentication API [21]. OAuth
[50] is an effort for the standardization of the prevalent API authentication protocols.
11
2.2.1 OAuth
OAuth is an API authentication protocol for a standardization effort. There are three communicating entities in OAuth: Service Provider, Consumer and User. Service
Provider hosts user data, and consumer is the web application that needs access to user data. The basic idea of API authentication and authorization protocols is to use a data structure, called token, that decouples the access right from user login credential [50][49].
The goal of OAuth is to let User delegate access right at a Service Provider to a web or desktop application without revealing User’s login credential at the Service Provider. The workflow of OAuth is [50]:
1. Consumer, upon receiving the request from the User, sends a request to Service
Provider’s Request Token URL to ask for a Request Token.
2. Service Provider grants Consumer an Unauthorized Request Token.
3. Consumer redirects User to Service Provider’s User Authorization URL.
4. Service Provider authenticates User and obtains consent (authorization) from
User.
5. After User authorizes Consumer’s Request Token, Service Provider redirects User
back to Consumer’s callback URL.
6. Consumer requests Service Provider for an exchange of the Authorized Request
Token for an Access Token.
7. Service Provider grants Consumer an Access Token.
8. Consumer uses the Access Token to access data on behalf of User.
12
It is an authentication and authorization process; Service Provider authenticates the identity of User and Consumer, and User authorizes Consumer with the access right.
After the publication of first version of OAuth v1.0, a Session Fixation attack was discovered [50][49] and then fixed in the second version of OAuth v1.0A. The basic workflow of the protocol remains the same. The main differences are as follows. In
OAuth 1.0, after User authorization, Service Provider redirects User back to Consumer.
In OAuth 1.0A, when Service Provider redirects User back to Consumer, Service
Provider issues a verifier to User. When User is redirected back to Consumer, Consumer obtains the verifier from User. Then Consumer must use the Authorized Request Token and also the verifier to exchange for an Access Token. It motivates our work to study security properties of both OAuth versions.
2.3 MATHEMATICAL MODEL SYNTHESIS
Synthesized mathematical models of protocol implementation have been used by many researchers. In [7] the authors proposed to use predicate formula to characterize implementations and to further use it for error detection and protocol fingerprinting.
Our algorithm for tree FSM reduction in Chapter 5.2 is based on the classical results described in [5][25][52].
The classic Angluin’s supervised automata learning algorithm is briefly summarized in the next section. 13
2.3.1 Angluin’s Learning Algorithm
The active synthesis approach uses the theory of machine learning [3] and black-box checking [55]. Angluin’s L* [3] is the classic supervised automata learning procedure.
Lfsm* [61] extends L* for learning completely specified Finite State Machine.
L* maintains an Observation Table (S, E, T) to represent the current conjecture of the unknown automata. S and E are nonempty finite prefix-closed set and nonempty finite suffix-closed set. T is a finite function that maps inputs to outputs. Each row of the
Observation Table labeled by elements of (S (S A)) is a sequence that leads the machine to that state, where A is the set of input symbols of the target automata. Each column is labeled by elements of E. At the beginning, S and E are both initialized to empty sets. L* assumes there is a minimal adequate Teacher who is able to answer the following two types of queries: (1) Membership query, yes or no on whether a string t is a member of the unknown regular set. (2) Counter example. The Teacher generates a string t for which the current conjecture produces different output than the target. The counter example is then used to correct and calculate the next conjecture until the Teacher confirms the conjecture is identical to the target. Refer to [3] for the detail of L*.
Lfsm* also maintains an observation table (S, E, T) to represent the current conjectured
FSM [40] where S and E represent the candidate states and separating sequences [40] of the current conjectured machine. At the beginning, S and E are both initialized to empty 14 sets. In practice, a minimal adequate Teacher does not exist and Lfsm* simulates the
Teacher with conformance testing [40][61].
2.4 PROTOCOL INPUT FORMAT REVERSE ENGINEERING
Tools to reverse engineer input format of protocols have been developed and proposed in the literature. Many of the tools rely on the insight that dynamically monitoring the execution of the program and analyzing how the program processes the received messages reveal wealth of information about the input format [68]. Polyglot [13] emphasizes on finding boundaries between message fields. AutoFormat [41] uncovers cross-filed relationship. REWARDS [42] indentifies data structures and their syntax and semantics through on-line type propagation and off-line type resolution. Tupni [19] reverse engineers the input sequence, message types and input constraints. Dispatcher
[12] focuses on understanding the command and control protocol used in botnet. [43] generates evidence for security vulnerability by dynamically associates execution point with relevant input. Works such as [18] and [41] aimed at reverse engineering of protocol message format using a variety of learning mechanisms. These results to a large extent enable us to reverse engineer the input format of a black-box as the input of our machine learning algorithm and to develop protocol message encoder and decoders.
15
2.5 FUZZ TESTING
The idea of using input fuzzing to detect flaws of communicating systems has been investigated for more than 20 years [4] and white-box approaches have been predominant. Many advanced theory and techniques such as symbolic execution [22][23] and binary interception [32] have been applied to find input data causing failures.
Existing works on black-box fuzzing are mostly ad-hoc and each is coupled with a specific protocol. The PROTO project [33] manually developed fuzz test suites for many popular Internet protocols for an evaluation. In recent years there is a number of open source network fuzzing tools, covering a wide spectrum of protocols, including TCP/IP,
HTTP, P2P and many others. Most of these tools hardcoded knowledge of target protocols and provided limited supports for automatic input selection.
16
CHAPTER 3: FORMAL MODEL
In the chapter, we introduce the state machine models used in the thesis.
3.1 FINITE STATE MACHINE
Finite State Machines (FSM) [40] have been widely used to model systems in diverse areas, including sequential circuits, lexical analysis and communication protocols. A
Finite State Machine consists of a finite number of states and produce outputs on their state transitions after receiving inputs.
Definition 3.1: A finite state machine (FSM) M is a quintuple [40]
M=(S, I, O, , ), where S is the set of states, I and O are input and output alphabet respectively. : S I→S is the transition function and : S I→O is the output function.
When the machine M is in a current state s in S and receives an input a from I, it moves to the next state specified by (s, a) and produces an output given by (s, a).
17
An FSM can be represented as a state transition diagram [40], a directed graph whose vertices correspond to the states of the machine and the edges correspond to the state transitions. We can an FSM a tree FSM if its state transition diagram is a tree.
A Finite State Machine is fully specified [40] if at each state upon any input there is a next state specified by state transition function and a specified output by the output function . Otherwise, the machine is an incomplete specified Finite State Machine
(ISFSM); at certain state with some input, the next state or output are not specified. Also the machine is deterministic [40] if at each state and upon an input, the machine follows a unique transition to a next state. Otherwise, the machine is nondeterministic. In this thesis, we only consider deterministic machines.
We can extend the transition function and output function from an input symbol to an input sequence [40]: for a given state s1, an input sequence x=a1a2… ak takes the machine successively to states si+1 = (si, ai), i = 1, 2, …, k, with the final state (s1, x) = sk+1, and produces an output sequence (s1, x) = b1b2…bk where bi = (si, ai), i = 1, 2, …, k.
Two states si and sj are equivalent [40] if and only if for every input sequence the machine will produce the same output sequence regardless of whether si and sj is the initial state. Two machines M and M’ are equivalent if and only for every state in M there is a corresponding equivalent state in M’, and vice versa.
However, in practice, protocols and systems usually have variables and operations based on the variable values. FSM is not powerful enough to model such systems in a succinct way. Extended Finite State Machine (EFSM) is FSM extended with variables.
18
Definition 3.2: An extended finite state machine M is a quintuple [40]
M = (I, O, S, , T), where I, O, S, , and T are the finite sets of input symbols, output symbols, states, variables, and transitions, respectively. Each transition t in T is a six-tuple
t = ( ) where , , and are the start (current) state, end (next) state, input and output, respectively. is a predicate on the current variable values, and defines an action on variable values. At any state s, the machine follows a transition t=(s, q, a, o, P,
A) upon input a if is valid for P: P( )=TRUE. In this case, the machine outputs o, changes the current variable values by action =A( ), and move to state q.
In an EFSM, each combination of a state and variable values is called a configuration
[40]. Each EFSM has an equivalent FSM with configurations as states, called its reachability graph. However, for practical protocol systems, the size of equivalent FSM may be huge or even infinite due to (1) the number of variables, (2) the domain of variables, and (3) the number of components in the system (number of EFSM’s).
Therefore, it is impossible to construct such FSM. It is the well-known state explosion problem [40][66].
A protocol or system often consists of multiple processes or entities. We model a protocol system among multiple entities as a collection of Communicating Finite State
Machines (CFSM’s) [40] M=(M1, M2, …, Mn). Each FSM Mi is a component machine. To
19 specify the input and output actions among component machines, we use the notion from
CSP [27]. A message msg sent from Mi to Mj is an output operation Mj!msg in Mi and a matching input operation Mi?msg in Mj. The matching operations are synchronized between Mi and Mj. Similarly, one can define communicating extended finite state machine (CEFSM) where each component machine is an EFSM.
3.2 ATTACKER MODEL
To analysis security properties against a protocol or system, we need to consider the participation of an attacker in a protocol session. We follow the Dolev-Yao [20], full knowledge [47] attacker model. The attacker could be a legitimate participant in a protocol session. It can initiate a protocol session with another peer, and it can also accept communication from another peer. Furthermore, the attacker is capable of intercepting massages from any communication channels. The intercepted messages become the knowledge of the attacker. As a result, the attacker can make up any message using its knowledge as the source to insert message fields. However, the attacker is limited by cryptographic constraint: it can’t decrypt cyphertext without the corresponding key. The attacker is modeled by a single-state EFSM MA, as in Figure 3.1.
20
Figure 3.1. An Attacher Machine MA
The variable φ represents its knowledge that contains a set of entries, which the attacker can insert in a message field [46]. It is initialized as φ0 which includes the information about the protocol that an attacker can obtain without actually executing the protocol, such as the public key of a server or the public keys of other communicating parties. With different protocol and application constraints the attacker’s knowledge can be refined or limited accordingly. For example, if a protocol or an application scenario assumes secure communication channels, then no message can be intercepted or modified by an attacker.
21
CHAPTER 4: SECURITY PROPERTY ANALYSIS OF PROTOCOL SPECIFICATION
Given a protocol specification, we want to determine if the protocol specification satisfies certain security properties, such as authentication, authorization, secrecy, anonymity and etc. We model the protocol specification as Communicating Extended
Finite State Machines (CEFSM) introduced in Chapter 2. The security property can be modeled as a Boolean statement. In this thesis, we focus on authentication and authorization properties. We want to determine if the formal model CEFSM satisfies the desired security property.
A major hurdle of formal analysis of protocol security properties is the well-known state explosion problem. Given the temporal property of authentication and authorization, we introduce trace inclusion transformation and online minimization for obtaining a model of a manageable size for a formal and rigorous analysis. Finally, we show that our state space reduction methods do not introduce any false negatives, and we can eliminate false positives.
For a case study, we analyze OAuth, a standardization of API authentication protocols.
Our automated analysis identifies a number of attacks in the original specification.
22
4.1 MODEL OF COMMUNICATION PROTOCOLS
We model a communication protocol system as a collection of communicating extended finite state machines (CEFSM) [40]. A protocol system with n principals can be modeled as a CEFSM M=(M1, M2, …, Mn, MA) with n+1 component machines where each component machine Mi, 1 i n, is a communicating principal specified in the protocol and MA is an attacker. Each component machine is an Extended Finite State
Machine (EFSM).
In each run of a protocol, an attacker may take actions of other participants. For example, in Needham-Schroeder authentication protocol, an attacker can take the actions of the initiator or the responder. Similarly, in OAuth, an attacker can take the role of: (1) both User and Consumer; (2) User but not Consumer; (3) Consumer but not User; (4)
Neither User nor Consumer.
Given a formal model M of a protocol, we want to verify if M satisfies certain authentication or authorization property.
4.2 AUTHENTICATION AND AUTHORIZATION PROPERTIES
An authentication or authorization property can often be formulated as a temporal property: one type of action e (authentication and authorization) should precede another action d (private data access) [59][62]. For instance, OAuth authorization can be stated 23 as: action e of User authorization must precede action d of Consumer data access.
Equivalently, if transition e is removed from the system, then transition d is no longer reachable from the initial system state. To verify such property, we want to check that for every execution trace in M, action (transition) d is always preceded by action (transition) e. Naturally, such property analysis can be reduced to a reachability analysis of M: Is any transition d reachable from the initial state without going through any transition e? Or, equivalently, if we remove transition e, is transition d still reachable from the initial state?
Given the predicates and actions on the transitions, to check the reachability in a
CEFSM is rather complex [40]. A common approach is to construct a reachability graph from the CEFSM for an analysis. A reachability graph consists of all the reachable configurations of the CEFSM, each of which consists of a global state (states of all component machines) and a set of variable values (variables of all component machines).
Initially, the protocol is in an initial global state where each component is in its initial state, along with a set of initial variable values. This is called the initial configuration of a protocol system. We want to construct a reachability graph from M that consists of all the configurations, which are reachable from the initial configuration. Note that a transition in a reachability graph does not contain any predicate or action on variable values anymore; the information is contained in the current and next configuration of a transition. Given the reachability graph, we can verify a temporal property as follows. We remove all the transitions which are derived from (authorization) action e, and check whether any transition from (data access) action d is still reachable from the initial
24 configuration. This is the case if and only if the temporal property is violated. It can be easily checked by a graph search.
However, the reachability graph can be infinite or of a formidable size due to the well- known state explosion problem. Furthermore, a protocol system can have several sessions
(runs), which are simultaneously executed. This can be modeled by a Cartesian product of protocol machines and that further explodes the state space. Therefore, we need to effectively reduce the state space first.
In this paper we propose two approaches to reduce the size of the reachability graph to tackle the state explosion problem: (1) Trace Inclusion Transformation. It simplifies the input, output, predicate and action on transitions of CEFSM while all the executable traces in the original CEFSM are preserved in the transformed machine. (2) Online
Minimization. It merges configurations together according to bisimulation equivalence yet without first constructing a reachability graph. After the state space reduction, we analyze the reduced protocol system model and prove security properties.
Note that our approach for formal analysis of protocol security properties is rather general and is applicable to a range of properties, including authorization, so long as they can be formulated as a temporal property.
25
4.3 TRACE INCLUSION TRANSFORMATION
Informally, an attack of a protocol machine M is an execution trace from the initial state to a certain state where the security property is violated.
Definition 4.1. A trace v0,t0,1,v1,t1,2,…tn-1,n,vn consists of a sequence of states v0, v1, …, vn and transitions t0,1, t1,2,…tn-1,n in M where transition ti,i+1 moves from state vi to vi+1. A trace from the initial configuration with initial state v0 and initial variable value is executable if and only if for all i: 0 i n-1, the predicate of transition ti,i+1 on the current variable values is true where the action of the transition ti,i+1 updates the variable values from to .
Definition 4.2. An attack trace in M is an executable trace from the initial configuration to a state with a security property violation.
For instance, in OAuth if there is an executable trace from the initial state to a state of
User’s data access by Consumer without going through User authorization transition, then it is a security property (authorization) violation and hence is an attack trace.
Given the complexity of real application protocols, we want to first simplify it to reduce the state space to make it feasible to identify attack traces. For such simplification, we have to preserve all the executable traces, including the attack traces which are to be
26 detected later. Such desirable simplification is called trace inclusion transformation. We first describe a special yet important case of trace inclusion transformation.
Message content adds complexity to formal protocol analysis. For instance, a message may contain session related fields, such as source and destination IP address, nonce and timestamp, whose values may not affect the result of security property analysis. We want to abstract out such system parameters, message fields and variables without losing any executable traces – a special case of trace inclusion transformation.
We define abstraction functions α: I→α(I) and β: O→β(O) that remove irrelevant message fields from the input and output of transitions. We also define a function ω: →
ω( ) that is a transformation of variables. By transformations of α, β and ω, we transform a CEFSM M=(M1, M2, …, Mn, MA) to CEFSM =( , , …, , MA), where EFSM
Mi=(I, O, Si, , Ti), 1 i n, is transformed to
=(α(I), β(O), Si, ω( ), ).
Note that Mi and have the same states and transitions, but the input and output on transition are simplified according to α and β, and the predicate and action are changed by ω. A transition t=( ) Ti is transformed to :
= , and γ: → and δ: → are the derived transformations from ω.
Given an EFSM, its transition diagram is a graph where the nodes are the states and the edges are the transitions. A transition diagram models the control portion of an EFSM.
Obviously, with the above transformations on input, output and variables, the transition diagram of EFSM Mi=(I, O, Si, , Ti) and =(α(I), β(O), Si, ω( ), ) are isomorphic. 27
Definition 4.3. Two EFSMs E and are isomorphic in control portion if their transition diagrams are isomorphic.
It can be easily shown:
Proposition 4.1. Given CEFSMs M=(M1, M2, …, Mn) and transformations of input, output, variable, predicate and action on its component EFSMs, the corresponding transformed CEFSM =( , , …, ) is isomorphic in control portion to M if each component EFSM is isomorphic to Mi in control portion, 1 i n.
It is crucial that after the transformations the executable traces of the simplified
CEFSM include all of the executable traces of the original machine M so that the transformations preserve all the attack traces to be analyzed. Recall that if EFSM and E are isomorphic in control portion after transformations, they have isomorphic transition diagram. Since a trace is specified by a sequence of interleaved states and transitions, each trace in E has a unique corresponding trace in .
Definition 4.4. An EFSM is a trace inclusion transformation from EFSM E if: (1) and E are isomorphic in control portion with the transformations on input, output, variables, predicate and action; and (2) For each executable trace of E, the corresponding transformed trace in is also executable.
28
After a trace inclusion transformation from EFSM E to each executable trace in E is also an executable trace in and hence the name: trace inclusion transformation. Since a transition in a CEFSM is executable if all the associated transitions in the component machines are executable, we have:
Proposition 4.2. With trace inclusion transformations on each component machine of
CEFSM M=(M1, M2, …, Mn, MA) from Mi to , 1 i n, the resulting CEFSM =( ,
, …, , MA) is also a trace inclusion transformation from M.
There are various transformations which are trace inclusive. We discuss a special and practically important case where the transformation ω*( ) on variables is a projection to a subset of variables of and the predicate on ω*( ) in the original model is independent of those variables not in ω*( ). Therefore, can be decomposed as:
= where is the logical “and” operator. A useful transformation is to disregard the clause
and keep action unchanged – an identity mapping. The derived transformations are:
γ*: →
δ*: → (4.1)
29
Proposition 4.3. The mapping functions of input, output, variables, predicates and actions in (4.1) provide a trace inclusion transformation of CEFSM M=( M1, M2, …, Mn,
MA) to =( , , …, , , , MA).
Proof. We want to show that for any executable trace c in M the corresponding trace in
, with the transformed input, output, variables, predicate and action, is also executable.
We prove by induction on the length l of trace c.
Let l=1. Suppose the initial variable values of and ω*( ) are and , respectively.
Since the length of c is 1, c starts at the initial state v0, follows a transition t=(v0, v1, at, ot,
, ) and moves to v1. Therefore, and must be both true. Since
the corresponding predicate on is and hence is also true, and is executable.
Since δ* is an identity mapping, the variable values in the next state v1 after transition c and are identical.
Inductively, for all executable trace c of length l in M that goes through the states v0, v1, …, vl, there is an executable trace of length l in that also goes through the states v0, v1, …, vl in the same order. Also, the variable values of ω*( ) at vl in M are the same at that at vl in . We show that for any trace of length l+1, the statement still holds.
For an executable trace c of length l+1 in M that goes through the states v0, v1, …, vl, vl+1, the subtrace c’ of length l going through v0, v1, …, vl is also an executable trace in
M. By the inductive hypothesis, there is an executable trace in of length l and going through states v0, v1, …, vl, and the end variable values are the same as that in M after c’. When the trace c’ extends from vl to vl+1 through transition t, the predicate
30
= is true, and hence must be also true. Therefore, the corresponding trace in of length l+1 is also executable on the last transition t with the same end variable values, since the action on transition t is an identity mapping.
How to determine the transformations α, β, ω, γ and δ? The ground rule is: all the executable traces are preserved after the transformations. Further, we want to simplify a transition system as much as we can. For instance, we can abstract out all the parameters and variables so long as they are irrelevant to the security properties under consideration.
Given the simplified machine , however, the reachability graph of may still be too large or even infinite. In the next section, we apply online minimization to build a minimal reachable graph G* that is bisimulation equivalent to the reachability graph of and preserves the executable and hence attack traces for an analysis.
4.4 ONLINE MINIMIZATION AND MINIMAL REACHABLE GRAPH
Given a protocol system modeled as a CEFSM M=(M1, M2, …, Mn, MA), we first make a trace inclusion transformation on M and obtain a simplified machine . However, its reachability graph may still be infinite or too large to analyze and we want to minimize it. However, there is no way to construct the reachability graph and then merge equivalent states for minimization. We apply an online minimization procedure [39], that
31 is, we construct a minimized reachability graph G*, called minimal reachable graph, directly from without first constructing the reachability graph . Here the state equivalence is bisimulation equivalence [34][62] so that trace information is preserved.
Therefore, an analysis on the minimal reachable graph G* is equivalent to that on the reachability graph . It is crucial since our security property analysis is based on protocol and attack traces.
A theory and general procedure on online minimization were published in [39] and algorithms for real-time transition systems were further developed in [69]. Here we develop online minimization algorithms of CEFSM for our applications.
Given a CEFSM , instead of exploring all possible configurations, we divide the configurations into blocks so that configurations in a same block undergo a same sequence of state transitions in . Initially, we start with a set of blocks where each
(global control) state of and all possible variable values forming a block (a block of configurations). Then we split blocks according to unstable transitions, that is, separate configurations to different blocks, which are bisimulation inequivalent. Specifically, a transition t from a block B to C is stable if every configuration in B is mapped by transition t into C. Otherwise, it is unstable, and we split the domain block B for stabilization. We repeat this process on all blocks that are reachable from the initial configuration until all transitions are stable. All the reachable blocks along with the stable transitions among them form a minimal reachable graph G* that is bisimulation equivalent to the reachability graph of .
32
Algorithm 4.1 (Minimal Reachable Graph for CEFSM )
Input: CEFSM =( , , …, , MA), s0=initial control states of , v0=initial variable values, B0=all possible variable values.
Output: a minimal reachable graph G*. begin
1. SG={
2. mark
3. stack={
4. queue= ; /* unstable blocks to be split */
5. while (stack )
6. while (stack )
7.
8. v = marked variable values in B;
9. for each outgoing transition tr:
10. if (tr is not stable and
11. queue.enqueue(
12. if (
13. v'= variable values after tr from v;
14. mark
15. stack.push(
16. while (queue )
33
17.
18. for each outgoing tr from
19. split
20. replace
21. for each incoming tr:
22. if (tr is not stable and
23. queue.enqueue(
24. if (
25. delete tr;
26. if (
27. add transition
28. if (
29. v’’= variable values reachable from marked variable values of C
30. mark
31. stack.push(
32. if (stack )
33. break;
34. remove unmarked blocks in SG;
35. G*=SG and transitions among marked configurations
End
34
In Algorithm 4.1, the first while loop (line 6-15) searches for reachable configurations starting from the initial configuration (initial control global states and initial variable values) and marks them. The second while loop splits domain variable value blocks for each outgoing unstable transition (line 18) and incoming unstable transition (line 21). We repeat this process until there is no unstable block/transition.
The online minimization by Algorithm 4.1 merges bisimulation equivalent configurations and hence any two equivalent configurations in G* can be reached from the initial configuration through a same transition sequence in . On the other hand,
CEFSM and its reachability graph have the same executable traces. Consequently, the constructed minimal reachable graph G* contains the same executable traces as .
We have:
Proposition 4.4. A reachable minimal graph G* from the online minimization Algorithm
4.1 of a CEFSM contains the same executable traces as .
Recall that the essence of authentication and authorization property analysis of a protocol system is on its executable traces. From Proposition 4.2 and 4.4, we can reduce the analysis of and hence the original machine M to the minimal reachable graph G*.
35
4.5 SECURITY PROPERTY ANALYSIS - FALSE NEGATIVES AND FALSE POSITIVES
Given an original CEFSM M, we apply trace inclusion transformation and then online minimization and obtain a minimal reachable graph G* that is of a manageable size and contains all the executable traces of M. With G* we can verify the desired authentication and authorization property, such as: action e (authentication and authorization) must always precede action d (User data access). Equivalently, none of the transitions (states) of action d is reachable from the initial state of M after removing all the transitions
(states) of action e. This can be done by a simple graph search of G*.
However, we may have introduced false negative and false positives due to the two state space reduction procedures. We show that trace inclusion transformation and online minimization do not introduce any false negative. We also present an efficient algorithm to filter out all the false positives.
4.5.1 False Negative
A protocol system has a desired security property if its CEFSM M does not contain any attack traces against that property. We have shown that all the traces, including the attack traces, of M are included in the traces of the minimal reachable graph G* after trace inclusion transformation and online minimization. Therefore, if G* does not contain any
36 attack traces, then M does not have any attack traces either and hence is secure. That is, a security property analysis of G* does not introduce any false negative:
Proposition 4.5. For temporal security properties, including authentication and authorization, an analysis of a minimal reachable graph of a CEFSM, after trace inclusion transformation and online minimization, does not introduce any false negative.
Sketch of Proof. We prove by contradiction. Suppose that there is no attack trace in the minimal reachable graph G* but there is an attack trace τ in the original CEFSM M that is an executable trace from the initial state to an attack state d (e.g. accessing to private data) without going through any authentication and authorization transitions e. Then it is also an executable trace after a trace inclusion transformation to , following the same states and transitions in both machines. Since and its reachability graph have the same executable traces and G* is bisimulation equivalent to , τ must be an executable attack trace in G*- a contradiction.
From the proof, we conclude:
Corollary 4.1. Every attack trace of the original CEFSM M is also an attack trace in G*.
37
4.5.2 False Positive
From Corollary 4.1, if the minimal reachable graph of a CEFSM does not contain any attack traces, then the protocol system is secure. However, the converse may not be true.
That is, an attack trace in G* may not be an attack trace in M. If this happens then there is a false positive from an analysis of G*. From Corollary 4.1, we only have to examine all the attack traces from G* and rule out all the possible false positives. This can be checked directly on the original CEFSM M:
Algorithm 4.2 (Checking False Positives)
Input: Original CEFSM M and an attack trace c=v0,t0,1,v1,t1,2,…tn-1,n,vn from G*
Output: whether c is an attack trace on M. begin
1. = initial variable value of ;
2.
3. if (isExecutable(M, i, v0,t0,1,v1,t1,2,…tn-1,n,vn, ))
4. return c is an attack trace in M
5. else
6. return c is a false attack trace end
subroutine isExecutable(M, i, vi,ti,i+1,vi+1,…tn-1,n,vn, )
38
7. if (i==n)
8. return true;
9. if (Pt( )==false)
10. return false;
11. else if for any input on ti,i+1 (isExecutable(M,i+1,vi+1,ti+1,i+2,vi+2,…tn-1,n,vn, ))
12. return true;
13. return false
In Algorithm 4.2, given an attack trace c from G*, we restore the input, output, predicate and action on the transitions along the trace and check if c is executable in M
(line 3). If c is not executable in M, then it is a false positive (line 6). We use a subroutine isExecutable to recursively check if the trace from state vi to vn is executable in M, given the variable values ( ) at vi in M. If the predicate Pt of transition ti,i+1 on ( ) is false (line
9), then the transition ti,i+1, hence the trace, from state vi to vn is not executable. While restoring the original input and output for each transition (line 11), there can be a range of possible values (e.g. parameters). If any one of the values makes the transition executable, then we move on – the trace is executable so far. However, if for all the possible input values the transition is not executable, then the trace is not executable – blocked at the current transition.
39
4.5.3 Simultaneous Execution Traces
On a protocol system there can be a number of simultaneous sessions – protocol runs, and their execution traces can be interleaved. For instance, an attacker can initiate a session while a normal session is in process. The two execution sequences may reach a same configuration (same system state and variable/parameter values), and the attacker can follow the execution of the normal protocol run and launch an attack.
To model the interleaved behaviors of m simultaneous sessions we can take the
Cartesian product of m CEFSMs, each of which is a protocol run. Or, equivalently, we can examine the Cartesian product of their corresponding minimal reachable graphs.
4.6 SUMMARY OF OUR APPROACH
We now summarize our approach. Given a protocol system, we model it by a CEFSM
M. We first reduce its state space by trace inclusion transformation and then by online minimization (Algorithm 4.1), obtaining a reachable minimal graph G*.
We then determine simultaneous runs of the protocol system. The general rule is: we make as few runs as possible yet covering all possible attacks. It depends on the application environment and practical constraints. For instance, it is proved that one attacker is equivalent to two or more attackers in the general Dolev-Yao model [9]. For
OAuth, four simultaneous sessions cover all possible attack scenarios (see Chapter 4.1). 40
Suppose that m runs are needed. We take the Cartesian product of m minimal reachable graph G* for an analysis.
We examine the temporal properties of the security protocol under investigation, such as authentication and authorization. We identify type e transitions (states) which correspond to the actions that must be taken (e.g. authentication and authorization) before any type d transitions (states) which are from the actions to be granted after satisfying the security requirements (e.g. private data access).
Finally, we remove all the type e transitions and check whether any of the type d transitions is still reachable from the initial system state. If none of them are reachable, the security property is verified. Otherwise, for every executable trace from the initial configuration to a type d transition, check whether it is executable using Algorithm 4.2; if it is executable then an attack trace is identified, otherwise, it is a false positive.
4.7 CASE STUDY: OAUTH
In this section, we apply our general methods for an analysis of OAuth.
41
4.7.1 A Formal Model of OAuth
We first construct EFSMs MSP, MC and MU for Service Provider, Consumer and User from the OAuth specification [50]. There are 4, 5 and 4 states in the MSP, MC and MU, respectively, as shown in Figure 4.1. For readability, only the input and output of the transitions are shown.
(a) Service Provider
(b) Consumer
(c) User
Figure 4.1. EFSMs of OAuth 1.0 (a) Service Provider (b) Consumer and (c) User
42
Service Provider has variables to keep track of Consumer and User information,
Unauthorized Request Token, Authorized Request Token, Access Token, callback URL of Consumer, timestamps and nonce. Consumer has variables for Unauthorized Request
Token, Authorized Request Token, Access Token, timestamps and nonce. User has variables for Unauthorized Request Token and Authorized Request Token. In OAuth
1.0A, all three parties have the same variables as in version 1.0, and there is an additional verifier.
For the attacker model MA, the initial knowledge includes the Service Provider’s
Request Token URL, User Authorization URL, Access Token URL, and the necessary identity information so that he can make request to Service Provider as a Consumer.
There are two types of API interfaces and attacker’s capabilities:
(1) All communication channels (API) are secure and all communicating parties are mutually authenticated. The attacker can be a legitimate User or Consumer, but it cannot impersonate Service Provider or eavesdrop the messages exchanged between other communicating principals. It is a weak form of attacker model. Most of the OAuth
Service Providers require HTTPS for API interfaces that falls into this category.
(2) A strong attacker model complies with the well-known Dolev-Yao model [20].
Here we assume that communication channels (API) are not secure and communicating parties are not mutually authenticated. The attacker can have full control of the network; it can intercept, delete, insert or modify any messages in transit. There are some OAuth
43
Service Providers that accept insecure HTTP for API interfaces [65], and also the HTTPS constraint can be easily bypassed [1].
Given M=(MSP, MC, MU, MA), the attacker MA can be a weak model MWA or a strong one MSA. We first discuss in detail the first case of weak model MWA, that is, all the API interfaces are secure with HTTPS; most of the OAuth implementations and the majority of API application follow this model. We have implemented all our algorithms and apply them to a formal CEFSM model of OAuth for an automated analysis of its security properties as follow.
4.7.2 Security Properties of OAuth
We verify the following five security properties of OAuth. The first three properties are on the authorization process and the last two are on User service security.
Property P1: If a Consumer accesses User’s data with an Access Token, then Service
Provider must have issued the Access Token to this Consumer.
To check P1, we first remove the transitions in the constructed minimal reachable graph G* of Service Provider issuing Access Token. We then check whether the transitions that represent Consumer accessing data are still reachable from the initial system state.
44
Property P2: If a Consumer obtains an Access Token by exchanging a Request Token, then the Request Token must have been authorized by User.
To check P2, we remove the transitions of User authorization and check if the transitions for Consumer to exchange for Access Token are still reachable from the initial system state.
Property P3: If a Consumer obtains an Access Token by exchanging a Request Token, then the Service Provider must have issued the Request Token to this Consumer.
To check P3, we remove the transitions of Service Provider issuing Request Token and check if the transitions for Consumer to obtain Access Token are still reachable from the initial system state.
Property P4: If a Consumer accesses data that belong to User, then that User must have authorized the access.
To check P4, we remove the transitions of User authorization and check if the transitions for Consumer accessing User data are still reachable from the initial system state.
Property P5: If a Consumer accesses data that belong to a User, then only this User can get the service from Consumer with the accessed data – not any other User.
45
After the Request Token is authorized, the User who has authorized the access and is redirected back to Consumer is the User who gets the service provided with the accessed data.
Thus to check P5, we remove the transitions where User authorizes the access or is redirected back to Consumer. Then we check if the transitions of Consumer accessing data are still reachable from the initial system state.
4.7.3 Trace Inclusion Transformation
Given a CEFSM model M of OAuth, we first define the abstraction function that simplifies M and produces a trace inclusion transformation . In OAuth, Consumer has to generate a nonce and a timestamp for each request made to Service Provider, and append a digital signature to the request. Since all the API interfaces are secure with
HTTPS, all the tokens are securely exchanged among Service Provider, Consumer and
User. Therefore, we define α and β as the functions that remove timestamps, nonce and signatures from the messages exchanged. We define function ω( ) as a projection to a subset of variables of with all the other variables and parameters removed, if they are related to timestamp, nonce and signature. From the OAuth specification, the predicate on ω( ) in M are independent of timestamp, nonce, signature and Token content, that is, independent of the variables not in ω( ). From Proposition 4.3, is a trace inclusion transformation from M. 46
4.7.4 Minimal Reachable Graph
We apply Algorithm 4.1 to to construct a minimal reachable graph G* for both versions of OAuth. The minimal reachable graph of OAuth 1.0 consists of 67 states and
OAuth 1.0A has 44 states. Note that the reachability graph is infinite. From
Chapter 4.6, four simultaneous runs, and hence a Cartesian product of 4 minimal reachable graphs G* is constructed for an analysis. The number of reachable states in the two Cartesian products is 167,648 and 124,800 for OAuth version 1.0 and 1.0A, respectively.
4.7.5 Results
As summarized in Chapter 4.5.2, if any attack trace is found, we use Algorithm 4.2 to check if it is a false positive. If no attack trace is found, from Proposition 4.5, we are sure that there is no false negative and hence the protocol is secure with respect to that security property.
We now summarize the results from out automated security property verification.
47
Our experiment shows that both OAuth 1.0 and 1.0A satisfy Property P1, P2 and P3.
Consumer can access User’s data only after Consumer has obtained an Access Token.
Meanwhile, to obtain Access Token, Consumer must first obtain a Request Token and has the Request Token authorized by the User.
Our study also shows that both versions of OAuth satisfy Property P4. Consumer accesses a User’s data only after the User has authorized the corresponding Request
Token. However, our investigation indicates that Property P5 is violated in OAuth 1.0.
That is, after User authorization of Consumer’s access right, the User may not obtain the service provide from the accessed data; some other User may. It is exactly how a Session
Fixation attack succeeded as reported in [50]. We identify a total of 25 different traces leading to this attack, including the one reported in [50]. For OAuth 1.0A, we also identify 1 attack trace from the Cartesian product of the minimal reachable graphs.
Algorithm 4.2 rules it out as a false positive. Therefore, we conclude that OAuth 1.0A is secure in terms of the five security properties we investigate.
4.7.6 Insecure Interfaces
We now discuss briefly the case that the API interfaces are insecure with HTTP, for instance. There are API authentication and authorization protocols in the public domain which still use insecure API [65]. We comment with respect to the five security properties in Chapter 4.7.2. 48
Our results show that neither OAuth 1.0 nor OAuth 1.0A satisfy security property P4 anymore. Our attack traces show that when an attacker obtains information of a Request
Token by eavesdropping or message interception he can authorize it on behalf of the
User. Consequently, Consumer can access User data without the authorization by the
User. Our attack traces also show that Property P5 is violated, that is, Session Fixation attack is still possible for both OAuth versions. Specifically, if an attacker seizes the verifier and brings it to Consumer then he can still obtain the service authorized by the
User. We also find attack traces on property P2, but we rule them out as false positives.
Similarly, if an implementation of OAuth is stateless, then there are security attacks for both versions.
The current OAuth specification uses nonce, timestamps and signatures to guard against possible attacks. If the API interfaces are secure, they are not needed. On the other hand, if the API interfaces are insecure, they are not sufficient to guarantee the desired security properties. This observation might suggest a redesign of the protocol. It may also involve rather complicated analysis with insecure API interfaces.
49
CHAPTER 5: RELIABILITY AND SECURITY TESTING OF PROTOCOL IMPLEMENTATION
Even with perfect protocol design, flaws may also be introduced to the implementation during the phase of coding, deployment and integration. In this chapter, we investigate a general theory and algorithms for automated protocol implementation model synthesis and implementation security testing.
As mentioned in Chapter 1, the first step in our approach is to synthesize a behavior model (FSM) to represent the protocol black box under test. Then the synthesized model is used to assist different security testing applications.
Given a protocol implementation B, we look at two different model synthesis approaches: active and passive approaches.
5.1 PROTOCOL IMPLEMENTATION MODEL SYNTHESIS: ACTIVE LEARNING: LISFSM*
The first step for active protocol implementation model synthesis is to obtain the input and output alphabet of the protocol implementation. Many researches [12][13][19][42]
[41][68] have focused on reverse engineering the message format of a black-box binary.
In our work, we assume that the I/O alphabet is known. L* [3] and Lfsm* [61] propose
50 methods to learn an automata and a Finite State Machine, respectively. However, they can only learn completely specified systems. For a large complicated system, often only a small subset of inputs is expected at a state, and others are ignored as unspecified receptions. For example, in an IRC server, at the initial state, only user login messages are expected and processed – users are not allowed any further actions, such as joining a channel and chatting before logging in. Sometimes, malicious code, and hence attacks, are based on system inputs, which are not specified in system normal operations.
Therefore, we want to take into consideration of unspecified system behaviors when we construct an implementation system model. In this paper, we extend L* and Lfsm* and modify the definition of Observation Table (Chapter 5.1.1) for learning incomplete specified Finite State Machine (ISFSM).
In L* and Lfsm*, the learning process starts with an empty Observation Table. Given a protocol implementation B and its I/O alphabet, we may also have the specification M of
B. M may not be identical to B since there may be discrepancy between specification (M) and implementation (B). However, M is supposed to be much close to B than an empty
Observation Table. Therefore in our approach, we want to modify the learning algorithm
(Chapter 5.1.2) so that we can start the learning procedure with M instead of an empty
Observation Table,.
51
5.1.1 Observation Table
We first define basic notations for our approach.
Definition 5.1: Given two string x and y, the concatenation of x and y is denoted as x y
Definition 5.2: Given two string s and t such that s is the prefix of t, denote t \ s to mean the string when we remove the prefix s from t. Therefore, y = x y \ x
Definition 5.3: C = {s a | a A and T(s a) is defined}
The observation table is denoted (S, E, T). Each row labeled by elements of (S C) is a sequence that leads the machine to that state. Each column is labeled by elements of E.
The entry for row s and column e is equal to T(s e) \ T(s), which stores the output from that state upon each separating sequence.
The main difference between L* and Lisfsm* is the following. In L*, the rows of the observation table are (S C’) where C’={s a | a A}. Lisfsm* keeps only those input symbols that are defined at each state. We show in out experiment that we can save a huge amount of space by maintaining an ISFSM instead of a fully specified FSM.
We then define the consistency and closeness of an Observation Table in our approach and show that with our definition, an Observation Table (S, E, T) still represents the minimal FSM that is consistent with T.
52
Definition 5.4: An observation table is closed if for each t in (S C) there exists an s in S such that row(t)=row(s).
Definition 5.5: An observation table is consistent if s1 and s2 are elements of S such that row(s1)=row(s2), for all a in A, either (1) row(s1 a)= row(s2 a), both T(s1 a) and T(s2
a) are defined, or (2) neither row(s1 a) nor row(s2 a) is in the Observation Table.
The definition of closeness and consistency in [3] is a special case of our definition here.
If the observation table (S, E, T) is closed and consistent, the corresponding FSM M(S,
E, T) can be defined as follows:
Q = {row(s) | s S}
q0 = row( )
(row(s), a) = row(s a), if T(s a) is defined
(row(s), a) = T(s a) \ T(s), if T(s a) is defined
We prove the following theorem which holds for L* still holds under the new definitions of closeness and consistency.
Theorem 5.1. If (S, E, T) is a closed and consistent observation table, then the FSM M(S,
E, T) is consistent with the finite function T. Any other FSM consistent with T but inequivalent to M(S, E, T) must have more states.
53
This theorem is proved by several lemmas.
Lemma 5.1. Assume that (S, E, T) is a closed and consistent observation table. For the
FSM M(S, E, T) and for every s in (S C), (q0, s) = row(s).
Proof: Since s (S C), T(s) is defined.
The lemma is proved by induction on the length of s.
For the base case, the length of s is 0, i.e. s = . (q0, s) = (q0, ) = q0 = row( ) = row(s).
Assume that for every s in (S C) of length at most k, (q0, s) = row(s).
Let t (S C) of length k+1, that is t = s a for some string s of length k and some a in
A. If t C, then s S. If t S, then s S since S is prefix closed. Then
(q0, t) = (q0, s a)
= ( (q0, s), a), by induction hypothesis
= (row(s), a), by the definition of and s S
= row(s a)
= row(t)
This completes the induction and the proof of Lemma 5.1.
Lemma 5.2. Assume that (S, E, T) is a closed and consistent observation table. For the
FSM M(S, E, T) and for every s in (S C), (q0, s) = T(s).
Proof: Since s (S C), T(s) is defined.
The lemma is proved by induction on the length of s.
54
For the base case, the length of s is 0, i.e. s = . (q0, s) = (q0, ) = = T( ).
Assume that for every s in (S C) of length at most k, (q0, s) = T(s).
Let t (S C) of length k+1, that is t = s a for some string s of length k and some a in
A. If t C, then s S. If t S, then s S since S is prefix closed. Then
(q0, t) = (q0, s a)
= (q0, s) ( (q0, s), a), by induction hypothesis and Lemma 5.1
= T(s) (row(s), a), by the definition of and s S
= T(s) (T(s a) \ T(s))
= T(s a)
= T(t)
This completes the induction and the proof of Lemma 5.2.
Lemma 5.3. Assume that (S, E, T) is a closed and consistent observation table. Then the
FSM M(S, E, T) is consistent with the finite function T. That is, for every s in (S C) and e in E, (q0, s e) = T(s e) if T(s e) is defined.
Proof: The lemma is proved by induction on the length of e.
For the base case, the length of e is 0, i.e. e = . Also s (S C), so T(s) is defined.
Then
(q0, s e) = (q0, s )
= (q0, s), by Lemma 5.2
= T(s) = T(s ) = T(s e)
55
Assume that for every e in E of length at most k, (q0, s e) = T(s e) if T(s e) is defined.
Let e1 E of length k+1, that is e1 = a e for some string e of length k and some a in
A.Since E is suffix closed, then e E. Because the observation table is closed, there exists s1 S such that row(s)=row(s1). Then
(q0, s e1) = (q0, s a e)
= (q0, s) ( (q0, s), a e), by Lemma 5.1 and Lemma 5.2
= T(s) (row(s), a e)
= T(s) (row(s1), a e)
= T(s) ( (q0, s1 a e) \ T(s1)), by induction hypothesis
= T(s) (T(s1 a e) \ T(s1))
= T(s a e)
= T(s e1)
Lemma 5.4. Assume that (S, E, T) is a closed and consistent observation table. Suppose the FSM M(S, E, T) has n states and is consistent with T. If M’=(Q’, q0’, ’, ’) is any
FSM consistent with T that has n or fewer states, then M’ is isomorphic to M(S, E, T).
Proof: Since M’ is consistent with T, for any every s in (S C) and e in E, ’(q0’, s e) =
T(s e).
’(q0’, s e) = ’(q0’, s) ’( ’(q0’, s), e) = T(s e). Therefore, ’(q0’, s) is equal to row(s) in M. As s ranges over all of S, ’(q0’, s) ranges over all of Q’. Therefore, M’ must have at least n states, i.e., it must have exactly n states.
56
Thus for each s in S, there is a unique q’ in Q’ such that row(s)=q’, namely ’(q0’, s).
Define the mapping for each s in S, (row(s))= ’(q0’, s). Since both M and M’ have n states, the mapping is one-to-one and onto.
To see that (q0) = q0’,
(q0) = (row( )) = ’(q0’, ) = q0’
To see that the mapping preserves the transition function,
( (row(s), a)) = (row(s a)) = ’(q0’, s a)
’( (row(s)), a) = ’( ’(q0’, s), a) = ’(q0’, s a)
Also, since both M and M’ are consistent with T, ’(q0’, s e) = T(s e) = (q0, s e).
This concludes the proof of Lemma 5.4 and the proof of Theorem 5.1.
Theorem 5.1 shows that, with the modified definition of closeness and consistency in
Definition 5.4 and 5.5, if the observation table (S, E, T) is consistent and closed then FSM
M(S, E, T) is the minimal FSM consistent with T.
5.1.2 Supervised Learning Algorithm Lisfsm*
Given a protocol implementation B, we want to synthesize the FSM of B. If we don’t have the (approximate) model M of B, we can follow the learning procedure in [3] starting with an empty Observation Table and gradually refine it. According to [3], we will eventually obtain the FSM of B. However, if we have M, we want to first construct 57 an Observation Table OT0 that represents M and then start the learning procedure with
OT0 instead of an empty Observation Table. Algorithm 5.1 below shows the procedure to transform an FSM to an Observation Table.
Algorithm 5.1 (FSM to Observation Table )
Input: an FSM M0 and its initial state s0;
Output: an Observation Table OT0 that represents M0; begin
1. for each state s in M0
2. seq = input sequence that brings M0 from s0 to s
3. add seq as a row of OT0
4. for each a A
5. if seq a is defined (output query)
6. add seq a as a row of OT0
7. SS = separating sequences of M0
8. for each seq in SS
9. add seq as a column of OT0
10. for each column e in OT0
11. for each row s in OT0
12. seq1 = ask Teacher output query of s
13. seq2 = ask Teacher output query of s e
14. column e row s of OT0 = seq2 \ seq1
58 end
Note that in line 5, 12 and 13 in the algorithm, we query the Teacher (in practice, query the protocol implementation B) instead of querying the given FSM M to fill out the content of OT0. The reasons are the following. First, M may not be correct due to the discrepancy between specification and implementation. By query the Teacher (the protocol implementation), we can correct M. Second, M may have more states than B. To ensure Lisfsm* terminates with correct conjecture, we must start with an Observation Table whose number of rows is no more than the number of states of B since the number of states of our conjectures increases monotonically during the process of learning as shown below.
We omit other detail of Lisfsm* since it is the same as [3] except that we could initial the
Observation Table with an FSM (may be incompletely specified).
For the rest of this section, we show that the learning procedure starting with OT0 still terminates with a minimal FSM that is consistent with B.
The correctness straightforwardly follows [3]. Here we focus on the termination.
Suppose B has n states. Once we initiate the Observation Table with m states where m n, we can guarantee Lisfsm* eventually terminates. Whenever a column or a row is added during the learning process, the number of distinct rows increases by one. Thus the total number of such operations (adding a new row or column) over the whole run of
Lisfsm* must be n-m since there is initially m distinct rows and cannot be more than n.
Hence, Lisfsm* eventually terminates and makes a conjecture. 59
Lisfsm* can make at most n-m incorrect conjectures. If a conjecture Mi(S, E, T) is found to be incorrect by a counter example seq, then the correct FSM that is consistent with T but inequivalent to Mi(S, E, T) (since they disagree on seq) must have at least one more state than Mi(S, E, T), by Theorem 5.1. Therefore, Mi(S, E, T) has at most n-1 states.
Lisfsm* eventually makes the next conjecture Mj(S’, E’, T’) with at least one more state than Mi(S, E, T). This shows Lisfsm* can make at most n-m conjectures since the number of states of these conjectures must be monotonically increasing.
5.2 PROTOCOL IMPLEMENTATION MODEL SYNTHESIS: PASSIVE SYNTHESIS WITH
PARTIAL FSM REDUCTION
In this section we focus on a synthesis method that is fundamentally different than the learning-based approach. This new method only requires passive monitoring of B with the anticipated tradeoff that the quality of synthesize model depends on the monitored traces.
Given a set of protocol execution traces, ideally we want to find a minimized deterministic FSM that contains all these traces and use it as the estimation of B. Denote all valid input and output messages (byte strings) as MSGI and MSGO, respectively. Thus
* * an implementation B can be seen as a computable function fB: MSGI →MSGO where B starts from its initial state. Given B, we want to construct a FSM MB (S, I, O, , ).
60
However, it is well known that the problem is inherently hard [40]. In this work we study a generic framework of developing efficient approximate solutions for this problem; and we present algorithms that construct in polynomial time a reduced FSM – not necessarily minimized – that is consistent with all the collected traces.
The synthesis process contains four main steps: (1) A large number of traces are collected and filtered by some user specified abstraction functions (e.g. removing the session dependent fields); (2) optionally, each trace is preprocessed for identifying and removing repetitive trace sections. We assume that they correspond to loops in the synthesized model; (3) construct a tree FSM that accepts all the resulting traces and only these traces; (4) apply a reduction algorithm on the tree FSM that merges equivalent states and obtains an equivalent FSM as a final synthesized model; and (5) apply any necessary post-processing steps that complements what is done in step (2). Now we describe these steps in greater details.
Step (1): The execution of target protocol implementation B is monitored for a period of time and all input/output traces are recorded. Each trace is a sequence of message pairs
61
* β is deterministic w.r.t. an implementation B if and only if for any x,y MSGI , α(x) =
α(y) β(fB(x))=β(fB(y)). That is, I/O mapping and abstraction are commutative.
Step (2): One useful preprocessing mechanism is loop removal. It greatly reduces the length of traces that must be handled in the next steps. Since the lengths of the monitored traces are finite, it is impossible to precisely identify loops. We make a simple assumption: given a trace, any repetitive and consecutive sub-traces are assumed to be from a loop. For instance, in a trace uababcv (each letter represents an I/O pair), we assume that sub-trace ab is from a loop and the full trace is: s0us1as2bs1as2bs1cs3vs4 where si, i=0, 1,2,3,4, are states and a loop of length 2 is observed that is repeated twice from state s1: s1as2bs1as2bs1. The repeated sub-trace (substring) identification problem was well studied in string matching literature and efficient polynomial time algorithms were developed [15]. We process each trace and remove all the loops but record the location and content of the loops in order to restore them later.
Step (3): Given a set of preprocessed traces, we construct a tree FSM that accepts all the trace and only these traces as follow. Starting from an empty FSM with only an initial state s0, traces are added one after another. Assume that α and β are specified manually as the initial abstraction of MSGI and MSGO. The algorithm is given below.
Algorithm 5.2 (Construction of Tree FSM )
Input: implementation B and initial abstraction α and β;
Output: a tree FSM MT; begin
62
1. MT = ({s0}, s0, α(MSGI), β(MSGO), , );
2. for each monitored trace {
3. s=s0;
4. for i=1 to L do
5. if ( (s, α(xi)) is not defined)
6. add a new state snew to MT;
7. add a new transition (s,snew,α(xi),β(yi)) to MT;
8. s=snew;
9. else if ( (s, α(xi))≠β(yi))
10. refine α and/or β;
11. restart the algorithm from line 1;
12. else s= (s, α(xi)); end
For each trace (line 2), Algorithm 5.2 follows a prefix of it that is already in the tree
FSM (line 12). If an input message is not defined, a new branch is created (line 5-8). If the current abstraction of messages appears to be non-deterministic, the discrepancy of output will be detected (line 9). In this case we have to refine α and/or β by adding more fields in abstract symbols to make it deterministic. With a finite number of traces this algorithm terminates with a tree FSM with a cost proportional to the total length of traces.
Obviously, each trace is accepted by the FSM and each path from the root represents a trace or prefix of a trace.
63
Step (4) Given a large number of monitored traces, MT can be large and we want to reduce its size while retaining the equivalence in terms of traces. Formally, this step finds
MX that has the same alphabets as MT and all traces produced by MT could also be produced by MX. Note the reverse is not necessarily true, i.e. MX could contain more traces. Since it is NP-hard to minimize an arbitrary partially specified FSM [17], it is clearly also NP-hard to minimize a tree FSM. In the next section we present two reduction algorithms: a fast conservative merging algorithm that does not introduce extra traces and a more aggressive and expensive heuristic algorithm. Both of the algorithms produce a deterministic FSM as result.
Step (5): Finally we process the reduced FSM and apply the symmetric operations of those included in step (2). In case of loop removal, in this step we add the recorded loop back to the reduced FSM. This is well defined behavior since the reduction is based merging on state merging and every state in the tree FSM has a corresponding one in the reduced FSM. Note however, recovering loops might render the FSM nondeterministic that will restrict its use for certain flaw detection methods. One of the two algorithms we study does not require loop removal and it can synthesize loops during state merging.
5.2.1 Loop Free Tree FSM Reduction Algorithm
Both of the algorithms we describe involve merging equivalent states in the Tree FSM according to some criteria. The first algorithm imposes a strict notion of equivalence 64 hence reduces the FSM conservatively. This algorithm never introduces loop and requires loop removal and recovery steps as discussed in the previous section.
Given a state (node) in a tree FSM, an input sequence is valid if it takes the machine along a tree path from that state. Two states are equivalent if for any valid input sequence the two states produce the same output sequence. Consequently, two states are equivalent if and only if the two subtrees rooted at the two states are isomorphic. As a special case, two subtrees of height 1 are isomorphic if there is a one-to-one correspondence between their tree edges and the corresponding edges have the same I/O symbols. A state is of height 1 if the subtree rooted at the state is of height 1.
We now identify all the equivalent states of a given tree FSM bottom up as follows.
We first examine all the states of height 1 and identify equivalent states by a pair-wise comparison. Then we color the states of height 1 such that two states are equivalent if and only if they have a same color. We then shrink all the processed subtrees of height 1 into a node (of height 0) with the color of its root node. We now have a tree FSM of height decreased by 1. However, the leaf nodes represent subtrees now. Consequently, two subtrees of height 1 are isomorphic if there is a one-to-one correspondence between their tree edges and the corresponding edges have the same I/O symbols; furthermore, their corresponding incidental leaf nodes have a same color. We repeat the process of identifying equivalent states of height 1 bottom up until the height of the tree is 1.
We now expand and recover the tree FSM and each node maintains the color it has obtained during the process. We now examine all the nodes and their colors and merge nodes of a same color top down as follows. If two nodes u and v have a same color, we
65 merge them as follows. We remove v and redirect all the tree edges, which end at v, to u.
Since we merge nodes top down no nodes are involved in merging more than once.
However, when the process terminates we obtain a DAG (Directed Acyclic Graph) FSM
MX. It is equivalent to the original FSM MT but of reduced size. We summarize the procedure in Algorithm 5.3.
Algorithm 5.3 (Tree FSM Reduction)
Input: a tree FSM MT;
Output: an equivalent and reduced DAG FSM MX; begin
1. repeat
2. for each pair of states u and v of height 1 do
3. if u v /* equivalent */
4. assign a same color to u and v;
5. shrink all subtrees of height 1 to a leaf node;
6. until the height of the tree is 1
7. recover tree FSM with nodes color reserved;
8. merge nodes of a same color top down and obtain a DAG FSM M X end
Suppose that the given tree FSM has n leaf nodes (states). A naive pair-wise comparison of subtrees of height 1 costs O(n) and there are of order n2 comparisons at
66 each level, and the cost for processing at each level is O(n3). Instead, for each node of height 1, we assign a string to it by the color of each of its child node along with the I/O symbols on the tree edge to that child node. Since both the I/O symbols and colors have a range of order n, we can radix sort the associated strings with all the nodes of height 1 and identify equivalent nodes (states) whose associated strings are identical. The cost is
O(n). Therefore, for each level of processing in Line 2, the cost is proportional to the number of leaf nodes, and hence:
Proposition 5.1. The total cost of the reduction Algorithm 5.3 is proportional to the number of nodes of a tree FSM.
We illustrate algorithm 5.3 using an example in Figure 5.1.
67
(b) DAG FSM without loops
(a) Tree FSM after loop removal
(c) Final result
Figure 5.1. Example of Algorithm 5.3
Given four traces: tr1={a,b,d,a}, tr2={c,c,a,b,d,e}, tr3={e,a,b,a,b,a,b,d,e} and tr4={e,d,a}, we first identify loops. In tr2 there is a self loop with {c,c} and in tr3 a loop of length 2 with {a,b,a,b,a,b}. The loops are removed for the time being and we obtain: tr2={a,b,d,e} and tr3={e,d,e}. Applying Algorithm 5.2, we construct a tree FSM as in
Figure 5.1(a). We initially color all the nodes white. We identify equivalent states of height 1 bottom up and have s4 s5; both are colored red. We shrink the two corresponding subtrees to leaf nodes. We repeat the process and have s3 s2; both are colored green. We now process top down and merge the two subtrees rooted at s3 and s2
68 with green color. We obtain a DAG tree in Figure 5.1(b). Finally, we attach the loops and obtain a reduced machine MX as in Figure 5.1(c).
5.2.2 A Heuristic Reduction Algorithm
In this section we present a heuristic algorithm that attempts to merge states of a tree
FSM more aggressively. It does not require loop removal and recovery. Generally, finding a minimal FSM involves partitioning states of a given FSM according to some notion of equivalence. A canonical searching method with backtracking was proposed by
Bierman [5] for a different problem domain, followed by many heuristic algorithms. In our work, for the sake of scalability we develop an efficient heuristic algorithm that removes backtracking. Now we describe the algorithm with an example.
Given the tree FSM MT, we want to find the minimal number R such that we could assign each state s in MT a number π[s] [1..R] without violating the following two types of constraints.
Type-1: for any two states s1 and s2, if there is an input sequence seq (s1, seq) ≠ (s2, seq), then π[s1] ≠ π[s2];
Type-2: for any four states s1, s2, s3, s4 and input symbol x, if (s1, x)=s3 and (s2, x)=s4, then π[s1]=π[s2] π[s3]=π[s4].
These constraints guarantee that state merging according to partition π reserves all traces from MT. According to Type-1 constraint, if two states could be separated in MT 69 then they must be assigned different numbers. We could calculate all state pairs with
Type-1 constraints using a typical iterative algorithm. At the beginning pairs separable by single input symbol are marked; then we find pairs whose successors under an input symbol are marked. This process is repeated until no more pairs could be found. Type-2 constraints state that if two states are assigned the same numbers, so should their successors under same input. Enumerating all constraints of this type is trivial as we could simply visit each pair of states in MT.
After the constraints are calculated for MT, we start to construct π by assigning states in order. Given R, it takes exponential time to determine whether the assignment using
[1..R] is feasible, because a partial assignment might cause confliction that will only be realized later. In such case backtracking is needed resulting a tree structure (exponential size) of the solution space.
We design a polynomial time algorithm to calculate suboptimal assignment. The principle is to avoid backtracking by always keeping partial assignment “promising”.
Clearly assigning each state to a different number is a trivial solution satisfies all constraints (but no states could be merged). Similarly when we consider the i-th state, we only try to assign it a number such that it is possible to complete valid assignments for all remaining states π(sk) for all k≥i. The minimization algorithm is outlined below.
Algorithm 5.4 (Reduction of tree FSM )
Input: a tree FSM MT with N states;
Output:a reduced FSM MX;
70 begin
1. calculate all type-1 constraints for MT;
2. calculate all type-2 constraints for MT;
3. for i=0 to N-1 do
4. π(si)=0; // assign current state;
5. while (! isPromising(π, i)) do π(si)= π(si)+1;
6. MX = MT;
7. merge all states of MX with same value in π; end
// check whether a partial assignment is promising subroutine isPromising(π, i)
8. for j=i+1 to N-1 do
9. π(sj)=-j; // complete π for all states;
10. foreach C2 in type-2 constraint do
11. if (π violates C2) return false;
12. refine π according to C2 if necessary;
13. if (π is changed) goto line 10;
14. foreach C1 in type-1 constraints do
15. if (π violates C1) return false;
16. return true;
71
The isPromising subroutine verifies whether a partial assignment (up to si) could be expanded to a valid full assignment. To do this, first we assign a unique hypothetical
(represented as negative) number to each of the rest states (line 8-9). Some of the hypothetical values need to be adjusted according to type-2 constraints; therefore we iteratively refine π (line 10-13). Essentially for any type-2 constraint of form
The main algorithm assigns value for each state sequentially. For state Si, we can prove that a promising value exists in [0..i] using induction therefore the algorithm never needs to trace back. After π is determined, we merge all states in MT with the same assignment to obtain a FSM MX.
The cost of this algorithm could be broken into several components. There are at most
O(N2) type-1 constraints and O(N2) type-2 constraints. Every time isPromising subroutine is called on a partial assignment, at most O(N2) passes of refinement will be conducted, one for each type-2 constraint. The total cost for the subroutine is then O(N2)×O(N)
2 2 +O(N ). The main algorithm first calculates all constraints with cost O(|AI|×N ) with some preprocessing efforts. For the i-th state at most i values will be verified (line 5) using isPromising. Put it altogether, the total complexity of algorithm 5.4 is
2 2 2 2 2 5 O(|AI|×N )+O(N )×(O(N )×O(N)+O(N )) = O(|AI|×N )+O(N ). Although this appears to
72 be expensive, in practice the actual numbers of constraint are usually moderate. Later in this thesis we show the performance result of this algorithm.
We illustrate algorithm 4 using an example in Figure 5.2.
Table 5.1 Partial Assignments by Algorithm 5.4
Assignment isPromising? [0,?,?,?,?,?,?] T (b) Minimized FSM [0,0,?,?,?,?,?] T [0,0,0,?,?,?,?] F
[0,0,1,?,?,?,?] T Table 5.2 Constraints [0,0,1,0,?,?,?] F Type-1 Constraints [0,0,1,1,?,?,?] T π[s0] ≠ π[s3] [0,0,1,1,0,?,?] T π[s ] ≠ π[s ] 0 5 [0,0,1,1,0,0,?] F Type-2 Constraints [0,0,1,1,0,1,?] T π[s ]=π[s ] π[s ]=π[s ] (a). Tree 1 2 2 3 [0,0,1,1,0,1,0] T FSM π[s3]=π[s5] π[s4]=π[s6]
Figure 5.2. Example of Algorithm 5.4
Figure 5.2(a) shows a tree FSM with 7 states, 3 input symbols and 3 output symbols.
Table 5.2 shows the constraints calculated. Note that two states (
Type-2 constraint because Type-1 constraint forbid them to have the same assignment.
Table 5.1 shows each partial assignment generated and whether it is promising. For
73 instance, the partial assignment [0,0,0,?,?,?,?] (“?” represents the state is unassigned) is not promising: when the subroutine refines it with Type-2 constraint; it will realize that s3 needs to be assign 0 (first Type-2 constraint). However this will lead to violation of the first Type-1 constraint. After all states are visited we get an assignment with two values, therefore we merge s0,s1,s4 and s6 into one state in Mx (Figure 5.2(b)) and s2,s3,s5 into another. Transitions of Mx are created from transitions of MT using the assignment as well.
5.3 COMPARISONS OF ACTIVE AND PASSIVE SYNTHESIS
We introduce both active and passive protocol implementation model synthesis approaches. Here we compare their pros and cons.
This active approach has very attractive properties due to the elegance of L* algorithm: the estimation is always promising in the sense that if the abstraction of B is indeed equivalent to an FSM of N states, at most N estimations will be made before the algorithm terminates and the resulting estimation is indeed minimal FSM equivalent to B given a minimal adequate Teacher exists. Nonetheless, this approach is not practical for many real problems due to the following two reasons. First, the success of learning relies on the idealization of teacher, i.e. the teacher can find counter-example efficiently.
However as we see the cost of teacher’s algorithm is high and in the worst case it may fail to find counter-examples [61], causing the learning process to terminate with an 74
* inaccurate model. Second, L isfsm algorithm requires large number of active input query to
B hence the tester needs to construct actual messages (in MSGI) from the abstract symbol
(in I). Depending on the characteristics of the target protocol, it might be difficult of padding missing fields with meaningful values.
Passive synthesis is fundamentally different than the learning based approach. It requires only passive monitoring. The tester does not have to construct real protocol message, which could be a very difficult engineering task. Therefore, it is easier to implement. The computational complexity is also lower than the active approach [61].
However, the FSM reduction does not guarantee the resulting FSM is minimal.
Sometimes, it may merge very few states especially Algorithm 5.2 which makes the resulting FSM approximately the same size of the tree FSM. The quality of the synthesize model greatly depends on the coverage of the monitored traces. The synthesized FSM does not guarantee to cover all behavior of B; if there is a part of behavior in B that does not show up in the monitored traces, there is no way the synthesize model covers that part of behavior.
To summarize, in general, the active learning approach gives more accurate model with higher cost and the challenge to construct real protocol messages. The passive synthesis is easier to implement but gives only approximate model.
75
5.4 CASE STUDY 1: MODEL BASED FUZZ TESTING
For the first case study, we use the synthesized model to build a real protocol fuzz testing tool. This section reports our experiments with the tool and its findings on one of the most popular application protocols used in today’s Internet: Microsoft MSN instant messaging (MSNIM) protocol.
5.4.1 Overview of Our Tool
Fuzz testing works by mutating a portion of the normal input data at the ingress interface of a protocol component in order to reveal unwanted behaviors [51]. It becomes increasingly popular in recent years as a means of security flaw detection due to extremely low cost and proven effectiveness. However, existing black-box protocol fuzzing tools are designed in a protocol specific and ad-hoc manner. The selection of input messages from a session to mutate is done either randomly or manually. The unstructured selection of input makes it impossible to measure the comprehensiveness
(i.e. how much is covered) of a test suite, and relying on human interfaces usually makes such testing tools hard to automate.
The key idea of this case study is to use a synthesized behavior model of the protocol implementation under test to guide input selection for fault coverage. Figure 5.3 illustrates the workflow of this two-phase approach. 76
Protocol Synthesis Flaw Detection Strategy Syntax Algorithm & Coverage Criteria
Black-box Formal I/O Message Selection of Protocol Behavioral Traces Input Data Implementation Model
Formal Model Based Flaw Detection
Syntax Based Flaw Detection
Random Detection
Figure 5.3. Workflow of Model Based Fuzz Testing
After network traces are represented by abstracting out input/output message symbols, a formal behavioral model is synthesized for capturing the design aspects of the protocol.
The model can be constructed using either active learning technique on protocol implementation or passive monitoring of input/output traces. The synthesized formal model represents in a succinct way the key states and transitions of a protocol implementation that systematically guides the flaw detection process.
The synthesized model is then used to define test coverage criteria and to guide test input selection. Fault coverage criteria based on a behavioral model has fundamental advantages over criteria from protocol message syntax. Although the latter is easy to implement, it is in general inaccurate. Messages of a same type may serve quite different roles in a protocol session. For instance, an ACK packet in TCP can trigger different processing logics depending on the receiver’s current state of congestion control
77 mechanism. Consequently, the same input serves different roles when fault discovery is concerned and, therefore, should be distinguished in the test selection.
To our best knowledge, this is the first automated tool providing measurable coverage and universal applicable to all network protocols.
5.4.2 MSN Instant Messaging Protocol
The MSNIM protocol (version MSNP9 at the time of our experiment) is a proprietary text-based protocol developed and used for Microsoft MSN messaging services. In recent year due to its fast growing popularity it has been extensively reverse-engineered.
Protocol syntax and description are available in natural language, meanwhile a large number of compatible chatting applications have been developed, which are able to interoperate with the official MSN servers. Because of this unique situation it is worth investigating the reliability and security properties of these unofficial client implementations. In our experiment, we choose two most popular open source client applications: aMSN [2] and Gaim (now under the name pidgin [56]). Since both applications support both Linux (Ubuntu) and Windows (Windows XP) operating system, we have 4 different implementations. In the experiments we treat them as pure black-box without using any knowledge from the source code. As a matter of fact, although these applications are built on portable low level libraries, we show that they exhibit differently on Linux and Windows in terms of reliability. 78
The syntax of the protocol is quite straightforward. A client communicates with the server sending text message over TCP connections. Each message contains a prefix specifying the type and a list of parameters. Certain types of messages can also carry a transaction ID and/or an encoded payload section. MSN protocol supports a diverse set of functionalities from simple presence information and text messaging to complicated features such as setting up video/audio conference. In our fuzz testing we focus on the login/logout procedure. When a user logs in from a client site, a sequence of messages is exchanged with the server to negotiate the protocol specification as well as to retrieve the user profile information such as list of contacts and their status.
5.4.3 Fuzz Testing Setup
We apply the passive synthesis algorithm described in Chapter 5.2. In this case study, the synthesized model is served as criteria to choose new test input. We do not need an accurate model so long as every new test always covers a different part of the black-box under test.
In order to take over the input and output of the MSN client we developed a special
SOCKS v5 proxy and forced the client to use it. The TCP connections are setup in the same way as a regular SOCKS v5 proxy, however both incoming and outgoing traffic are intercepted. Figure 5.4 shows the software architecture of our modified proxy and test controller. 79
Test Execution Controller Fuzz Goal Fuzz Test Sequence Protocol Spec. MSN Service Detector Operator Selector Synthesizer Server
d d n n a a
r r r r e e e e d d d MSN Service d Internet
Incoming Message o o o o c c c c
Client n n e e E E D
D
U U D D P P Outgoing Message Modified SOCKS 5 Proxy
Figure 5.4. Software Architecture of a MSN Client Fuzz Testing Tool
First we develop a simple message encoder (decoder) that translates the incoming
(outgoing) traffic from (to) MSN Protocol Data Unit (PDU). All incoming and outgoing messages (between client and server) are directed to the protocol specification synthesizer implementing Algorithm 5.1 to 5.4. Incoming messages are mutated selectively according to certain test sequence selection strategy and the current fuzz operator. Our goal is to crash the MSN client – see if the MSN client crashes under a mutated input. Finally, the goal of fuzz testing is determined automatically after mutated input messages are sent to the client.
80
5.4.4 Synthesized FSM Model
We choose to apply the proposed passive heuristic reduction (Algorithm 5.4) techniques to synthesize a model for the MSN client. By monitoring the normal sessions of the MSN protocol we are able to synthesize an FSM model as illustrated in Figure 5.5.
Figure 5.5. Synthesize FSM Model of MSNIM Protocol Client
The activities in these monitored sessions include login/logout, text messaging, changing of status and file transfer. We define both MSGI and MSGO as all valid MSN protocol messages, and the initial abstraction function α and β as the mapping of a message to its type (the three letter prefix of a message). During the construction of tree
FSM it turns out that the two initial abstraction functions are deterministic, therefore I
81 and O are the set of message types. The tree FSM contains 28 input symbols, 14 output symbols and 94 states. Minimizing it only takes 93 milliseconds with a Java implementation of Algorithm 5.4 running on a commodity PC, generating an FSM with
14 states and 48 transitions. The transition coverage criterion on this model is used to determine which input message to fuzz. One natural question is the accuracy of the synthesized model. Although it is difficult to obtain authorized formal documentation of the protocol, one can develop one independently from natural language description for an evaluation of the synthesized model. Also, the goal of this fuzz testing tool is not to obtain an accurate model but to effectively crash the MSN client under test. Our results presented in Chapter 5.4.6 show that the synthesized model servers well for this purpose.
5.4.5 Fuzzing Goal and Operators
The goal of testing MSN client implementations is to find an observable failure state, i.e. a crash. A crash is manifested in various ways on different platforms, and, consequently, the implication for security also varies. For instance on Windows such a failure will bring up an error message box that indicates a crashed process while on Linux the crashing of process is directly exploitable by attackers. As far as black-box testing is concerned, a crash instance represents an error.
We developed 26 single message fuzzing functions, which fall into four categories: (1)
Fuzzing data fields; (2) Fuzzing message type; (3) Intra-session message reordering; and 82
(4) Transition substitution. The constructed FSM formal model guarantees that the fuzz functions achieve a high coverage. We further elaborate these four categories below.
Data Field Fuzzing (16 functions): Functions belonging to this category modify one or more data fields of an input message and form a new one that is not acceptable to a normal client The modifications include deleting some or all fields, inserting and repeating fields, and altering the value of a field. A single field is altered depending on its type: for a string literal we expand it to an extremely long string; for a numeric field we change its sign or decimal point; and, similarly, for E-mail address, URL or IP address types we choose suitable functions.
Message Type Fuzzing (4 functions): Two fuzzing functions in this category simply change the message type to a different (defined or undefined) one. Other functions repeat or remove the message type field in a message. Since the new message is not likely to be a legitimate one of the modified type, the implementation might fail when the transition corresponding to the new type is triggered.
Intra-Session Message Reordering (3 functions): This type of fuzzing functions twists the order of normal message exchange with inserting, repeating, or dropping an intercepted message within the same session. Unlike the previous two categories these function do not generate illegitimate messages, instead, they force an unexpected transition.
Transition Substitution (3 functions): We also substitute an input message with another one that is likely to trigger a different transition that goes from the current state
(in MX) to another state. It seems that these functions should not cause a failure because
83 the client is expecting the new transition. However there are two issues. First, MX is not accurate and, as discussed, it may contain artificial paths. In fact, failures caused by transition substitution also provide opportunities to refine MX. Second, due to the assumption of fuzz testing we may not be able to construct an arbitrary input message. In our implementation we take advantage of the messages monitored during synthesis phase by storing and reusing the similar ones.
5.4.6 Result and Analysis
After the model is synthesized and the fuzzing functions are developed, we run our experiments on all four implementations guided by the transition coverage criteria. The test execution procedure (login/logout phase of the protocol) is fully automated and each pass attempts to cover a new transition in the model. We analyze the results in this section.
The first encouraging finding is that model based fuzz testing is very effective; almost all fuzzing functions find crash instances. Specifically, we find a total of 89 instances from aMSN and 61 instances from Gaim. We emphasize again that the instances do not necessarily correspond to individual bugs in the client application; instead, each of them represents a unique way of crashing the protocol implementation. On the other hand, we believe such result is extremely helpful for protocol testing process, especially given its high level of automation. We also show the number of instances found by each function 84 in Figure 5.6. Overall aMSN exhibits consistent behavior on Linux and Windows, while the two versions for Gaim are quite diverse. Further analysis on this issue will require software engineering efforts and is out of the scope of this thesis.
Figure 5.6. Number of Crash Instances from Fuzzing
85
We also measure the progress of finding new crash instance with the increase of transition coverage metric. The relationship for two of the fuzz operators
FZ_CHANGE_CMD and FZ_INSERT_CMD is shown in Figure 5.7.
Figure 5.7. Number of Crash Instances and Transition Coverage
Since each pass of execution guarantees to test a new transition, it is not surprising to see such a ladder-shaped progress. Note that only about 35% of the transitions have been covered since the synthesized model is not restricted to login/logout phase of the protocol, but our testing phase is.
Another major finding from our experiments is that different transitions in MT with the same input symbol might behave differently in terms of finding crash instances under the same fuzzing function. In MSN protocol this phenomenon is quite common since certain types of messages can serve for multiple purposes and are, therefore, processed by different logics (source code): some are immune to a fuzzing function while others are not. The messages with different roles are exactly reflected in MX by multiple transitions
86 with a same input symbol. This finding shows the synthesized behavioral model is truly useful and also justifies our transition coverage criteria.
The above observation has motivated our further comparison between the flaw detection capability of our model based approach and syntax based methods such as message type coverage that is frequently used in practice. Message type coverage measures the percentage of different protocol messages that is fuzzed by a function during an experiment. Since it does not need protocol specification (only the grammar) it can be easily implemented. In our study we also used message type coverage as a guide on all four client implementations. That is, for each pass of execution, the first occurrence of a message with uncovered type is fuzzed, and then the type is marked as covered.
As expected, the experiment guided by message type coverage detects less crash instances than our experiment using transition coverage of FSM model. For a message type that corresponds to multiple transitions in the model, any one transition can be selected causing that type to be marked. Consequently, the actual vulnerable transition might be missed. Given the synthesized FSM model MX we can calculate approximately the maximum and expected number of crash instances detected by achieving complete message type coverage. Let t=
87
If we further assume that all transitions with same input type have the same probability to be covered, we can also derive the expected number of crash instances and the probability that one experiment detects a maximal number of instances as follows:
We note that the formula above is not precise since it does not consider the topology of
MX, since a transition in MX may never be reached without reaching another transition. If an implementation is always executed from the initial state, the actual number of crash instances can be different than the theoretical result. As a matter of fact, Both ExpMT and
ProbMT, which we calculate, are lower bounds. We measure the actual numbers by complete message type coverage for one implementation (Gaim on Windows), as shown in Figure 5.8.
88
Figure 5.8. Maximum, Expected and Actual Number of Crash Instances in Gaim (Windows) under Complete Message Type Coverage
For about half of the functions some crash instances are missed, and it clearly shows the inefficiency of such syntax-based coverage criteria. Based on the synthesized model of Figure 5.5, we also calculate ProbMT for each function as shown in Figure 5.9.
Figure 5.9. Probability of Detecting Maximal Number of Crash Instance from Message Coverage
89
For the majority of functions, the probability of detecting all crash instances is below
60%. Our experiments prove the advantage of using a behavioral model over simple syntax based test selection schemes. On the other hand, transition coverage is not the only
(nor the most exhaustive) criteria one can use on an FSM model. Alternatives such as path coverage can be more effective under certain circumstances.
5.5 CASE STUDY 2: IMPLANTED MALICIOUS CODE DETECTION
In this section, we look at another security flaw detection application. Given a protocol implementation, we want to determine if there is any malicious function hidden inside the protocol implementation and the purpose of the malicious function. Here we need to synthesize a model that could accurately describe the behavior of the hidden function.
We choose to apply our generalized active learning algorithm for this problem.
The first step for active learning is to obtain the input and output alphabet of the protocol implementation. Many researches [12][13][19][41][42][43][68] have focused on reverse engineering the message format of a black-box binary. In our work, we assume that the I/O alphabet is known, including input of both the normal and malicious behavior of a black-box.
90
Given a protocol implementation B=B0’+B where B0 is the normal operation of the protocol implementation and B’ is the malicious function hidden inside the protocol implementation (B’ is empty if there is no hidden function) and its I/O alphabet, we want to synthesize the model of B (including B’). By studying the semantics of the synthesized model can tell us if there is any malicious behavior hidden inside a black box and the behavior of the hidden function.
Given the input alphabet of B, any input not belong to normal protocol execution indicates abnormal behavior hidden inside the black-box. Sometimes, however, the I/O alphabet of the implanted function is a subset of the normal I/O. For example, a MSN client message flooder which repeatedly sends out random messages to its contacts does not generate any message that is not a normal MSN message. In this case, we could not tell if any malicious function is embedded by only reverse engineering the input message format, which is another reason we couldn’t simply look for abnormal input alphabet.
Besides the I/O alphabet, we may also have the specification of B0. Take the contaminated IRC server for example; B0 is the normal IRC server function, B’ is the implanted backdoor. The specification of the IRC server M0 may be available (from RFC or developers manual). M0 can be a good start as we try to synthesize the FSM of B. We also show the result as we start the learning algorithm with M0 instead of an empty
Observation Table. Note that in our approach, we don’t require M0 to be identical to B0.
During the learning process, we can correct M0 and also learn B’.
In this section, we look at two case studies. The first case study is an IRC server with backdoor where the malicious function has a different set of input and output alphabet
91 than the normal IRC server function. The second case is a MSN client with message flooder where the malicious function does not introduces any extra input or output symbol that is not a normal MSN messages.
5.5.1 UnrealIRCd
UnrealIRCd [64] is an open source IRC (Internet Relay Chat) server. In November
2009, UnrealIRCd 3.2.8.1 was replaced by a new version with a backdoor in it at some of its mirror sites. The backdoor allows a remote attacker to execute arbitrary system commands with the privileges of the user running the IRC server. The malicious code was not detected until seven months later.
We applied our method to the backdoored IRC server and managed to synthesize the
FSM model and identify the backdoor function.
The input alphabet of UnrealIRCd 3.2.8.1 includes normal IRC protocol messages and also malicious input that will trigger the backdoor: the string “AB;” followed by any system commands.
Figure 5.10(a) and Figure 5.10(b) are the ISFSM for UnrealIRC server with and without the backdoor, respectively.
92
(a)
(b)
*To save space and preserve readability, we lump several transitions together. INMSG is a set of input messages that includes INMSG={AWAY, INVITE, LIST, NAMES, PING, MODE, TIME, WHO, WHOIS, WHOWAS}. They all produce the same output 451 at S0 and S2. Their outputs at S3 is OUTMSG1= {305, 403, 321323, 366, PONG, 401, 391, 315, 311378312317318, 406369} respectively. Their outputs at S4 is OUTMSG2= {305, 443, 321222323, 353366, PONG, 324329, 391, 352315, 311378319312317318, 406369}. **X is any system command. Figure 5.10. ISFSM model for IRC server (a) with backdoor (b) without backdoor.
The red transitions represent those actions triggered by the malicious input commands.
By comparing the two figures, we can tell that the backdoor function does not introduce 93 any extra state, but it adds self-loop transitions at each state – the backdoor commands execute at any state and the FSM stays at the same state after its execution.
We also mentioned in Chapter 5.1, learning an ISFSM will save huge space of the
Observation Table. In this particular example, we use 23.5% less of space when maintaining an observation table for an ISFSM than an FSM.
We also manage to synthesize the ISFSM of the backdoored IRC from a approximate model of a clean IRC instead of an empty FSM. We don’t really have the FSM of the clean IRC server since we are not the developer and examining the source code to build an FSM is not a practical practice. We collect some execution traces from a clean IRC server, and we reduce the traces to an ISFSM M0 using the algorithms in Chapter 5.2. We then apply Algorithm 5.1 to transfer M0 to OT0. We then start learning the ISFSM of the backdoored IRC server from OT0. Our experiment shows that we need 1213 queries to obtain Figure 5.10(a) if we start from an empty Observation Table, but only 409 queries if we start from OT0.
5.5.2 MSN Client with Message flooder
The message flooder continuously sends out random text messages to people on the contact list with a certain time interval between two messages.
Figure 5.11(a) and Figure 5.11(b) are the ISFSM for the MSN client with and without the message flooder, respectively. 94
(a) (b)
Figure 5.11. ISFSM model for the MSN client (a) with message flooder, (b) without message flooder.
Note that Figure 5.11(a) and Figure 5.11(b) only show the ISFSM of user idle phase after user login but before user initial any conversation with any other contact. Normally, there should not be any text messages (MSG message) sent to any contact as in Figure
5.11(b). The message flooder causes the MSN client to send out text messages. The message flooder also changes the behavior of the MSN client. MSN client without the message flooder would answer to only the first server challenge message (CHL message) at this idle stage, but with the message flooder, it will respond to multiple CHL messages.
95
CHAPTER 6: CONCLUSION
In this thesis, we propose formal methodology for studying security of network protocols. Formal methods fundamentally changed the manual, random and heuristic drawback of existing approaches. We study security issues of both protocol specification and implementation.
For protocol specification, a formal and automated approach for analysis of protocol security properties is proposed. Our approach is sound and complete. We find attack traces that lead to violation of the checked security property or prove the protocol specification is secure against the checked property. Trace inclusion transformation and online minimization are powerful tools for state space reduction. Our state space reduction techniques could be useful for other applications, which require significant state space reduction. The proposed framework is applicable to any system property as long as it can be formulated as a temporal property.
For protocol implementation, we first study model synthesis methods to create the formal model of a protocol implementation. Both active and passive methods are studied.
The formal synthesized model is then used to assist two different security and reliability flaw detection application. Our experiments show that behavior model greatly improve the coverage and efficiency of existing black box security testing approaches. To the best
96 of our knowledge, we are the first to propose an automated black box testing tool providing measurable coverage and universally applicable to all network protocols.
Extensive case studies show the effectiveness and efficiency of the proposed methods.
Also, our approach and framework are generic and could be adapted into different verification and security vulnerability detection application.
97
References
[1] [Online]. http://stub.bz/sslrebinding/
[2] aMSN - Opening your communication. [Online]. http://www.amsn-project.net/
[3] Dana Angluin, "Learning Regular Sets from Queries and Counterexamples,"
Information and Computation/information and Control, vol. 75, no. 2, pp. 87-106,
1987.
[4] James H. Barton, Edward W. Czeck, Zary Segall, and Daniel P. Siewiorek, "Fault
Injection Experiments Using FIAT," IEEE Transactions on Computers, vol. 39, no.
4, pp. 575-582, 1990.
[5] A.W. Biermann and J. A Feldman, "On the Synthesis of Finite-State Machines from
Samples of Their Behavior," IEEE Transactions on Computers, vol. 21, no. 6, pp.
592-597, 1972.
[6] Browser-Based Authentication (BBAuth) - YDN. [Online].
http://developer.yahoo.com/bbauth/
[7] David Brumley, Juan Caballero, Zhenkai Liang, James Newsome, and Dawn Song,
"Towards Automatic Discovery of Deviations in Binary Implementations with
Applications to Error Detection and Fingerprint Generation," in 16th USENIX
Security Symposium, 2007.
[8] Randal E. Bryant, "Graph-Based Algorithms for Boolean Function Manipulation,"
98
IEEE Transactions on Computers, vol. 35, no. 8, pp. 677-691, 1986.
[9] Jerry R. Burch, Edmund Clarke, Kenneth L. Mcmillan, and David L. Dill,
"Symbolic Model Checking: 1020 States and Beyond," in 5th IEEE LICS
Symposium on Logic in Computer Science, 1990, pp. 428-439.
[10] Michael Burrows, Martín Abadi, and Roger Needham, "A Logic of Authentication,"
ACM Transaction of Computer Systems, vol. 8, no. 1, pp. 18-36, February 1990.
[11] Frederick Butler, Iliano Cervesato, Aaron D. Jaggard, Andre Scedrov, and
Christopher Walstad, "Formal Analysis of Kerberos 5," Theoretical Computer
Science, vol. 367, no. 1, 2006.
[12] Juan Caballero, Pongsin Poosankam, Dawn Song, and Christian Kreibich,
"Dispatcher: Enabling Active Botnet Infiltration using Automatic Protocol Reverse-
Engineering," in 16th ACM Conference on Computer and Communications Security
(CCS 09), 2009, pp. 621-634.
[13] Juan Caballero, Heng Yin, Zhenkai Liang, and Dawn Song, "Polyglot: Automatic
Extraction of Protocol Message Format using Dynamic Binary Analysis," in 14th
ACM Conference on Computer and and Communications Security (CCS 07), 2007.
[14] Iliano Cervesato, Aaron D. Jaggard, Andre Scedrov, Joe-Kai Tsay, and Christopher
Walstad, "Breaking and Fixing Public-Key Kerberos," Information and
Computation, vol. 206, no. 2-4, February 2008.
[15] Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein,
Introduction to Algorithms, 1st ed.: MIT press, 1990.
99
[16] Olivier Coudert, Christian Berthet, and Jean Christophe Madre, "Verification of
Synchronous Sequential Machines Based on Symbolic Execution," in International
Workshop on Automatic Verification Methods for Finite State Systems, 1990, pp.
365-373.
[17] Steve Crites, Francis Hsu, and Hao Chen, "OMash: Enabling Secure Web Mashups
via Object Abstraction," in ACM Conference on Computer and Communications
Security (CCS 08), 2008.
[18] Weidong Cui, Jayanthkumar Kannan, and Helen J. Wang, "Discoverer : Automatic
Protocol Reverse Engineering from Network Traces," in 16th USENIX Security
Symposium, 2007.
[19] Weidong Cui, Marcus Peinado, Karl Chen, Helen J. Wang, and Luis Irun-Briz,
"Tupni: Automatic Reverse Engineering of Input Formats," in 15th ACM Conference
on Computer and Communications Security (CCS 08), 2008.
[20] Danny Dolev and Andrew C. Yao, "On the Security of Public Key Protocols," IEEE
Transaction on Information Theory, vol. IT-29, no. 2, pp. 198-208, March 1983.
[21] Flickr Services. [Online]. http://www.flickr.com/services/api/auth.spec.html
[22] Patrice Godefroid, Nils Klarlund, and Koushik Sen, "DART: Directed Automated
Random Testing," in 2005 ACM SIGPLAN conference on Programming language
design and implementation (PLDI), 2005, pp. 213-223.
[23] Patrice Godefroid, Michael Y. Levin, and David Molnar, "Automated Whitebox
Fuzz Testing," Microsoft, Technical Report MS-TR-2007-58, 2007.
100
[24] Lawrence A. Gordon, Martin P. Loeb, William Lucyshyn, and Robert Richardson,
"2005 CSI/FBI Computer Crime and Security Survey," 2005.
[25] Sezer Gören and F. Joel Ferguson, "On State Reduction in Incompletely Specified
Finite-State Machines," Computers and Electrical Engineering , vol. 33, no. 1, pp.
58-69, 2007.
[26] Ragib Hasan, Marianne Winslett, Richard Conlan, Brian Slesinsky, and Nandakumar
Ramani, "Please Permit Me: Stateless Delegated Authorization in Mashups," in 24th
Computer Security Application Conference (ACSAC 08), 2008, pp. 173-182.
[27] C. A. R. Hoare, Communicating Sequential Processes.: Prentice Hall, 1985.
[28] Jon Howell, Collin Jackson, Helen Wang, and Xiaofeng Fan, "MashupOS:
Operating System Abstractions for Client Mashups," in 11th USENIX Workshop on
Hot Topics in Operating Systems, 2007.
[29] Yating Hsu and David Lee, "Authentication and Authorization Protocol Security
Property Analysis with Trace Inclusion Transformation and Online Minimization,"
in International Conference on Network Protocols, 2010.
[30] Yating Hsu, Guoqiang Shu, and David Lee, "A Model-based Approach to Security
Flaw Detection of Network Protocol Implementation," in International Conference
on Network Protocols, 2008, pp. 114-123.
[31] Mei Lin Hui and Gavin Lowe, "Safe Simplifying Transformations for Security
Protocols or not just the Needham Schroeder Public Key Protocol," in 12th
Computer Security Foundations Workshop, 1999, pp. 32-43.
101
[32] Galen Hunt and Doug Brubacher, "Detours: Binary Interception of Win32
Functions," in USENIX Windows NT Symposium 99, 1999, pp. 135-143.
[33] Rauli Kaksonen, M. Laakso, and A. Takanen, "System Security Assessment through
Specification Mutations and Fault Injection," in IFIP TC6/TC11 International
Conference on Communications and Multimedia Security Issues of the New Century,
2001.
[34] Paris C. Kanellakis and Scott A. Smolka, "CCS Expressions, Finite State Processes,
and Three Problems of Equivalence," in 2nd ACM Symposium on Principles of
Distributed Computing, 1983, pp. 228-240.
[35] Richard A. Kemmerer, Catherine Meadows, and Jonathan K. Millen, "Three System
for Cryptographic Protocol Analysis," Journal of Cryptology, vol. 7, no. 2, pp. 79-
130, 1994.
[36] Frederik De Keukelaere, Sumeer Bhola, Michael Steiner, Suresh Chari, and Sachiko
Yoshihama, "SMash: Secure Component Model for Cross-Domain Mashups on
Unmodified Browsers," in International Conference on World Wide Web (WWW
08), 2008, pp. 534-544.
[37] Narayanan Kulathuramaiyer, "Mashups: Emerging Application Development
Paradigm for a Digital Journal," Journal of Universal Computer Science, vol. 13, no.
4, pp. 531-542, 2007.
[38] David Lee and Krishan Sabnana, "Reverse Engineering of Communication
Protocols," in IEEE International Conference of Network Protocols, 1993, pp. 208-
102
216.
[39] David Lee and Mihalis Yannakakis, "Online Minimization of Transition Systems,"
in 24th Annual ACM Symposium on Theory of Computing, 1992.
[40] David Lee and Mihalis Yannakakis, "Principles and Methods of Testing Finite State
Machine - A Survey," IEEE, vol. 84, no. 8, pp. 1090-1123, August 1996.
[41] Zhiqiang Lin, Xuxian Jiang, and Xiangyu Zhang, "Automatic Protocol Format
Reverse Engineering through Context-aware Monitored Execution," in 15 th
Network and Distributed System Security (NDSS), 2008.
[42] Zhiqiang Lin, Xiangyu Zhang, and Dongyan Xu, "Automatic Reverse Engineering
of Data Structures from Binary Execution," in 17th Network and Distributed System
Security Symposium, 2010.
[43] Zhiqiang Lin, Xiangyu Zhang, and Dongyan Xu, "Convicting Exploitable Software
Vulnerabilities: An Efficient Input Provenance Based Approach," in 38th IEEE/IFIP
Dependable Systems and Networks, 2008, pp. 247-256.
[44] Gavin Lowe, "Breaking and Fixing the Needham-Schroeder Public Key Protocol
using FDR," in 2nd International Workshop on Tools and Algorithms for
Construction and Analysis of Systems, 1996, pp. 147-166.
[45] Gavin Lowe, "Towards a Completeness Result for Model Checking of Security
Protocols," Journal of Computer Science, vol. 7, no. 2-3, pp. 89-146, 1999.
[46] John C. Mitchell, Mark Mitchell, and Ulrich Stern, "Automated Analysis of
Cryptographic Protocols Using Murphi," in IEEE Symposium on Security and
103
Privacy, 1997.
[47] John C. Mitchell, Vitaly Shmatikov, and Ulrich Stern, "Finite-State Analysis of SSL
3.0 and Related Protocols," in 7th USENIX Security Symposium, 1998.
[48] B. Clifford Neuman and Stuart G. Stubblebine, "A Note on the Use of Timestamps
as Nonces," ACM SIGOPS Operating Systems Review, vol. 27, no. 2, pp. 10-14,
1993.
[49] OAuth « hueniverse. [Online]. http://hueniverse.com/oauth/
[50] OAuth Community Site. [Online]. http://oauth.net/
[51] Peter Oehlert, "Violating Assumptions with Fuzzing," IEEE Security and Privacy,
vol. 3, no. 2, pp. 58-62, March-April 2005.
[52] Arlindo L. Oliveira and Stephen Edwards, "Limits of Exact Algorithms For
Inference of Minimum Size Finite State Machines," in 7th Workshop on Algorithmic
Learning Theory, 1996, pp. 59-66.
[53] OpenAuth. [Online]. http://dev.aol.com/api/openauth
[54] Ryan Paul. Compromising Twitter's OAuth security system. [Online].
http://arstechnica.com/security/guides/2010/09/twitter-a-case-study-on-how-to-do-
oauth-wrong.ars/2
[55] Doron Peled, Moshe Y. Vardi, and Mihalis Yannakakis, "Black Box Checking," in
IFIP TC6 WG6.1 Joint International Conference on Formal Description Techniques
for Distributed Systems and Communication Protocols (FORTE XII) and Protocol
Specification, Testing and Verification (PSTV XIX), 1999.
104
[56] Pidgin, the universal chat client. [Online]. http://www.pidgin.im/
[57] (2011, May) Researchers Spot Security Flaws in Google’s ClientLogin Protocol.
[Online]. http://www.livehacking.com/2011/05/17/researchers-spot-security-flaws-
in-googles-clientlogin-protocol/
[58] A.W Roscoe, "Proving Security Protocols With Model Checkers By Data
Independence Techniques," in 11th Computer Security Foundations Workshop,
1998, pp. 84-95.
[59] Peter Ryan and Steve Schneider, Modelling and Analysis of Security Protocols.:
Addison-Wesley, 2001.
[60] Guoqiang Shu, Yating Hsu, and David Lee, "Detecting Communication Protocol
Security Flaws by Formal Fuzz Testing and Machine Learning," in IFIP
International Conference on Formal Techniques for Networked and Distributed
Systems (FORTE), vol. 5048, 2008, pp. 299-304.
[61] Guoqiang Shu and David Lee, "Testing Security Properties of Protocol
Implementations – a Machine Learning Based Approach," in International
Conference on Distributed Computing Systems (ICDCS), 2007.
[62] Joseph Sifakis, "A Unified Approach for Studying the Properties of Transition
Systems," Theoretical Computer Science, vol. 18, no. 3, pp. 227-258, 1982.
[63] United States Computer Emergency Readiness Team (US-CERT). [Online].
http://www.us-cert.gov/
[64] UnrealIRCd. [Online]. http://www.unrealircd.com/
105
[65] Using the Meetup API. [Online]. http://www.meetup.com/meetup_api/
[66] Antti Valmari, "The State Explosion Problem," in Lectures on Petri Nets I: Basic
Models, Advances in Petri Nets, 1996, pp. 429-528.
[67] vsftpd. [Online]. https://security.appspot.com/vsftpd.html
[68] Gilbert Wondracek, Paolo Milani Comparetti, Christopher Kruegel, and Engin
Kirda, "Automatic Network Protocol Analysis," in 15th Network and Distributed
System Security Symposium (NDSS 08), 2008.
[69] Mihalis Yannakakis and David Lee, "An Efficient Algorithm for Minimizing Real-
Time Transition Systems," Formal Methods in System Design - Special issue on
Computer Aided Verification, vol. 11, no. 2, Auguest 1997.
[70] Kacem Zeroual and M. El Yassini, "A Protocol Synthesis Algorithm: a Relational
Approach," in IEEE International Conference on Network Protocols, 1995.
106