Proceedings
Foundations of Computer Security Affiliated with LICS’02
FLoC’02
Copenhagen, Denmark July 25–26, 2002
Edited by Iliano Cervesato
With support from Office of Naval Research International Field Office
Table of Contents
Preface ...... iii Workshop Committees ...... v
Foundations of Security
On the Decidability of Cryptographic Protocols with Open-ended Data Structures ...... 3 Ralf Küsters
Game Strategies In Network Security ...... 13 Kong-Wei Lye and Jeannette M. Wing
Modular Information Flow Analysis for Process Calculi ...... 23 Sylvain Conchon
Logical Approaches
A Trustworthy Proof Checker ...... 37 Andrew W. Appel, Neophytos Michael, Aaron Stump, and Roberto Virga
Finding Counterexamples to Inductive Conjectures ...... 49 Graham Steel, Alan Bundy, and Ewen Denney
Automatic SAT-Compilation of Protocol Insecurity Problems via Reduction to Planning ...... 59 Alessandro Armando and Luca Compagna
Invited Talk
Defining security is difficult and error prone ...... 71 Dieter Gollmann
i ii TABLE OF CONTENTS
Verification of Security Protocols
Identifying Potential Type Confusion in Authenticated Messages ...... 75 Catherine Meadows
Proving Cryptographic Protocols Safe From Guessing Attacks ...... 85 Ernie Cohen
Programming Language Security
More Enforceable Security Policies...... 95 Lujo Bauer, Jarred Ligatti and David Walker
A Component Security Infrastructure ...... 105 Yu David Liu and Scott F. Smith
Static Use-Based Object Confinement ...... 117 Christian Skalka and Scott F. Smith
Panel
The Future of Protocol Verification ...... 129 Serge Auxetier, Iliano Cervesato and Heiko Mantel (moderators)
Author Index ...... 130 Preface
Computer security is an established field of Computer Science of both theoretical and practical sig- nificance. In recent years, there has been increasing interest in logic-based foundations for various methods in computer security, including the formal specification, analysis and design of crypto- graphic protocols and their applications, the formal definition of various aspects of security such as access control mechanisms, mobile code security and denial-of-service attacks, and the modeling of information flow and its application to confidentiality policies, system composition, and covert channel analysis. This workshop continues a tradition, initiated with the Workshops on Formal Methods and Security Protocols — FMSP — in 1998 and 1999 and then the Workshop on Formal Methods and Computer Security — FMCS — in 2000, of bringing together formal methods and the security community. The aim of this particular workshop is to provide a forum for continued activity in this area, to bring computer security researchers in contact with the FLoC community, and to give FLoC attendees an opportunity to talk to experts in computer security. Given the affinity of themes, FCS was synchronized with the FLoC’02 Verification Workshop (VERIFY). Sessions with a likely overlap in audience were held jointly. Moreover, authors who thought their paper to be of interest for both FCS and VERIFY could indicate that it be considered a joint submission, and it was reviewed by members of both program committees. FCS received 22 submissions, 10 of which were joint with VERIFY. The review phase selected 11 of them for presentation; 5 of these were joint with VERIFY. This unexpected number of papers lead to extending FCS by one day. Many people have been involved in the organization of the workshop. John Mitchell, assisted by the Organizing Committee, is to be thanked for bringing FCS into existence as part of FLoC. The Program Committee did an outstanding job selecting the papers to be presented, in particular given the short review time. We are very grateful to the VERIFY chairs, Heiko Mantel and Serge Autexier, for sharing the organizational load and for the numerous discussions. Sebastian Skalberg, Henning Makholm and Klaus Ebbe Grue, our interface to FLoC, turned a potential bureaucratic nightmare into a smooth ride. Finally we are grateful to the authors, the panelists and the attendees who make this workshop an enjoyable and fruitful event.
Iliano Cervesato FCS’02 Program Chair
iii iv PREFACE Workshop Committees
Program Committee
Iliano Cervesato (chair), ITT Industries, USA Véronique Cortier, ENS Cachan, France Grit Denker, SRI International, USA Carl Gunter, University of Pennsylvania, USA Alan Jeffrey, DePaul University, USA Somesh Jha, University of Wisconsin — Madison, USA Trevor Jim, AT&T Labs, USA Heiko Mantel, DFKI Saarbrücken, Germany Catherine Meadows, Naval Research Laboratory, USA Flemming Nielson, Technical University of Denmark Birgit Pfitzmann, IBM Zürich, Switzerland David Sands, Chalmers University of Technology, Sweden Stephen Weeks, InterTrust, USA
Organizing Committee
Martín Abadi, University of California — Santa Cruz, USA Hubert Comon, ENS Cachan, France Joseph Halpern, Cornell University, USA Gavin Lowe, Oxford University, UK Jonathan K. Millen, SRI International, USA Michael Mislove, Tulane University, USA John Mitchell (chair), Stanford University, USA Bill Roscoe, Oxford University, UK Peter Ryan, University of Newcastle upon Tyne, UK Steve Schneider, Royal Holloway University of London, UK Vitaly Shmatikov, SRI International, USA Paul Syverson, Naval Research Laboratory, USA Michael Waidner, IBM Zürich, Switzerland Rebecca Wright, AT&T Labs, USA
v vi WORKSHOP COMMITTEES Session I
Foundations of Security
1
On the Decidability of Cryptographic Protocols with Open-ended Data Structures Ralf Küsters Institut für Informatik und Praktische Mathematik Christian-Albrechts-Universität zu Kiel, Germany [email protected]
Abstract receive-send action; see Section 2 for concrete examples. This paper addresses open-ended protocols, and thus, Formal analysis of cryptographic protocols has mainly deals with one of the challenges pointed out by Meadows concentrated on protocols with closed-ended data struc- [17]. The goal is to devise a protocol model rich enough tures, where closed-ended data structure means that the to capture a large class of open-ended protocols such that messages exchanged between principals have fixed and fi- security is decidable; the long-term goal is to develop tools nite format. However, in many protocols the data struc- for automatic verification of open-ended protocols. tures used are open-ended, i.e., messages have an un- Open-ended protocols make it necessary to model prin- bounded number of data fields. Formal analysis of pro- cipals who can perform in one receive-send action an un- tocols with open-ended data structures is one of the chal- bounded number of internal actions; only then can they lenges pointed out by Meadows. This work studies de- handle open-ended data structures. Therefore, the first cidability issues for such protocols. We propose a proto- problem is to find a good computational model for receive- col model in which principals are described by transduc- send actions. It turns out that one cannot simply ex- ers, i.e., finite automata with output, and show that in this tend the existing models. More specifically, Rusinowitch model security is decidable and PSPACE-hard in presence and Turuani [21] describe receive-send actions by single of the standard Dolev-Yao intruder. rewrite rules and show security to be NP-complete. In this model, principals have unbounded memory. Furthermore, 1 Introduction the terms in the rewrite rules may be non-linear, i.e., multi- ple occurrence of one variable is allowed, and thus, a prin- Formal methods are very successful in analyzing the se- cipal can compare messages of arbitrary size for equality. curity of cryptographic protocols. Using these methods, To handle open-ended protocols, we generalize the model many flaws have been found in published protocols. By by Rusinowitch and Turuani in a canonical way and show now, a large variety of different methods and tools for that if receive-send actions are described by sets of rewrite cryptographic protocol analysis is available (see [17] for rules, security is undecidable, even with i) finite memory an overview). In particular, for different interesting classes and non-linear terms, or ii) unbounded memory and linear of protocols and intruders, security has been shown to be terms. Consequently, we need a computational model in decidable, usually based on the Dolev-Yao model [7] (see which principals have finite memory and cannot compare the paragraph on related work). messages of arbitrary size for equality. Previous work has mostly concentrated on protocols For this reason, we propose to use transducers, i.e., fi- with closed-ended data structures, where messages ex- nite automata with output, as the computational model for changed between principals have fixed and finite format. receive-send actions, since transducers satisfy the above In what follows, we will refer to these protocols as closed- restrictions — they have finite memory and cannot com- ended protocols. In many protocols, however, the data pare messages of arbitrary size for equality —, and still structures are open-ended: the exchanged messages may can deal with open-ended data structures. In Section 5.1 have an unbounded number of data fields that must be pro- our so-called transducer-based model is discussed in de- cessed by a principal in one receive-send action, where tail. The main technical result of this paper is that receive-send action means that a principal receives a mes- in the transducer-based model, security is decidable and sage and reacts, after some internal computation, by send- PSPACE-hard under the following assumptions: the num- ing a message. One can, for example, think of a message ber of sessions is bounded, i.e., a protocol is analyzed that consists of an a priori unbounded sequence of requests assuming a fixed number of interleaved protocol runs; and a server who needs to process such a message in one nonces and complex keys are not allowed. We, however,
3 4 RALF KÜSTERS put no restrictions on the Dolev-Yao intruder; in particu- model for describing open-ended protocols (Section 3). In lar, the message size is unbounded. These are standard as- this model, receive-send actions can be arbitrary computa- sumptions also made in most decidable models for closed- tions. In Section 4, we consider the instances of the generic ended protocols [21, 2, 13].1 Just as in these works, the model in which receive-send actions are specified by sets security property we study in the present paper is secrecy. of rewrite rules, and show the mentioned undecidability The results indicate that from a computational point of result. The transducer-based model, the instance of the view, the analysis of open-ended protocols is harder than generic protocol model in which receive-send actions are for closed-ended protocols, for which security is “only” given by transducers, is introduced in Section 5. This sec- NP-complete [21]. The additional complexity comes from tion also contains the mentioned discussion. In Section 6 the fact that now we have, beside the Dolev-Yao intruder, the actual decidability and complexity results are stated. another source of infinite behavior: the unbounded num- Finally, we conclude in Section 7. ber of internal actions (i.e., paths in the transducers of un- bounded length). This makes it necessary to devise new Due to space limitations, in this paper we have largely proof techniques to show decidability. Roughly speaking, omitted technical details and rather focused on the intro- using that transducers only have finite memory we will use duction and the discussion of our models. We only provide a pumping argument showing that the length of paths in the proof ideas of our results. The full proofs can be found the transducers can be bounded in the size of the problem in the technical report [14]. It also contains a description instance. of the recursive authentication protocol (see Section 2) in our transducer-based model. Related work. All decidability and undecidability re- sults obtained so far only apply to closed-ended proto- cols. Decidability depends on the following parameters: 2 Examples of Open-ended Proto- bounded or unbounded number of sessions, bounded or cols unbounded message size, absence or presence of pairing, nonces, and/or complex keys. An example of an open-ended protocol is the IKE Proto- Usually, if one allows an unbounded number of ses- col [12], in which a principal needs to pick a security as- sions, security is undecidable [1, 8, 2, 9]. There are sociation (SA), the collection of algorithms and other in- only a few exceptions: For instance, if the message formations used for encryption and authentication, among size is bounded and nonces are disallowed, security is an a priori unbounded list of SAs. Such a list is an open- EXPTIME-complete [8]; if pairing is disallowed, security ended data structure, since it has an unbounded number is in P [6, 2]. The situation is much better if one puts a of data fields to be examined by a principal. An attack bound on the number of sessions; from results shown by on IKE, found by Zhou [24] and independently Ferguson Lowe [15] and Stoller [23] it follows that, under certain and Schneier [10], shows that when modeling open-ended conditions, one can assume such bounds without loss of protocols, the open-ended data structures must be taken generality. With a bounded number of sessions and with- into account, since otherwise some attacks might not be out nonces, security is decidable even if pairing is allowed found. In other words, as also pointed out by Meadows and the message size is unbounded [21, 2, 13]. In fact, in [17], open-endedness is security relevant. this setting, security is NP-complete, with [21] or without Other typical open-ended protocols are group proto- [2] complex keys. We make exactly the same assumptions cols, for example, the recursive authentication protocol in our models, where we use atomic keys. (RA protocol) [5] and the A-GDH.2 protocol [3], which To the best of our knowledge, the only contributions on is part of the CLIQUES project [22]. In the RA protocol, a formal analysis of open-ended protocols are the follow- key distribution server receives an a priori unbounded se- ing: The recursive authentication protocol [5] has been an- quence of request messages (containing pairs of principals alyzed by Paulson [19], using the Isabelle theorem prover, who want to share session keys) and must generate a cor- as well as by Bryans and Schneider [4], using the PVS the- responding sequence of certificates (containing the session orem prover; the A-GDH.2 protocol [3] has been analyzed keys). These sequences are open-ended data structures: In by Meadows [16] with the NRL Analyzer, and manually one receive-send action the server needs to process an un- by Pereira and Quisquater [20], based on a model simi- bounded number of data fields, namely the sequence of lar to the strand spaces model. As mentioned, decidability pairs of principals. Group protocols often allow an un- issues have not been studied so far. bounded number of receive-send actions in one protocol run. In our models, we will, however, always assume a Structure of the paper. In Section 2, we give exam- fixed bound on the number of receive-send actions, since ples of open-ended protocols. We then define a generic otherwise, just as in the case of an unbounded number of sessions, security, in general, leads to undecidability. Nev- 1In [21, 13], however, complex keys are allowed. ertheless, even with such a fixed bound it is still necessary ON THE DECIDABILITY OF CRYPTOGRAPHIC PROTOCOLS WITH OPEN-ENDED DATA STRUCTURES 5 to model open-ended data structures. In the RA protocol, a These are standard assumptions also made in decidable bound on the number of receive-send actions would imply models for closed-ended protocols. They coincide with that the sequence of requests generated by the principals the ones in [2], and except for complex keys, with those in is bounded. Nevertheless, the intruder can generate arbi- [21, 13]. trarily long request messages. Thus, the data structures are Let us now give a formal definition of the generic pro- still open-ended, and the server should be modeled in such tocol model. a way that, as in the actual protocol, he can process open- ended data structures. 3.1 Messages
In [14], we provide a formal description of the RA pro- ¡ tocol in our transducer-based model. The definition of messages is rather standard. Let denote a finite set of atomic messages, containing keys,
names of principals, etc. as well as the special atomic mes- ¡
3 A Generic Protocol Model sage ¢¤£¦¥¨§©£ . The set of messages (over ) is the least set that satisfies the following properties:
Our generic protocol model and the underlying assump-
¡
tions basically coincide with the ones proposed by Rusi- ;
nowitch et al. [21] and Amadio et al. [2] for closed-ended if ¤ , then ;
protocols. However, the important difference is that in the
¡
generic model, receive-send actions are, roughly speaking,
if and , then enc ;
binary relations over the message space, and thus can be
interpreted as arbitrary computations. In the models of if , then hash . Rusinowitch et al. and Amadio et al. , receive-send actions As usual, concatenation is an associative operation, i.e.,
are described by single rewrite rules or processes without
#$ % !©" . Note that we only allow for
loops, respectively. atomic keys, i.e., in a message enc '&( , is always an Thus, the generic protocol model is a very general
atomic message. -,/.
framework for open-ended protocols. In fact, it is much *+
# )0 Let ) denote the empty message and
too general to study decidability issues. Therefore, in sub- ) the set of messages containing ) . Note that is not allowed
sequent sections we will consider different instances of 4* inside encryptions or hashes, that is, enc 132 and
this model. 7* hash 562 . The main features of the generic protocol model can be
Later, we will consider terms, i.e., messages with vari-
.:9<; 9
summarizes as follows: +
# :=>=>=: 0
ables. Let 8 be a set of variables.
8 Then a term B (over ) is a message over the atomic mes-
a generic protocol is described by a finite set of prin- ¡ ,
cipals; sages 8 , where variables are not allowed as keys, i.e., 9
terms of the form enc C'&( for some variable are forbid-
*
8 B
the internal state space of a principal may be infinite den. A substitution D is a mapping from into . If is B
(which, for example, enables a principal to store arbi- a term, then DE!B¤ denotes the message obtained from by
9 9
DE
trarily long messages); replacing every variable in B by .
FG£GH: JIK!B¤ B
The depth of a term is the maximum number every principal is described by a finite sequence of
of nested encryptions and hashes in B , i.e.,
receive-send actions;
¡
+ + ,
#ML FG£GH: JIKK #NL O 8 FG£GH: JI%)G , for every ,
receive-send actions are arbitrary computations.
+ .
#QPSR:T FU£ We make the following assumptions: + B¤¤ #NFU£ the intruder is the standard Dolev-Yao intruder; in + !B¤\ #MFU£ principals and the intruder cannot generate new 3.2 The Intruder Model nonces, i.e., the nonces used in the analysis are only We use the standard Dolev-Yao intruder model [7]. That is, those already contained in the protocol description; an intruder has complete control over the network and can keys are atomic; derive new messages from his current knowledge by com- posing, decomposing, encrypting, decrypting, and hashing the number of sessions is bounded. More precisely, messages. As usual in the Dolev-Yao model, we make the the sessions considered in the analysis are only those perfect cryptography assumption. We do not impose any encoded in the protocol description itself. restrictions on the size of messages. 6 RALF KÜSTERS ¡ The (possibly infinite) set of messages F the intruder flexible principals: for instance, those that nondeterminis- 7* can derive from ¡ is the smallest set satisfying the tically choose one principal, who they want to talk to, or following conditions: one SA from the list of SAs in the IKE Protocol. ¡ ¡ We also remark that a protocol is not parametrized by F ; . In particular, when we say that is secure, we mean ¡ ¡ ¡ that is secure given the principals as defined in the pro- F F F if , then and (decomposition); tocol. We do not mean that is secure for every number of principals. ¡ ¡ ¡ F F F if enc and , then (decryption); 3.4 Attacks on Protocols ¡ ¡ ¡ F F F if and , then In an attack on a protocol, the receive-send actions of the (composition); principals are interleaved in some way and the intruder, ¡£¢ ¡ ¡ who has complete control over the communication, tries F # 2 ) F if , , and , then ¡ to produce inputs for the principals such that from the cor- F enc (encryption); responding outputs and his initial knowledge he can derive ¡ ¡ F #)2 ! F if and , then hash the secret message ¢¤£¦¥¨§©£ . Formally, an attack is defined (hashing). as follows. ? . ¡ # E ¤ 0 3.3 Protocols Definition 2 Let be a generic pro- # ¦¥ ¨§ © ¨ tocol with ¤ , for . An attack on Protocols are described by sets of principals and every is a tuple consisting of the following components: principal is defined by a sequence of receive-send actions, . X "!#$%E &'© 0 which, in a protocol run, are performed one after the other. a total ordering on the set X (QX V© )* V such that implies (the execution Since we are interested in attacks, the definition of a proto- 2 col also contains the initial intruder knowledge. Formally, order of the receive-send actions); principals and protocols are defined as follows. X ,- .%© a mapping + assigning to every , , , a tuple ¦¥ ¨§ ¨©' Definition 1 A generic principal ¤ is a tuple where A 0 +6X # /10 ¤20 ¤ ¨/10¨3 ¤ ¥ is the (possibly infinite) set of states of ; with A ¤ § is the set of initial states of ; /10 ¨/10¨3 4¥5 ¤$ – (the state of before/after per- © \7 is the number of receive-send actions to be per- forming 6 ); and 4* formed by ¤ ; 0 80 \ – (the input message received and . L :=>=>=J © [G0 is a mapping assigning to every output message sent by ); * * ¥ ¥ a receive-send action . such that . ? ¡ ; E ¤"0 A generic protocol is a tuple where / 9§ : for every ; is the number of principals; ¡ ,N. 0¨; 20 F ! < = X X0¨ ? . for every ; 0 ¤ is a family of generic principals, and 8 ©? > , ; A ¡ * is the initial intruder knowledge. 0 . 6\ ¦/10 ¤20 ¤ /10¨3 @ 8>© for every , . 3, . Note that receive-send actions are arbitrary relations. In- ¡ 0 ¢\£:¥<§ £> F !A An attack is called successful if tuitively, they take an input message (2. component) E 8 © 0 . and nondeterministically, depending on the current state (1. component), return an output message (3. component) The decision problem we are interested in is the following: plus a new state (4. component). Later, when we consider instances of the generic protocol model, one receive-send ATTACK: Given a protocol , decide whether there exists action of a principal will consist of an unbounded number a successful attack on . of internal actions. By allowing receive-send actions to 2Although, we assume a linear ordering on the receive-send actions be nondeterministic and principals to have a set of initial performed by a principal, we could as well allow partial orderings (as in states, instead of a single initial state, one can model more [21]) without any impact on the decidability and complexity results. ON THE DECIDABILITY OF CRYPTOGRAPHIC PROTOCOLS WITH OPEN-ENDED DATA STRUCTURES 7 A protocol guarantees secrecy if there does not exist a suc- Theorem 3 For rule-based protocols, ATTACK is unde- cessful attack. In this case, we say that the protocol is cidable. secure. Whether ATTACK is decidable or not heavily depends on By reduction from Post’s Correspondence Problem (PCP), what kinds of receive-send actions a principal is allowed this theorem is easy to show. It holds true, even for pro- to perform. In the subsequent sections, we look at differ- tocols consisting of only one principal, which may only ent instances of generic protocols, i.e., different computa- perform one receive-send action. In other words, the un- tional models for receive-send actions, and study the prob- decidability comes from the internal actions alone. lem ATTACK for the classes of protocols thus obtained. However, the reduction does not work if only linear terms are allowed in rewrite rules. In linear terms, every variable occurs at most once, and therefore, one cannot 4 Undecidability Results compare submessages of arbitrary size for equality. Nev- ertheless, if principals can store one message and compare We extend the model proposed by Rusinowitch and Tu- it with a submessage of the message being processed, we ruani [21] in a straightforward way such that open-ended still have undecidability. Such protocols are called linear- protocols can be handled, and show that this extension term one-memory protocols; see [14] for the formal defi- leads to undecidability of security. nition and the proof of undecidability, which is again by a The model by Rusinowitch and Turuani can be consid- rather straightforward reduction from PCP. ered as the instance of the generic protocol model in which receive-send actions are described by single rewrite rules Theorem 4 For linear-term one-memory protocols, AT- 3 TACK is undecidable. ¡ B¤ B B¤ of the form B , where and are terms. The in- ternal state of a principal is given implicitly by the values assigned to the variables occurring in the rewrite rules – 5 The Transducer-based Protocol different rules may share variables. In particular, a prin- cipal has unbounded memory to store information for use Model in subsequent receive-send actions. Roughly speaking, a The previous section indicates that, informally speaking, message is transformed by a receive-send action of the when principals can process open-ended data structures ¢ B' DE!B¤% D form B into the message , where is a substitu- and, in addition, can # DEB¤ tion satisfying . In [21], it is shown that in this setting, ATTACK is NP-complete. 1. compare submessages of arbitrary size (which is pos- Of course, in this model open-ended data structures can- sible if terms are not linear), or not be handled since the left hand-side B of a rewrite rule has a fixed and finite format, and thus, one can only pro- 2. store one message and compare it with a submessage cess messages with a fixed number of data fields. of the message being processed, A natural extension of this model, which allows to deal then security is undecidable. To obtain decidability, we with open-ended data structures, is to describe receive- need a device with only finite memory, and which does not send actions by sets of rewrite rules, which can nondeter- allow to compare messages of arbitrary size. This moti- ministically be applied to the input message, where, as in vates to use transducers to describe receive-send actions. the model of Rusinowitch and Turuani, rewriting means In what follows, we define the corresponding instance of £ B¤ top-level rewriting: If the rule B is applied to the in- the generic protocol model. In Section 5.1, we will dis- DEB¤% put message yielding as output, another rule (non- cuss capabilities and restrictions of our transducer-based deterministically chosen from the set of rules) may be ap- model. ¤ ¤£¥ plied to DEB'% . To the resulting output yet another rule may If is a finite alphabet, will denote the set of finite be applied and so on, until no rule is or can be applied ¤ words over , including the empty word ) . anymore. The applications of the rules are the internal ¤ ¦¥ ¨§6 § © actions of principals. The instance of the generic proto- Definition 5 A transducer ¦ is a tuple col model in which receive-send actions are described by where sets of rewrite rules as described above is called rule-based ¦ ¥ is the finite set of states of ; protocol model. In [14], we give a formal definition of this ¤ model. In this model, we distinguish between input, out- is the finite input alphabet; put, and process rules, and also put further restrictions on the rewrite rules such that they can be applied only a finite § is the finite output alphabet; (but a priori unbounded) number of times. ¥ ¦ § is the set of initial states of ; 3 ¥ Since Rusinowitch and Turuani allow complex keys, the terms are ¤ ¥ ¥ § .¥ more general than the ones we use here. However, we will only consider © is the finite set of transitions terms as defined in Section 3.1. of ¦ ; and 8 RALF KÜSTERS ¥ ¦ is the set of final states of . attack, the input to a message transducer is always a mes- sage. The second property imposes some restriction on the ¦ ¡ / A path (of length ) in from to is of the form ; ; ; A A A ?@ A ?K@A ? ; 9 9 9 “internal behavior” of a message transducer. Both proper- £¢ / £¢ /¥¤E=>=:=Z ¦¢ / / #§¡ / with , ? A 9 ties do not seem to be too restrictive. They should be satis- #£/ / £¢ ¨/ © / , and for every ; ; ?@A 9 9 3 fied for most protocols; at least they are for the transducers ©¨ L is called strict if , and and are non- ; ?@A 9 9 in the model of the recursive authentication protocol (as empty words. The word =>=>= is the input label and ?@ A ; described in [14]). =:=>= ¢ ¢ is the output label of . A path of length 9 An open issue is whether the properties on the internal ) ¡ £¢S / ¦ L has input and output label . We write 9 and external behavior are decidable, i.e., given a transducer ¤ ¦¢S / ¦ ¡ / (¡ ) if there exists a (strict) path from to 9 over does it satisfy 1. and 2. of Definition 7. The main ¢ in ¦ with input label and output label . . 9 + problem is the quantification over messages, i.e., over a ¦ ¥ ¦ £S # ¡Z £¢ ¨/< !¡ ¨/ If , then 9 ¥ ¤ ¥ context-free rather than a regular set. Nevertheless, in the ¡ ¦¢S / ¦ 0 ¥ § &¥ ¦ . The output of 9 + . 9 ¤ ¥ model of the recursive authentication protocol it is easy to ¦ # ¢ ! on input is defined by there exists 9 see that the transducers constructed satisfy the properties. ¢ + 9§ / ¡Z ¦¢ ¨/< ¦¦§ ¨ X0 ¡ and with . ¦ ¥ E£S # ¦O ¦S ¦¥ , . , . ¤ For , we define 4* ¤ ¥ ¥ )0¨ § )0¨ ¥ ¦ If © , is called ¥ . By the definition of message transducers, 4* transducer with letter transitions in contrast to transducers 4* S§ ¦¥% ¥ § if is the set of initial with word transitions. The following remark shows that it ¦ states and is the set of final states of . Thus, message suffices to consider transducers with letter transitions. transducers specify receive-send actions of principals (in ¤ the sense of Definition 1) in a natural way. # ¦¥ ¨§6 § ©O¨ Remark 6 Let ¦ be a transducer. ¤ In order to define one principal (i.e., the whole sequence # ¦¥ 1 §6¨§ ¨© 1 Then there exists a transducer ¦ of receive-send actions a principal performs) by a single ¥ ¦ 1 £6 # with letter transitions such that ¥ , and transducer, we consider so-called extended message trans- ¦ £6 ¦ ¥ ; ? ¤ for every . ¦ # ¦¥ © :§ :=>=>=:¨§ ¤ ducers: is an extended mes- A + ¤ !#" !$&% # ¦¥ ¨§ ¨© ¨§ In order to specify the receive-send actions of a principal, sage-transducer if ¦ is a 0 0¨3 we consider special transducers, so-called message trans- message transducer for all 2> . Given such an extended ; ¦¥ § ¨E¨ ducers, which satisfy certain properties. Message trans- message transducer, it defines the principal A V #')( ( § § @ !$+% ducers interpret messages as words over the finite alphabet with !#* for . In this set- ¤ 0¨3 , consisting of the atomic messages as well as the let- ting, an internal action of a0 principal corresponds to apply- ters “enc ”, “hash ”, and “)”, that is, ing one transition in the extended message transducer. ¡ ¡ ,3. + ,3. ¤ Definition 8 A transducer-based protocol is a generic K:!¨ 0 ' J0U= # enc hash protocol where the principals are defined by extended mes- Messages considered as words over ¤ have always a bal- sage transducers. anced number of opening parentheses, i.e., “enc ” and “hash ”, and closing parentheses, i.e., “)”. 5.1 Discussion of the Transducer-based Pro- A message transducer reads a message (interpreted as a tocol Model word) from left to right, thereby producing some output. If messages are considered as finite trees (where leaves In this section, we aim at clarifying capabilities and lim- are labeled with atomic messages and internal nodes are itations of the transducer-based protocol model. To this labeled with the encryption or hash symbol), a message end, we compare this model with the models usually used transducer traverses such a tree from top to bottom and for closed-ended protocols. To make the discussion more from left to right. concrete, we concentrate on the model proposed by Rusi- nowitch and Turuani (see Section 4), which, among the ¡ Definition 7 A message transducer ¦ (over ) is a tu- decidable models used for closed-ended protocols, is very ¤ ¤ ¤ ¥ ¨§ ¨© ¨ ¥ ¨§ ¨© ple such that is a powerful. In what follows, we refer to their model as the transducer with letter transitions, and rewriting model. As pointed out in Section 3, the main dif- * * ference between the two models is the way receive-send ¦ 1. for every , ; and actions are described. In the rewriting model receive-send ¤ ¥ actions are described by single rewrite rules and in the ¨/ &¥ ¡ £ / 2. for all ¡ , , and , if 7* transducer-based model by message transducers. ¦ , then . Let us start to explain the capabilities of message trans- The first property is a condition on the “external behavior” ducers compared to rewrite rules. of a message transducer: Whenever a message transducer gets a message as input, then the corresponding outputs are Open-ended data structures. As mentioned in Sec- also messages (rather than arbitrary words). Note that in an tion 4, with a single rewrite rule one cannot process an ON THE DECIDABILITY OF CRYPTOGRAPHIC PROTOCOLS WITH OPEN-ENDED DATA STRUCTURES 9 unbounded number of data fields. This is, however, possi- copied. Again, a transducer would need to store in ble with transducers. some way, which is not possible because of the finite For example, considering the IKE protocol (see Sec- memory. As illustrate above, a transducer could how- ¡ K tion 2), it is easy to specify a transducer which i) reads a ever simulate a rule such as enc hash . list of SAs, each given as a sequence of atomic messages, 4. Linear terms: A transducer cannot simulate all ii) picks one SA, and iii) returns it. With a single rewrite rewrite rules with linear left and right hand-side. rule, one could not parse the whole list of SAs. K£¢ Consider for example the rule enc The transducer-based model of the recursive authenti- ¤¢ ¢ hash , where and are variables, and is cation protocol (described in [14]) shows that transducers an atomic message. Since in the output, the order of can also handle more involved open-ended data structures: and is switched, a transducer would have to store The server in this protocol generates a sequence of certifi- the messages substituted for and . However, this cates (containing session keys) from a request message of A ? ; requires unbounded memory. &:&>& &>&>&X the form hash hash hash , where the ’s are sequences of atomic messages and the nesting The undecidability results presented in Section 4 indicate depth of the hashes is a priori unbounded (see [14] for the that, if open-ended data structures are involved, the restric- exact definition of the messages.) tions 1. and 2. seem to be unavoidable. The question is Of course, a transducer cannot match opening and clos- whether this is also the case for the remaining two restric- ing parenthesis, if they are nested arbitrarily deep, since tions. We will comment on this below. messages are interpreted as words. However, often this is First, let us point out some work-arounds. In 1., it not necessary: In the IKE protocol, the list of SAs is a mes- often (at least under reasonable assumptions) suffices to sage without any nesting. In the recursive authentication store atomic messages such as principal names, keys, and protocol, the structure of request messages is very sim- nonces. Thus, one does not always need unbounded mem- ple, and can be parsed by a transducer. Note that a trans- ory. One example is the recursive authentication protocol. ducer does not need to check whether the number of clos- In 4., it might be possible to modify the linear terms such ing parenthesis in the request message matches the number that they can be parsed by a message transducer, and such of hashes because all words sent to a message transducer that the security of the protocol is not affected. In the ex- (by the intruder) are messages, and thus, well-formed. ample, if one changes the order of and in the output, the rewrite rule can easily be simulated by a transducer. Simulating rewrite rules. Transducers can simulate cer- Finally, a work-around for the restrictions 2. to 4., is to put tain receive-send actions described by single rewrite rules. a bound on the size of messages that can be substituted for ¡ the variables. This approach is usually pursued in protocol K Consider for example the rule enc hash , ¡ analysis based on finite-state model checking (e.g., [18]), where is a variable and an atomic message: First, the ¡ where, however, transducers have the additional advantage transducer would read “enc ” and output “hash ”, and then read, letter by letter, the rest of the input message, i.e., of being able to process open-ended data structures. For messages of bounded size, all transformations performed “ ” – more precisely, the message substituted for – and simultaneously write it into the output. by rewrite rules can also be carried out by message trans- ducers. Moreover, in this setting message transducers can handle type flaws. Let us now turn to the limitations of the transducer-based Of course, it is desirable to avoid such work-arounds if model compared to the rewriting model. The main limita- possible to make the analysis of a protocol more precise tions are the following: and reliable. One approach, which might lift some of the 1. Finite memory: In the rewriting model, principals can restrictions (e.g., 3. and 4.), is to consider tree transducers store messages of arbitrary size to use them in subse- instead of word transducers to describe receive-send ac- quent receive-send actions. This is not possible with tions. It seems, however, necessary to devise new kinds of transducers, since they only have finite memory. tree transducers or extend existing once, for example tree transducers with look-ahead, that are especially tailored 2. Comparing messages: In the rewriting model, prin- to modeling receive-send actions. A second approach is cipals can check whether submessages of the input to combine different computational models for receive- ¡ message coincide. For example, if BS# hash , send actions. For instance, a hybrid model in which some ¡ with an atomic message and a variable, a prin- receive-actions are described by rewrite rules and others cipal can check whether plain text and hash match. by transducers might still be decidable. Transducers cannot do this. 3. Copying messages: In the rewriting model, principals 6 The Main Result can copy messages of arbitrary size. For example, in ¡ K the rule enc hash , the message is The main technical result of this paper is the following: 10 RALF KÜSTERS Theorem 9 For transducer-based protocols, ATTACK is labels satisfy the conditions. (Note that for every message ¡ ¡ 4* F % decidable and PSPACE-hard. and finite set , can be decided.) In particular, as a “by-product” our decision procedure will In what follows, we sketch the proof idea of the theorem. yield an actual attack (if any). See [14] for the detailed proof. The hardness result is easy to show. It is by reduction from the finite automata intersection problem, which has The pumping argument. First, we define a solvability been shown to be PSPACE-complete by Kozen [11]. preserving (quasi-)ordering on messages, which allows to The decidability result is much more involved, because replace single messages in the intruder knowledge by new we have two sources of infinite behavior in the model. ones such that if in the original problem a successful attack First, the intruder can perform an unbounded number of exists, then also in the modified problem. This reduces steps to derive a new message, and second, to perform the pumping argument to the following problem: Truncate one receive-send action, a principal can carry out an un- paths in message transducers in such a way that the out- bounded number of internal actions. Note that because put of the original path is equivalent (w.r.t. the solvability transducers may have ) -transitions, i.e., not in every tran- preserving ordering) to the output of the truncated path. sition a letter is read from the input, the number of transi- It remains to find criteria for truncating paths in this way. tions taken in one receive-send action is not even bounded To this end, we introduce another quasi-ordering, the so- in the size of the input message or the problem instance. called path truncation ordering, which indicates at which While the former source of infinity was already present positions a path can be truncated. To really obtain a bound in the (decidable) models for closed-ended protocols [21, on the length of paths, it then remains to show that the 2, 13], the latter is new. To prove Theorem 9, one there- equivalence relation corresponding to the path truncation fore not only needs to show that the number of actions per- ordering has finite index – more accurately, an index that formed by the intruder can be bounded, but also the num- can be bounded in the size of the problem instance. With ber of internal actions of principals. In fact, it suffices to this, and the fact that message transducers have only fi- establish the latter, since if we can bound the number of in- nite memory, the length of paths can be restricted. Finally, ternal actions, a principal only reads messages of bounded to show the bound on the index, one needs to establish a length and therefore the intruder only needs to produce bound on the depth of messages (i.e., the depth of nested messages of size bounded by this length. To bound the encryptions and hashes) in successful attacks. Again, we number of internal actions, we apply a pumping argument make use of the fact that message transducers have only showing that long paths in a message transducer can be finite memory. truncated. This argument uses that principals (the extended In what follows, the argument is described in more de- message transducers describing them) have finite memory. tail. Due to lack of space, the formal definitions of the More formally, we will show that the following problem orderings as well as the proofs of their properties are omit- is decidable. This immediately implies Theorem 9. ted. They can be found in the technical report. ¡ 7* PATHPROBLEM. Given a finite set and ; @ A ¡¡ L ¦ >=:=>=: ¦ ¦ 3# message transducers with Preserving the solvability of instances of the path prob- ¡ . . ¡ ¤ 4 ¥¤ ¥5¤ / 0V¨© \ /£¢ 0¨ for , decide whether there lem. For every , we define a quasi-ordering on * ¡ ¦ ¤\ exist messages , , such that messages (the so-called solvability preserving order- @ A ¡ ,3. ¡ ¢\>=>=:=> ¢ ; @A ing) which depends on the transducers and F >=>=:= ¤ 0 1. for every , has the following property, which we call (*): For every @A ¡ ¡ ¨¦ :=>=>=: ¢ solvable instance of the path problem, / ! ¤ /£¢ ¦ A 2. for every , and ¡ 7* §¦ , . ¡ every , and with , the instance ; @ A @ A ¡©¨ . ,3. >=>=:=>\ 0¨ ¢\£:¥¨§©£ F 3. . 0 0V ¦ >=:=>= ¢ ¤ is solvable as well. / ! ¤ /£¢ ¦ Assume that a path is replaced We write an instance of the PATHPROBLEM as ; @A ¡ by a shorter path such that the corresponding input and ¨¦ >=>=:=>¨¦ and a solution of such an instance as a 9 @A ; output labels of the shorter path, say and , satisfy ; @A ¡Q, . A ; @ A ! ¤ >=:=>=:¤ ¤ tuple of messages. The size 9 MF >=:=>=:¤ 0 ¦ and . Then, of instances is defined as the size of the representation for 3 ¦ 9 ; @ A ¡ after has returned on input , the resulting in- ¡ , . ; @ A ¦ :=>=>=> ¦ and . :=>=>=:¤ 0 truder knowledge is instead of ¡ ,N. ; A Using a pumping argument, we show that in order to @ :=>=:= ¤ \ 0 ¡ . Using (*), we conclude that find the messages , , for every , it suffices to there still exists a solution for the rest of the instance, i.e., A @A ¡ ,3. ; @A / /£¢ ¦ consider paths from to in bounded in length by >=>=:=>\ 0V ¦ :=>=:=> ¢ for . the size of the problem instance – the argument will also Consequently, it remains to find3 criteria for truncating show that the bounds can be computed effectively. Thus, long paths such that a decision procedure can enumerate all paths of length re- stricted by the (computed) bound and check whether their 4a reflexive and transitive ordering ON THE DECIDABILITY OF CRYPTOGRAPHIC PROTOCOLS WITH OPEN-ENDED DATA STRUCTURES 11 ¡M,3. ; @ A F >=:=>=:¤ 0 ¦ 1. and of . As a corollary, one obtains that the depth of output A messages can be bounded in the depth of input messages, ¦ 2. . 3 and using (***), that both the depth of input and output Truncating paths such that Condition 1. is satisfied is rather messages can be bounded in the size of the problem in- easy. The involved part is Condition 2. To this end, we stance. £¤ introduce the path truncation ordering. With this, one can show that the index of is bounded. & Moreover, the © in 2. (the level of the half-messages , ' ¦( ¦) , , ) is bounded in the size of the problem instance. Truncating paths. We extend ¦ to a quasi-ordering £¨ Therefore, can serve as the desired criterion for trun- ¦¡ (the path truncation ordering) on so-called left half- cating paths. messages. Left half-messages are prefixes of messages (considered as words over ¤ ). In particular, left half- messages may lack some closing parentheses. The “ © ” in 7 Conclusion ¦¡ is the number of missing parentheses (the level of left ¦¢ half-messages); only relates left half-messages of level We have introduced a generic protocol model for analyzing © . Analogously, right half-messages are suffixes of mes- the security of open-ended protocols, i.e., protocols with sages. Thus, they may have too many closing parentheses; open-ended data structures, and investigated the decidabil- the number of additional parentheses determines the level ity of different instances of this model. In one instance, £¤ of right half-messages. The equivalence relation on left receive-send actions are modeled by sets of rewrite rules. ¦¥ half-messages corresponding to has the following prop- We have shown that in this instance, security is undecid- E erty, which we call (**): For all left half-messages of able. This result indicated that to obtain decidability, prin- © ¦ © §£¨ level and right half-messages of level , implies cipals should only have finite memory and should not be ©¦ £ ¦ ¦ ¦ . (Note that and are messages.) able to compare messages of arbitrary size. This motivated # Now, consider two positions in the path our transducer-based model, which complies to these re- ¢ / ¤9¤\ ¨/ 6 ¦ such that , are the output la- strictions, but still captures certain open-ended protocols. ¦ bels up to these positions, and ¦ , are the output labels We have shown that in this model security is decidable and # ¦ # ¦ beginning at these positions, i.e., . PSPACE-hard; it remains to establish a tight complexity ¦ ¦ Clearly, , are left half-messages and , are right bound. These results have been shown for the shared key © half-messages. Assume that , have the same level setting and secrecy properties. We conjecture that they ¦ ¦ © £ (in particular, , have level ) and . Then, carry over rather easily to public key encryption and au- + # ¦ £ ¦ # by (**) it follows , where thentication. is the output label of the path obtained by cutting out the As pointed out in Section 5.1, a promising future di- 5 £ subpath in between and . Thus, provides us with rection is to combine the transducer-based model with the the desired criterion for “safely” (in the sense of Condition models for closed-ended protocols and to devise tree trans- 2.) truncating paths. In order to conclude that the length of ducers suitable for describing receive-send actions. We paths can be bounded in the size of the problem instance, it will also try to incorporate complex keys, since they are © £¤ remains to show that and the index of (i.e., the number used in many protocols. We believe that the proof tech- £¤ of equivalence classes modulo on left half-messages of niques devised in this paper will help to show decidability level © ) can be bounded in the size of the problem instance. also in the more powerful models. Finally, encouraged by To this end, the following is shown. the work that has been done for closed-ended protocols, the long-term goal of the work started here is to develop Bounding the depth of messages. We first show tools for automatic verification of open-ended protocols, @ A ; ; @ A \ :=>=:=>¤ ¤ (***): If is a solution of if possible by integrating the new algorithms into existing ; @A ¡ ¦ :=>=>=: ¦ , then, for every , their also exists a tools. solution if the depth of is bounded in the size of the problem instance. Acknowledgement I would like to thank Thomas Wilke We then show how the depth of the output message and the anonymous referees for useful comments and ¦ / /£¢ can be bounded: Let be a path in from to or a suggestions for improving the presentation of this paper. strict path in ¦ , and be a position in such that is Thanks also to Catherine Meadows for pointing me to the the input label of up to position and is the output paper by Pereira and Quisquater. label of up to . Then, the level of can be bounded by a polynomial in the level of and the number of states 5 Bibliography One little technical problem is that does not need to be a mes- sage since it may contain a word of the form enc , which is not a [1] R.M. Amadio and W. Charatonik. On name gen- message. However, if one considers three positions ¨ "!¢ "# , then one %$ can show that either or is a message. eration and set-based analysis in Dolev-Yao model. 12 RALF KÜSTERS Technical Report RR-4379, INRIA, 2002. [15] G. Lowe. Towards a Completeness Result for Model Checking of Security Protocols. Journal of Computer [2] R.M. Amadio, D. Lugiez, and V. Vanackère. On the Security, 7(2–3):89–146, 1999. symbolic reduction of processes with cryptographic functions. Technical Report RR-4147, INRIA, 2001. [16] C. Meadows. Extending formal cryptographic proto- col analysis techniques for group protocols and low- [3] G. Ateniese, M. Steiner, and G. Tsudik. Authen- level cryptographic primitives. In P. Degano, editor, ticated group key agreement and friends. In Pro- Proceedings of the First Workshop on Issues in the ceedings of the 5th ACM Conference on Computer Theory of Security (WITS’00), pages 87–92, 2000. and Communication Secruity (CCS’98), pages 17– 26, San Francisco, CA, 1998. ACM Press. [17] C. Meadows. Open issues in formal methods for cryptographic protocol analysis. In Proceedings of [4] J. Bryans and S.A. Schneider. CSP, PVS, and a Re- DISCEX 2000, pages 237–250. IEEE Computer So- cursive Authentication Protocol. In DIMACS Work- ciety Press, 2000. shop on Formal Verification of Security Protocols, 1997. [18] J. Mitchell, M. Mitchell, and U. Stern. Automated [5] J.A. Bull and D.J. Otway. The authentica- Analysis of Cryptographic Protocols using Murphi. tion protocol. Technical Report DRA/CIS3/PROJ/ In Proceedings of the 1997 IEEE Symposium on Se- CORBA/SC/1/CSM/436-04/03, Defence Research curity and Privacy, pages 141–151. IEEE Computer Agency, Malvern, UK, 1997. Society Press, 1997. [6] D. Dolev, S. Even, and R.M. Karp. On the Security [19] L.C. Pauslon. Mechanized Proofs for a Recursive of Ping-Pong Protocols. Information and Control, Authentication Protocol. In 10th IEEE Computer Se- 55:57–68, 1982. curity Foundations Workshop (CSFW-10), pages 84– 95, 1997. [7] D. Dolev and A.C. Yao. On the Security of Public- Key Protocols. IEEE Transactions on Information [20] O. Pereira and J.-J. Quisquater. A Security Analysis Theory, 29(2):198–208, 1983. of the Cliques Protocols Suites. In Proceedings of the 14th IEEE Computer Security Foundations Workshop [8] N.A. Durgin, P.D. Lincoln, J.C. Mitchell, and A. Sce- (CSFW-14), pages 73–81, 2001. drov. Undecidability of bounded security protocols. In Workshop on Formal Methods and Security Proto- [21] M. Rusinowitch and M. Turuani. Protocol Insecurity cols (FMSP’99), 1999. with Finite Number of Sessions is NP-complete. In 14th IEEE Computer Security Foundations Workshop [9] S. Even and O. Goldreich. On the Security of Multi- (CSFW-14), pages 174–190, 2001. Party Ping-Pong Protocols. In IEEE Symposium on Foundations of Computer Science (FOCS’83), pages [22] M. Steiner, G. Tsudik, and M. Waidner. CLIQUES: 34–39, 1983. A new approach to key agreement. In IEEE Inter- national Conference on Distributed Computing Sys- [10] N. Ferguson and B. Schneier. A Cryptographic Eval- tems, pages 380–387. IEEE Computer Society Press, uation of IPsec. Technical report, 2000. Available 1998. from http://www.counterpane.com/ipsec.pdf. [23] S. D. Stoller. A bound on attacks on authentication [11] M.R. Garey and D.S. Johnson. Computers and protocols. In Proceedings of the 2nd IFIP Interna- Intractability: A Guide to the Theory of NP- tional Conference on Theoretical Computer Science. Completeness. Freeman, San Francisco, 1979. Kluwer, 2002. To appear. [12] D. Harkins and D. Carrel. The Internet Key Exchange [24] J. Zhou. Fixing a security flaw in IKE protocols. (IKE), November 1998. RFC 2409. Electronic Letter, 35(13):1072–1073, 1999. [13] A. Huima. Efficient infinite-state analysis of secu- rity protocols. In Workshop on Formal Methods and Security Protocols (FMSP’99), 1999. [14] R. Küsters. On the Decidability of Crypto- graphic Protocols with Open-ended Data Struc- tures. Technical Report 0204, Institut für Informatik und Praktische Mathematik, CAU Kiel, Germany, 2002. Available from http://www.informatik.uni- kiel.de/reports/2002/0204.html. Game Strategies In Network Security Kong-Wei Lye Jeannette M. Wing Department of Electrical and Computer Engineering Computer Science Department Carnegie Mellon University 5000 Forbes Avenue, Pittsburgh, PA 15213-3890, USA [email protected] [email protected] Abstract important data from a workstation on the network. We compute Nash equilibria (best responses) for the attacker This paper presents a game-theoretic method for analyzing and administrator using a non-linear program and explain the security of computer networks. We view the interac- one of the solutions found for our example in Section 5. tions between an attacker and the administrator as a two- We discuss the implications of our approach in Section 6 player stochastic game and construct a model for the game. and compare our work with previous work in the literature Using a non-linear program, we compute Nash equilibria in Section 7. Finally, we summarize our results and point or best-response strategies for the players (attacker and ad- to future directions in Section 8. ministrator). We then explain why the strategies are realis- tic and how administrators can use these results to enhance the security of their network. Firewall Keywords: Stochastic Games, Non-linear Programming, Attacker Border router Network Security. Internet 1 Introduction Public Private Private Government agencies, schools, retailers, banks, and a web server file server workstation growing number of goods and service providers today all use the Internet as their integral way of conducting Figure 1: A Network Example daily business. Individuals, good or bad, can also easily connect to the internet. Due to the ubiquity of the Internet, computer security has now become more important than ever to organizations such as governments, banks, and 2 Networks as Stochastic Games businesses. Security specialists have long been interested in knowing what an intruder can do to a computer network, In this section, we first introduce the formal model of a and what can be done to prevent or counteract attacks. In stochastic game. We then use this model for our net- this paper, we describe how game theory can be used to work attack example and explain how the state set, actions find strategies for both an attacker and the administrator. sets, cost/reward functions, and transition probabilities can We illustrate our approach in an example (Figure 1) of a A be defined or derived. FA ormally, a two-player stochas- ¤ local network connected to the Internet and consider the ¤ ¢ ¢ ¥ ¡ ¢ # tic game is a tuple where .¤£¦A £¦¥ . A §©¨ interactions between them as a general-sum stochastic 0 ¢ # >&:&>& 0 >&>&:& is the state set and , ¡ ¡ game. In Section 2, we introduce the formal model for [G # ! ¢ ! # , , is the action set of player . The ac- ¡ stochastic games and relate the elements of this model to ¥ ¢ ¢ tion set for player at state is a subsetA of , i.e., + A ¤ those in our network example. In Section 3, we explain the ¢ ¢ # ¢ ¥ ¢ ¢ L >[ A and . + ¤ concept of a Nash equilibrium for stochastic games and ¢ ¢ is the state transition function. , ¡ explain what it means to the attacker and administrator. ¡ 1 [G L ¤ [ # is the reward function of player . is Then, in Section 4, we describe three possible attack a discount factor for discounting future rewards, i.e., at the scenarios for our network example. In these scenarios, an current state, a state transition has a reward worth its full attacker on the Internet attempts to deface the homepage on the public web server on the network, launch an 1We use the term “reward” in general here; in later sections, positive internal denial-of-service (DOS) attack, and capture some values are rewards and negative values are costs. 13 14 KONG-WEI LYE AND JEANNETTE M. WING value, but the reward for the transition from the next state been compromised and means at least one user account . 3 ¨ W0 is worth times its value at the current state. has been compromised. We use the variable to The game is played as follows: at a discrete time instant represent the state of the data on the node. and mean A A ¡ 6 B , the game is in state . Player 1 chooses an action the data has and has not been corrupted or stolen respec- ¤ ¤ A A A ¦¤ # Z¡ J ¨ ¨ ¢ ¢ from and player 2 chooses an action from . tively. For example, if , then the ¤ A ¢ # ¡ JW \ Player 1 then receives a reward and web server is running ftpd and httpd, a sniffer program has ¤ ¤ ¤ W ¢ # ¡ X\ player 2 receives a reward . The game been implanted, and a user account has been compromised A A A £ then moves to a new state with conditional probability but no data has yet been corrupted or stolen. A A 3 ¤ ¤ \ W ¥ \ ! \ Prob equal to . The traffic information for the whole network is cap- . § 3 3 A # © 0¨ In our example, we let the attacker be player 1 and the tured in the traffic state B where and . ¤ © L :[<0 administrator be player 2. We provide two views of the are nodes and indicates the load car- game: the attacker’s view (Figure 3) and the administra- ried on this link. A value of 1 indicates maximum capac- tor’s view (Figure 4). We describe these figures in detail ity. A For example, in a 10Base-T connection, the values ¤ later in Section 4. L , , , and 1 represent 0Mbps, 3.3Mbps, 6.7Mbps, and A A A 10Mbps respectively. In our example, the trafA fic state is ¥ ¥ B # -© ¨© ¨© ¨© ¨ B # ¨ ¤ ¤ ¤ . We let ¢ 2.1 Network state ¢ for normal traffic conditions. In general, the state of the network can contain various The potential state space for our network example is kinds of features such as type of hardware, software, con- very large but we shall discuss how to handle this prob- nectivity, user privileges, etc. Using more features in the lem in Section 6. The full state space in our example has a ¥ ! ! 4! ! ! ! "$# ! ! B !U# G state allows us to represent the network better, but often size of ¤ makes the analysis more complex and difficult. We view 4 billion states but¢ there are only 18 states (15 in Figure 3 the network example as a graph (Figure 2). A node in the and 3 others in Figure 4) relevant to our illustration here. graph is a physical entity such as a workstation or router. In these figures, each state is represented using a box with We model the external world as a single computer (node a symbolic state name and the values of the state variables. E) and represent the web server, file server, and worksta- For convenience, we shall mostly refer to the states using tion by nodes W, F, and N, respectively. An edge in the their symbolic state names. graph represents a direct communication path (physical or virtual). For example, the external computer (node E) has direct access to only the public web server (node W). 2.2 Actions An action pair (one from the attacker and one from the l administrator) causes the system to move from one state E EW W to another in a probabilistic manner. A single action for the attacker can be any part of his attack strategy, such lWF lNW as flooding a server with SYN packets or downloading the password file. When a player does nothing, we denote this inaction as % . The action set for the attacker )( +*-, F N '& ¢ consists of all the actions he can take in all l . +( +*-, FN $& # the states, ¢ Attack_httpd, Attack_ftpd, Continue_hacking, Deface_website_leave, Install_sniffer, Figure 2: Network State Run_DOS_virus, Crack_file_server_root_password, Crack_workstation_root_password, Capture_data, Shut- %Z0 % Instantiating our game model, we let a superstate down_network, , where denotes inaction. His ¥ +( +*-, ¥¤¨ ¤B ¨ ¦¤ ¢.& be the state of the network. , actions in each state is a subset of . For ¥ ¢ , and are the node states for the web server, file example, in the state Normal_operation (see Fig- ¢ server, and workstation respectively, and B is the traf- ure 3, topmost state), the attacker has an action set . § )( +*-, /10£243'5£6 0¡7982:5+;:<=0©> '& # fic state for the whole network. Each node (where ¢ _ Attack_httpd, Attack_ftpd, .©¨ § 0 # - \ ¨ %Z0 ) has a node state to . Actions for the administrator are mainly pre- represent information about hardware and software config- ventive or restorative measures. In our? example, the .© 9 . A B,X DC4, ¨E¡Z U 0 ¢.&?@ # urations. is a list of software appli- administrator has an action set ¡ cations running on the node and , , , and denote ftpd, Remove_compromised_account_restart_httpd, httpd, nfsd, and some user process respectively. For mali- Restore_website_remove_compromised_account, 9 cious codes, and represent sniffer programs and viruses Remove_virus_compromised_account, .¡ ¨0 respectively. is a variable used to represent the Install_sniffer_detector, Remove_sniffer_detector, state of the user accounts. means no user account has Remove_compromised_account_restart_ftpd, GAME STRATEGIES IN NETWORK SECURITY 15 Z0 Remove_compromised_account_sniffer, % . In state its reputation and its customers to lose confidence. Mead- Ftpd_attacked (see? Figure 4), the administrator has an ows’s work on cost-based analysis of DOS discusses how . A B,W DC4, ¡ ;-7£¢ 5); ;-5¥¤§¦¡8¨¢ &?@ # action set ¢ _ install_sniffer_detector, costs can be assigned to an attacker’s actions using cate- Z0 % . gories such as cheap, medium, expensive, and very expen- A node with a compromised account may or may not sive [Mea01]. be observable by the administrator. When it is not observ- In our model, we restrict ourselves to the amount of re- able, we model the situation as the administrator having an covery effort (time) required by the administrator. The re- empty action set in the state. We assume that the admin- ward for an attacker’s action is mostly defined in terms istrator does not know whether there is an attacker or not. of the amount of effort the administrator has to make to Also, the attacker may have several objectives and strate- bring the network from one state to another. For exam- gies that the administrator does not know. Furthermore, ple, when a particular service crashes, it may take the not all of the attacker’s actions can be observed. administrator 10 or 15 minutes of time to determine the cause and restart the service 2. In Figure 4, it costs the administrator 10 minutes to remove a compromised user 2.3 State transition probabilities account and to restart the httpd (from state Httpd_hacked In our example, we assign state transition probabilities to state Normal_operation). For the attacker, this amount based on intuition. In real life, case studies, statistics, sim- of time would be his reward. To reflect the severity of the ulations, and knowledge engineering can provide the re- loss of the important financial data in our network exam- quired probabilities. In Figures 3 and 4, state transitions ple, we assign a very high reward for the attacker’s action are represented by arrows. Each arrow is labeled with an that leads to the state where he gains this data. For ex- action, a transition probability, and a cost/reward. In the ample, from state Workstation_hacked to state Worksta- formal game model, a state transition probability is a func- tion_data_stolen_1 in Figure 3, the reward is 999. There tion of both players’ actions. Such probabilities are used are also some transitions in which the cost to the admin- in the non-linear program (Section 3) for computing a so- istrator is not the same magnitude as the reward to the at- lution to the game. However, in order to separate the game tacker. It is such transitions that make the game a general- into two views, we show the transitions as simply due to sum game instead of a zero-sum game. a single player’s actions. For example, with the second dashed arrow from the top in Figure 3, we show the de- © ¥ © ¥ 3 Nash Equilibrium ! ¥ rived probability Prob( _ _ , Continue_attacking ) = 0.5 as due to only the attacker’s ? ? We no? w return to the formal model for stochastic games. . action Continue_attacking. When the network is in state A § # ¡ !-, ¡ # [G¡ LK0 ¨ Let be § the Normal_operation and neither the attacker nor adminis- + § set of probability vectors of length . trator takes any action, it will tend to stay in the same ¡ ¦ is a stationary strategy for player . is the vec- A ¨ state. We model this situation as having a near-identity § ¥& U¨ &:&>& G /. U tor where is the ! "# $ % ',! stochastic matrix, i.e., we let Prob _ ¡ probability that player should use to take action in state % ( (= L =*) Normal_operation, % , )=1- for some small . ¥S@A . A stationary strategy is a strategy which is indepen- #! % % + # 2 Then Prob Normal_operation, , )= for all ¥& dent of time and history. A mixed or randomized stationary ! "# $ % ' _ where is the number of states. G L10 0 ¢ There are also state transitions that are infeasible. For ex- strategy is one where and U¨ #7[ ample, it may not be possible for the network to move from and a pure strategy is one where for some ¢ a normal operation state to a completely shutdown state . without going through some intermediate states. Infeasi- The objective of each player is to maximize some B expected return. Let be the state at time and ble state transitions are assigned transition probabilities of ¡ ¢ B 0. be the reward received by player at time . We define an expected return to be the column vec- A ¥ 9 9 £ 9 £ % % % # &>&>& 5. 2 243 2 243 2 243 ¥ " " tor " where ? 9 ¨ . ? ? ; % % 243 2.4 Costs and rewards 2 ! # <0 # , ¢ 3 2 2 " " . The ex- ¨ . ¡ 3 % 2 2 3 & 0 pectation operator " is used to mean that player There are costs (negative values) and rewards (positive val- ¡ plays , i.e., player chooses an action using the prob- ? ues) associated with the actions of the administrator and ? A ¡ ¡ ability distribution at and receives an im- ? ? attacker. The attacker’s actions have mostly rewards and ? ? 3 3 ¤ A ¡ 6. ¡ # ¡ ¢ mediate reward for ¡ such rewards are in terms of the amount of damage he does 3 3 3 ¤ 3 87 % 7 % 3 3 3 L # U\ \ # [U¡ " to the network. Some costs, however, are difficult to quan- . , is & & tify. For example, the loss of marketing strategy informa- 2These numbers were given by the department’s network manager. tion to a competitor can cause large monetary losses. A 3 : <;>=/?5@ We use 9 to refer to an matrix with elements <;>=C?% defaced corporate website may cause the company to lose : . 16 KONG-WEI LYE AND JEANNETTE M. WING player k’s reward matrix in state . sition, and the gain or cost in minutes of restorative effort #¡ For an infinite-horizon game, we let and use a incurred on the administrator. The three scenarios are in- 9 [ discount factor to discount future rewards. dicated using bold, dotted, and dashed arrows in Figure 3. is then the expected total discounted rewards that player ¡ Due to space constraints, not all state transitions for every will receive when starting at state . For a finite-horizon action are shown. From one state to the next, state variable 9 ¢ /# [ game, L and . is also called the value changes are highlighted using boldface. ¡ vector of player . A Scenario 1: A common target for use as a launching ¤ ¥ ¥ ¦ A A A A A A Nash equilibrium in stationary strategies § is base in an attack is the public web server. The web server % 9 9 ¤ ¤ ¥ ¥ ¥ £ £ 0 § typically runs an httpd and an ftpd and a common tech- A A one which satisfies § 3 9 9 ¤ ¤ ¤ ¤ ¤ ¥ ¥ ¥ nique for the attacker to gain a root shell is buffer over- A ¦ 10 § £ and component- 9 ¤ flow. Once the attacker gets a root shell, he can deface ¦ wise. Here, is the value vector of the game for ¡ the website and leave. This scenario is shown by the state playerA when both players play their stationary strategies ¤ transitions indicated by bold arrows in Figure 3. and respectively and is used to mean the left- hand-side vector is component-wise, greater than or equal From state Normal_operation, the attacker takes action to the right-hand-side vector. At this equilibrium, there is Attack_httpd. With a probability of 1.0 and a reward of 10, no mutual incentive for either one ofA the players to deviate he moves the system to state Httpd_attacked. This state ¤ ¥ ¥ from their equilibrium strategies and . A deviation indicates increased traffic between the external computer A A will mean that one orA both of them will have lower ex- and the web server as a result of his attack action. Taking 9 9 ¤ ¤ ¤ £ ¦ pected returns, i.e., and/or . A pair action Continue_attacking, he has a 0.5 probability of suc- of Nash equilibrium strategies A is also known as best re- cess of gaining a user or root access through bringing down ¥ sponses, i.e., if player 1 plays , player 2’s best response the httpd, and the system moves to state Httpd_hacked. ¤ ¥ Once he has root access in the web server, he can deface is and vice versa. A ¤ the website, restart the httpd and leaves, moving the net- In our network example, and corresponds to A theA attacker’s and administrator’s strategies respectively. work to state Website_defaced. 9 ¤ A ¦ corresponds to the expected return for the at- Scenario 2: The other thing that the attacker can do after 9 ¤ ¤ A £ tacker and corresponds to the expected return he has hacked into the web server is to launch a DOS attack for the administrator when they use the strategies and from inside the network. This is shown by the state transi- ¤ A . In a Nash equilibrium, when the attacker and adminis- tions drawn using dotted arrows (starts from the middle of ¤ ¥ ¥ trator use their best-response strategies and respec- Figure 3), with each state having more internal traffic than tively, neither will gain a higher expected return if the other the previous. continues using his Nash strategy. From state Webserver_sniffer, the attacker takes ac- Every general-sum discounted stochastic game has at tion Run_DOS_virus. With probability 1 and a reward of least one Nash equilibrium in stationary mixed strategies 30, the network moves into state Webserver_DOS_1. In (see [FV96]) (not necessarily unique) and finding these this state,A the traffic load on all internal links has increased ¤ equilibria is non-trivial. In our network example, find- from to . From this state, the network degrades to state ing multiple Nash equilibria means finding multiple pairs Webserver_DOS_2 with probability 0.8 even when the at- of Nash strategies. In each pair, a strategy for one player tacker does nothing. The traffic load is now at full capacity is a best-response to the strategy for the other player and of 1 in all the links. We assume that there is a 0.2 proba- vice versa. A non-linear program found in [FV96] can be bility that the administrator notices this and takes action used to find the equilibrium strategies for both players in to recover the system. In the very last state, the network a general-sum stochastic game. We shall refer to this non- grinds to a halt and nothing productive can take place. linear program as NLP-1 and use it to find Nash equilibria Scenario 3: Once the attacker has hacked into the web for our network example later in Section 5. server, he can install a sniffer and a backdoor program. The sniffer will sniff out passwords from the users in the work- station when they access the file server or web server. Us- 4 Attack and Response Scenarios ing the backdoor program, the attacker then comes back to collect his password list from the sniffer program, cracks In this section, we describe three different attack and re- the root password, logs on to the workstation, and searches sponse scenarios. We show in Figure 3, the viewpoint of the local hard disk. This scenario is shown by the state the attacker, how he sees the state of the network change transitions indicated by dashed arrows in Figure 3. as a result of his actions. The viewpoint of the admin- From state Normal_operation, the attacker takes action istrator is shown in Figure 4. In both figures, a state is Attack_ftpd. With a probability of 1.0 and a reward of 10, represented using a box containing the symbolic name and he uses the buffer overflow or a similar attack technique the values of the state variables for that state. Each transi- and moves the system to state Ftpd_attacked. There is in- tion is labeled with an action, the probability of the tran- creased traffic between the external computer and the web GAME STRATEGIES IN NETWORK SECURITY 17 Normal_operation <<(f,h),u,i>,<(f,n),u,i>,<(p),u,i>, <1/3,1/3,1/3,1/3>> Attack_ftpd, 1.0, 10 Attack_httpd, 1.0, 10 Continue_ Continue_ Httpd_attacked Ftpd_attacked attacking, attacking, <