arXiv:0806.2139v1 [cs.GT] 12 Jun 2008 h oprt o hl omc etr oevr na in Moreover, better! much do while a players play “irrational” for “should” when cooperate players game rational who this that in case equilibrium a Nash make the dilemma). prisoner’s to repeated hard of is discussion It Sec- more (see defect for always 3 to tion is such dilemma games prisoner’s in repeated numerous equilibrium as from Nash the suffers example, However, equilibrium For Nash problems. schools!). known, business reason well in a is is taught as (there is do theory people game what that into insight gives librium the- Nash’s of result key the fact, in sis). equilibri was, Nash (this a 1950] be [Nash to random guaranteed (i.e., is mixed there strategies, that possible allow ized) many we is finitely and attractive only strategies, has so deterministic player equilibrium each where Nash games beliefs makes other in those to what what response of best about point a Part fixed beliefs plays a and correct doing, is are holds It by players player play. better it of each do that state where is can steady equilibrium possible player Nash a no behind represent that intuition The such game) deviating. the in player equilibriu Nash a a Intuitively, is theory. game in equilibrium equilibrium Nash n I-546,adb FS ne rn FA9550-05-1-0055 ITR-0325453 grant grants under under AFOSR by under and NSF IIS-0534064, by and Computing part Distributed in of supported Work Principles on Symposium ACM o ut e ae,tikn ntrso ahequi- Nash of terms in thinking games, few a quite For h ae ouincnet httyt drs hs short- these address discussed. to are of equilibrium try aspects Nash that all of concepts comings of aware Solution not not does game. are it players the and where account, cases into with cost deal computation coalitions “faulty with take tolerate deal not not not does does does it (it example, behavior), “unexpected” robust for or not perspective: is science equilibrium equi- Nash computer Nash a considering from when librium arise problems reasonab new descriptively However, nor normatively neither prisone repeated is commu- of dilemma theory equilibrium Nash game the the example, numerous in for from known nity; suffers well are it Some However, problems. theory. equi- game of notion in commonly-used librium most the is equilibrium Nash ∗ erne rmthe from Reprinted taeyprofile strategy stems omnyue oinof notion commonly-used most the is Introduction 1 acleto fsrtge,oefreach for one strategies, of collection (a rceig fteTet-eet Annual Twenty-Seventh the of Proceedings Abstract ouinCnet o h 1tCentury 21st the for Concepts Solution eodNs Equilibrium: Nash Beyond http://www.cs.cornell.edu/home/halpern et fCmue Science Computer of Dept. [email protected] oehY Halpern Y. Joseph 2008. , onl University Cornell taa Y14853 NY Ithaca, it , r’s le. ” um m . - . • • • oigtrepolm ihNs qiiru,alinspired all concerns. equilibrium, science fol Nash computer by the with address problems concepts successes, three solution lowing some alternative these Despite of 1991; none Tirole 1994].) and Rubinstein [Fudenberg and as Osborne such text, theory game dard strategies inated ilequilibrium tial rprequilibrium proper et fNs qiiru aebe nrdcd in- introduced, refine- been others, and have many to equilibrium among alternatives Nash alternative cluding, Various developing of on ments community concepts. economics solution the of in discussion a is for game 1990] the [Kreps problems.) if these (See of doing some are once? players only correct other played obtain what players re- about do (like beliefs equilibrium how Nash dilemma), unique prisoner’s a peated even is And there played. where be games will one in Players which equilib- knowing equilibria? of Nash Nash way a multiple no have are should there why when once, arise played rium only is that game xml,telreacin lydoe h internet. for the in, over reasonable played always auctions not large is the This example, all and game. situation in every the players in all the made including be can game, common that the moves have of possible structure players the of that knowledge presumes equilibrium Nash the deal at can are cryptography. that that of concepts concerns heart solution players, resource-bounded need We with concerns computational account. take into not we does games, equilibrium large Nash In agents. both. colluding expect with should deal nor equilibrium behavior, 1985; it “unexpected” Paterson or Nash does and and “faulty” Lynch, with [Fischer, deal tolerance Byzan- not does 1980]. on Lamport fault and work agreement Shostak, as computing Pease, example such distributed for tine to, in problems leading focus chosen on asynchrony, strategies the been to player, has responses other the best on by are choosing been players that has agents—rational literature strategies of theory interacting, concerns game agents the strategic multiple in com- focus with distributed the concerned and are science puting computer both Although o upiigy hr a enagetda fwork of deal great a been has there surprisingly, Not , Teentosaedsusdi stan- in discussed are notions (These . tebighn)pretequilibrium perfect hand) (trembling and , ∗ trtddlto fwal dom- weakly of deletion iterated rationalizability , sequen- - , In the following sections, I discuss each of these issues in be viewed as a way of adding fault tolerance to equilib- more detail, and sketch solution concepts that can deal with rium notions. To capture this intuition, Abraham et al. them, with pointers to the relevant literature. [Abraham, Dolev, Gonen, and Halpern 2006] define a strat- egy profile to be t-immune if no player who does not deviate 2 Robust and Resilient Equilibrium is worse off if up to t players do deviate. Note the differ- Nash equilibrium tolerates deviations by one player. It is ence between resilience and immunity. A strategy profile is perfectly consistent with Nash equilibrium that two players resilient if deviators do not gain by deviating; a profile is could do much better by deviating in a coordinated way. For immune if non-deviators do not get hurt by deviators. In the example, consider a game with n> 1 players where players example above, although everyonebargainingis a k-resilient much play either 0 or 1. If everyone plays 0, everyone get a Nash equilibrium for all k ≥ 0, it is not 1-immune. payoff of 1; if exactly two players plays 1 and the rest play Of course, we may want to combine resilience and re- 0, then the two who play 1 get a payoffof 2, and the rest get silience; a strategy is (k,t)-robust if it is both k-resilient and 0; otherwise, everyone gets 0. Clearly everyone playing 0 is t-immune. (All the informal definitions here are completely a Nash equilibrium, but any pair of players can do better by formalized in [Abraham, Dolev, Gonen, and Halpern 2006; deviating and playing 1. Abraham, Dolev, and Halpern 2008].) A Nash equilibrium Say that a Nash equilibrium is k-resilient if it tolerates is just a (1,0)-robust equilibrium. Unfortunately, for (k,t) 6= deviations by coalitions of up to k players. The notion of (1, 0), a (k,t)-robust equilibrium does not exist in gen- resilience is an old one in the game theory literature, going eral. Nevertheless, there are a number of games of inter- back to Aumann [1959]. Various extensions of Nash equi- est where they do exist; in particular, they can exist if play- librium have been proposed in the game theory literature to ers can take advantage of a mediator, or trusted third party. deal with coalitions [Bernheim, Peleg, and Whinston 1989; To take just one example, consider Byzantine agreement Moreno and Wooders 1996]. However, these notions do not [Pease, Shostak, and Lamport 1980]. Recall that in Byzan- deal with players who act in unexpected ways. tine agreement there are n soldiers, up to t of which may be There can be many reasons that players act in unexpected faulty (the t stands for traitor), one of which is the general. ways. One, of course, is that they are indeed irrational. The general has an initial preference to attack or retreat. We However, often seemingly irrational behavior can be ex- want a protocol that guarantees that (1) all nonfaulty soldiers plained by players having unexpected utilities. For example, reach the same decision, and (2) if the general is nonfaulty, in a peer-to-peer network like Kazaa or Gnutella, it would then the decision is the general’s preference. It is trivial seem that no rational agent should share files. Whether or to solve Byzantine agreement with a mediator: the general not you can get a file depends only on whether other people simply sends the mediator his preference, and the mediator share files. Moreover,there are disincentives for sharing (the sends it to all the soldiers. possibility of lawsuits, use of bandwidth, etc.). Neverthe- The obvious question of interest is whether we can less, people do share files. However, studies of the Gnutella implement the mediator. That is, can the players in network have shown that almost 70 percent of users share no the system, just talking among themselves (using what files and nearly 50 percent of responses are from the top 1 economists call “cheap talk”) simulate the effects of the percent of sharing hosts [Adar and Huberman 2000]. Is the mediator. This is a question that has been of interest to behavior of the sharing hosts irrational? It is if we assume both the computer science community and the game the- appropriateutilities. But perhaps sharing hosts get a big kick ory community. In game theory, the focus has been on out of being the ones that provide everyoneelse with the mu- whether a Nash equilibrium in a game with a mediator sic they play. Is that so irrational? In other cases, seemingly can be implemented using cheap talk (cf. [Barany 1992; irrational behavior can be explained by faulty computers or Ben-Porath 2003; Forges 1990; Gerardi 2004; Heller 2005; a faulty network (this, of course, is the concern that work Urbano and Vila 2002; Urbano and Vila 2004]). In cryptog- on Byzantine agreement is trying to address), or a lack of raphy, the focus has been on secure multiparty computation understanding of the game. [Goldreich et al. 1987; Shamir et al. 1981; Yao 1982]. Here To give just one example of a stylized game where this it is assumed that each agent i has some private information issue might be relevant, consider a group of n bargaining xi (such private information, like the general’s preference, agents. If they all stay and bargain, then all get 2. However, is typically called the player’s type in game theory). Fix a if any agent leaves the bargaining table, those who leave get function f. The goal is have agent i learn f(x1,...,xn) 1, while those who stay get 0. Clearly everyone staying at without learning anything about xj for j 6= i beyond what the bargaining table is a k-resilient Nash equilibrium for all is revealed by the value of f(x1,...,xn). With a trusted k ≥ 0, and it is Pareto optimal (everyone in fact gets the mediator, this is trivial: each agent i just gives the mediator highest possible payoff). But, especially if n is large, this its private value xi; the mediator then sends each agent i the equilibrium is rather “fragile”; all it takes is one person to value f(x1,...,xn). Work on multiparty computation pro- leave the bargaining table for those who stay to get 0. vides general conditions under which this can be done (see Whatever the reason, as pointed out by Abraham et [Goldreich 2004] for an overview). Somewhat surprisingly, al. [2006], it seems important to design strategies that tol- despite there being over 20 years of work on this problem erate such unanticipated behaviors, so that the payoffs of in both computer science and game theory, until recently, the users with “standard” utilities do not get affected by there has been no interaction between the communities on the nonstandard players using different strategies. This can this topic. Abraham et al. [2006, 2008] essentially characterize when • If n > k + 3t then, assuming cryptography and mediators can be implemented. To understand the results, polynomially-bounded players, mediators can be ǫ- three games need to be considered: an underlying game Γ, implemented using cheap talk, but if n ≤ 2k +2t, then an extension Γd of Γ with a mediator, and a cheap-talkexten- the running time depends on the utilities in the game and sion ΓCT of Γ. Γ is assumed to be a normal-form Bayesian ǫ. game: each player has a type from some type space with a • If n ≤ k + 3t, then even assuming cryptography, known distribution over types, and must choose an action polynomially-bounded players, and a (k + t)-punishment (where the choice can depend on his type). The utilities of strategy, mediators cannot, in general, be ǫ-implemented the players dependon the types and actions taken. For exam- using cheap talk. ple, in Byzantine agreement, the possible types of the gen- eral are 0 and 1, his possible initial preferences (the types of • If n > k + t then, assuming cryptography, polynomially- the other players are irrelevant). The players’ actions are to bounded players, and a public-key infrastructure (PKI), attack or retreat. The assumption that there is a distribution we can ǫ-implement a mediator. over the general’s preferences is standard in game theory, al- All the possibility results showing that mediators can though not so much in distributed computing. Nonetheless, be implemented use techniques from secure multiparty in many applications of Byzantine agreement, it seems rea- computation. The results showing that that if n ≤ 3k +3t, sonable to assume such a distribution. Roughly speaking, then we cannot implement a mediator without knowing a cheap talk game implements a game with a mediator if it utilities and that, even if utilities are known, a punish- induces the same distribution over actions in the underlying ment strategy is required, use the fact that Byzantine game, for each type vector of the players. With this back- agreement cannot be reached if t < n/3; the impos- ground, I can summarize the results of Abraham et al. sibility result for n ≤ 2k + 3t also uses a variant of Byzantine agreement. These results provide an excellent • If n > 3k +3t, a (k,t)-robust strategy ~σ with a medi- illustration of how the interaction between computer ator can be implemented using cheap talk (that is, there science and game theory can lead to fruitful insights. is a (k,t)-robust strategy ~σ′ in the cheap talk game such ′ Related work on implementing mediators can be found that ~σ and ~σ induce the same distribution over actions in [GordonandKatz 2006; HalpernandTeague2004; in the underlying game). Moreover, the implementation Izmalkov, Micali, and Lepinski 2005; Kol and Naor 2008; requires no knowledge of other agents’ utilities, and the Lepinski, Micali, Peikert, and Shelat 2004; cheap talk protocol has bounded running time that does Lysyanskaya and Triandopoulos 2006]. not depend on the utilities. • If n ≤ 3k +3t then, in general, mediators cannot be im- 3 Taking Computation Into Account plemented using cheap talk without knowledge of other Nash equilibrium does not take computation into account. agents’ utilities. Moreover, even if other agents’ utilities To see why this might be a problem, consider the following are known, mediators cannot, in general, be implemented example, taken from [Halpern and Pass 2008]. without having a (k + t)-punishment strategy (that is, a strategy that, if used by all but at most k + t players, guar- Example 3.1: You are given a number n-bit number x. You antees that every playergets a worse outcomethan theydo can guess whether it is prime, or play safe and say nothing. with the equilibrium strategy) nor with bounded running If you guess right, you get $10; if you guess wrong, you lose time. $10; if you play safe, you get $1. There is only one Nash equilibrium in this 1-player game: giving the right answer. • If n> 2k +3t, then mediators can be implemented using But if n is large, this is almost certainly not what people will cheap talk if there is a punishment strategy (and utilities do. Even though primality testing can be done in polynomial are known) in finite expected running time that does not time, the costs for doing so (buying a larger computer, for depend on the utilities. example, or writing an appropriate program), will probably • If n ≤ 2k +3t then mediators cannot, in general, be im- not be worth it for most people. The point here is that Nash plemented, even if there is a punishment strategy and util- equilibrium is not taking the cost of computing whether x is ities are known. prime into account. There have been attempts in the game theory community • If n> 2k +2t and there are broadcast channels then, for to define solution concepts that take computation into ac- all ǫ, mediators can be ǫ-implemented (intuitively, there count, going back to the work of Rubinstein [1986]. (See is an implementation where players get utility within ǫ of [Kalai 1990] for an overview of the work in this area in the what they could get by deviating) using cheap talk, with 1980s, and [Ben-Sasson, Kalai, and Kalai 2007] for more bounded expected running time that does not depend on recent work.) Rubinstein assumed that players choose a fi- the utilities. nite automaton to play the game rather than choosing a strat- • If n ≤ 2k +2t then mediators cannot, in general, be ǫ- egy directly; a player’s utility depends both on the move implemented, even with broadcast channels. Moreover, made by the automaton and the complexity of the automaton even assuming cryptography and polynomially-bounded (identified with the number of states of the automaton). In- players, the expected running time of an implementation tuitively, automata that use more states are seen as represent- depends on the utility functions of the players and ǫ. ing more complicated procedures. Rafael Pass and I [2008] provide a general game-theoretic framework that takes com- Now consider finitely repeated prisoner’s dilemma putation into account. (All the discussion in this section is (FRPD), where prisoner’s dilemma is played for some fixed taken from [Halpern and Pass 2008].) Like Rubinstein, we number N of rounds. The only Nash equilibrium is to al- view all players as choosing a machine, but we use Turing ways defect; this can be seen by a backwards induction ar- machines, rather than finite automata. We associate a com- gument. (The last round is like the one-shot game, so both plexity, not just with a machine, but with the machine and its players should defect; given that they are both defecting at input. This is important in Example 3.1, where the complex- the last round, they should both defect at the second-last ity of computing whether x is prime depends, in general, on round; and so on.) This seems quite unreasonable. And, the length of x. indeed, in experiments, people do not always defect. In fact, The complexity could represent the running time of or quite often they cooperate throughout the game. Are they ir- space used by the machine on that input. The complexity rational? It is hard to call this irrational behavior, given that can also be used to capture the complexity of the machine the “irrational” players do much better than supposedly ra- itself (e.g., the number of states, as in Rubinstein’s case) or tional players who always defect. There have been many at- to model the cost of searching for a new strategy to replace tempts to explain cooperation in FRPD in the literature (see, one that the player already has. (One of the reasons that for example, [Kreps, Milgrom, Roberts, and Wilson 1982]). players follow a recommended strategy is that there may be Indeed, there have even been well-known attempts that take too much effort involved in trying to find a new one; I return computation into account; it can be shown that if players to this point later.) are restricted to using a finite automaton with bounded com- We again consider Bayesian games, where each player plexity, then there exist equilibria that allow for cooper- has a type. In a standard Bayesian game, an agent’s util- ation [Neyman 1985; Papadimitriou and Yannakakis 1994]. ity depends on the type profile and the action profile (that is, However, the strategies used in those equilibria are quite every player’s type, and the action chosen by each player). complex, and require the use of large automata; as a conse- In a computational Bayesian game, each player i chooses quence this approach does not seem to provide a satisfactory a Turing machine. Player i’s type ti is taken to be the in- explanation as to why people choose to cooperate. put to player i’s Turing machine Mi. The output of Mi on Using the framework described above leads to a straight- input ti is taken to be player i’s action. There is also a com- forward explanation. Consider the tit-for-tat strategy, which plexity associated with the pair (Mi,ti). Player i’s utility proceeds as follows: a player cooperates at the first round, again depends on the type profile and the action profile, and and then at round m +1, does whatever his opponent did at also on the complexity profile. The reason we consider the round m. Thus, if the opponent cooperated at the previous whole complexity profile in determining player i’s utility, as round, then you reward him by continuing to cooperate; if he opposed to just i’s complexity, is that, for example, i might defected at the previous round, you punish him by defecting. be happy as long as his machine takes fewer steps than j’s. If both players play tit-for-tat, then they cooperate through- Given these definitions, we can define Nash equilibrium as out the game. Interestingly, tit-for-tat does exceedingly well usual. With this definition, by defining the complexity ap- in FRPD tournaments, where computer programs play each propriately, it will be the case that playing safe for suffi- other [Axelrod 1984]. ciently large inputs will be an equilibrium. Tit-for-tat is a simple program, which needs very little Computational Nash equilibrium also gives a plausible memory. Suppose that we charge even a modest amount explanation of observed behavior in finitely-repeated pris- for memory usage, and that there is a discount factor δ, with oner’s dilemma. .5 <δ< 1, so that if the player gets a rewardof rm in round Example 3.2: Recall that prisoner’s dilemma, in prisoner’s m, his total reward over the whole N-round game is taken N m dilemma, there are two prisoners, who can choose to either to be Pm=1 δ rm. In this case, it is easy to see that, no cooperate or defect. As described in the table below, if they matter what the cost of memory is, as long as it is positive, both cooperate, they both get 3; if they both defect, then both for a sufficiently long game, it will be a Nash equilibrium get 1; if one defects and the other cooperates, the defector for both players to play tit-for-tat. For the best response to gets 5 and the cooperator gets −5. (Intuitively, the cooper- tit-for-tat is to play tit-for-tat up to the last round, and then ator stays silent, while the defector “rats out” his partner. If to defect. But following this strategy requires the player to they both rat each other out, they both go to jail.) keep track of the round number, which requires the use of extra memory. The extra gain of $2 achieved by defecting at C D the last round, if sufficiently discounted, will not be worth C (3,3) (−5, 5) the cost of keeping track of the round number. D (5, −5) (-3,-3) Note that even if only one player is computationally bounded and is charged for memory, and memory is free for the other player, then there is a Nash equilibrium where the It is easy to see that defecting dominates cooperating: no bounded player plays tit-for-tat, while the other player plays matter what the other player does, a player is better off de- the best response of cooperatingup (but not including) to the fecting than cooperating. Thus, “rational” players should round of the game, and then defecting. defect. And, indeed, (D,D) is the only Nash equilibrium of this game. Although (C, C) gives both players a better Although with standard games there is always a Nash payoff than (D,D), this is not an equilibrium. equilibrium, this is not the case when we take computation into account, as the following example shows. 4 Taking (Lack of) Awareness Into Account Example 3.3: Consider roshambo (rock-paper-scissors). We Standard game theory models implicitly assume that all sig- model playing rock, paper, and scissors as playing 0, 1, and nificant aspects of the game (payoffs, moves available, etc.) 2, respectively. The payoff to player 1 of the outcome (i, j) are common knowledge among the players. However, this is 1 if i = j ⊕ 1 (where ⊕ denotes addition mod 3), −1 if is not always a reasonable assumption. For example, sleazy j = i⊕1,and0if i = j. Player 2’s playoffs are the negative companies assume that consumers are not aware that they of those of player 1; the game is a zero-sum game. As is can lodge complaints if there are problems; in a war setting, well known, the unique Nash equilibrium of this game has having technology that an enemy is unaware of (and thus be- the players randomizing uniformly between 0, 1, and 2. ing able to make moves that the enemy is unaware of) can Now consider a computational version of roshambo. Sup- be critical; in financial markets, some investors may not be pose that we take the complexity of a deterministic strategy aware of certain investment strategies (complicated hedging to be 1, and the complexity of a strategy that uses random- strategies, for example, or tax-avoidance strategies). ization to be 2, and take player i’s utility to be his payoff in To understand the impact of adding the possibility of the underlying Bayesian game minus the complexity of his unawareness to the analysis of games, consider the game strategy. Intuitively, programs involving randomization are shown in Figure 1 (this example, and all the discussion in more complicated than those that do not randomize. With this section, is taken from [Halpern and Rˆego 2006]). One this utility function, it is easy to see that there is no Nash Nash equilibrium of this game has A playing acrossA and equilibrium. For suppose that (M1,M2) is an equilibrium. B playing downB. However, suppose that A is not aware If M1 uses randomization, then 1 can do better by playing that B can play downB. In that case, if A is rational, A will the deterministic strategy j ⊕ 1, where j is the action that play downA. Therefore, Nash equilibrium does not seem to gets the highest probability according to M2 (or is the de- be the appropriate solution concept here. Although A would terministic choice of player 2 if M2 does not use random- play acrossA if A knew that B were going to play downB, A ization). Similarly, M2 cannot use randomization. But it is cannot even contemplate this possibility, let alone know it. well known (and easy to check) that there is no equilibrium for roshambo with deterministic strategies. Is the lack of Nash equilibrium a problem? Perhaps not. Taking computation into account should cause us to rethink things. In particular, we may want to consider other solution concepts. But, as the examples above show, Nash equilib- rium does seem to make reasonable predictions in a num- ber of games of interest. Perhaps of even more interest, us- ing computational Nash equilibrium lets us provide a game- theoretic account of security. The standard framework for multiparty security does not take into account whether players have an incentive to ex- ecute the protocol. That is, if there were a trusted media- Figure 1: A simple game. tor, would player i actually use the recommended protocol even if i would be happy to use the services of the media- To find an appropriate analogue of Nash equilibrium in tor to compute the function f? Nor does it take into account games where players may be unaware of some possible whether the adversary has an incentive to underminethe pro- moves, we must first find an appropriate representation for tocol. such games. The first step in doing so is to explicitly repre- Roughly speaking, the game-theoretic definition says sent what players are aware of at each node. We do this by that Π is a game-theoretically secure (cheap-talk) proto- using what we call an augmented game. col for computing f if, for all choices of the utility func- Recall that an extensive game is described by a game tion, if it is a Nash equilibrium to play with the me- tree. Each node in the tree describes a partial history of the diator to compute f, then it is also a Nash equilibrium game—the sequence of moves that led to that node. Asso- to use Π to compute f. Note that this definition does ciated with each node is the player that moves at that node. not mention privacy. It does not need to; this is taken Some nodes where a player i moves are grouped together care of by choosing the utilities appropriately. Pass and into an information set for player i. Intuitively, if player I [2008] show that, under minimal assumptions, this defi- i is at some node in an information set I, then i does not nition is essentially equivalent to a variant of zero knowl- know which node of I describes the true situation; thus, at edge [Goldwasser, Micali, and Rackoff 1989] called precise all nodes in I, i must make the same move. An augmented zero knowledge [Micali and Pass 2006]. Thus, the two ap- game is an extensive game with one more feature: associ- proaches used for dealing with “deviating” players in two ated with each node in the game tree where player i moves game theory and cryptography—Nash equilibrium and zero- is the level of awareness of player i—the set of histories that knowledge “simulation”—are intimately connected; indeed, player i is aware of. they are essentially equivalent once we take computation We use the player’s awareness level as a way of keep- into account appropriately. ing track of how the player’s awareness changes over time. For example, perhaps A playing acrossA will result in B be- coming aware of the possibility of playing downB. In finan- cial settings, one effect of players using certain investment strategies is that other players become aware of the possibil- ity of using that strategy. Strategic thinking in such games must take this possibility into account. We would model this possibility by having some probability of B’s awareness level changing. (The formal definition of augmented game can be found in [Halpern and Rˆego 2006].) For example, suppose that in the game shown in Figure 1 Figure 3: The augmented game ΓB. • players A and B are aware of all histories of the game; • player A is uncertain as to whether player B is aware of augmented game, but a collection of them. Moreover, we run hacross , down i and believes that he is unaware of A B need to describe, at each history in an augmented game, it with probability p; and which augmented game the player playing at that history be- • the type of player B that is aware of the run hacrossA, lieves is the actual augmented game being played. downBi is aware that player A is aware of all histories, To capture these intuitions, starting with an underlying and he knows A is uncertain about his awareness level extensive-form game Γ, we define a game with awareness and knows the probability p. based on Γ to be a tuple Γ∗ = (G, Γm, F), where Because A and B are actually aware of all histories of • G is a countable set of augmented games based on Γ, of the underlying game, from the point of view of the modeler, which one is Γm; the augmented game is essentially identical to the game de- + scribed in Figure 1, with the awareness level of both players • F maps an augmented game Γ ∈ G and a history h in Γ+ such that P +(h)= i to a pair (Γh, I), where Γh ∈ G A and B consisting of all histories of the underlying game. h However, when A moves at the node labeled A in the mod- and I is an information set for player i in game Γ . eler’s game, she believes that the actual augmented game is Intuitively, Γm is the game from the point of view of an om- ΓA, as described in Figure 2. In ΓA, nature’s initial move niscient modeler. If player i moves at h in game Γ+ ∈ G captures A’s uncertainty about B’s awareness level. At the and F(Γ+,h)=(Γh, I), then Γh is the game that i believes information set labeled A.1, A is aware of all the runs of to be the true game when the history is h, and I consists the underlying game. Moreover, at this information set, A of the set of histories in Γh he currently considers possi- believes that the true game is ΓA. ble. For example, in the examples described in Figures 2 At the node labeled B.1, B is aware of all the runs of and 3, taking Γm to the augmented game in Figure 1, we the underlying game and believes that the true game is the have F(Γm, h i) = (ΓA, I), where I is the information set A modeler’s game; but at the node labeled B.2, B is not aware labeled A.1 in Figure 2, and F(Γ , hunaware,acrossAi) = B that he can play downB, and so believes that the true game (Γ , {hacrossAi}). There are a number of consistency con- is the augmented game ΓB described in Figure 3. At the ditions that have to be satisfied by the function F; the details nodes labeled A.3 and B.3 in the game ΓB, neither A nor B can be found in [Halpern and Rˆego 2006]. is aware of the move downB. Moreover, both players think The standard notion of Nash equilibrium consists of a pro- the true game is ΓB. file of strategies, one for each player. Our generalization consists of a profile of strategies, one for each pair (i, Γ′), where Γ′ is a game that agent i considers to be the true game in some situation. Intuitively, the strategy for a player i at Γ′ is the strategy i would play in situations where i believes that the true game is Γ′. To understand why we may need to con- sider different strategies consider, for example, the game of Figure 1. B would play differently depending on whether or not he was aware of downB. Roughly speaking, a profile ~σ of strategies, one for each pair (i, Γ′),isa generalized Nash equilibrium if σi,Γ′ is a best response for player i if the true ′ game is Γ , given the strategies σj,Γ′ being used by the other players in Γ′. As shown in [Halpern and Rˆego 2006], every game with awareness has a generalized Nash equilibrium. A standard extensive-form game Γ can be viewed as a special case of a game with awareness, by taking Γm = Γ, G = {Γm}, and F(Γm,h)= (Γm, I), where I is the infor- Figure 2: The augmented game ΓA. mation set that contains h. Intuitively, Γ corresponds to the game of awareness where it is common knowledge that Γ is As this example should make clear, to model a game with being played. We call this the canonical representation of possibly unaware players, we need to consider, not just one Γ as a game with awareness. It is not hard to show that a strategy profile ~σ is a Nash equilibrium of Γ iff it is a gen- equilibrium [Halpern and Rˆego 2006]. eralized Nash equilibrium of the canonical representation of There has been a great deal of work recently on Γ as a game with awareness. Thus, generalized Nash equi- modeling unawareness in games. The first papers on librium can be viewed as a generalization of standard Nash the topic was by Feinberg [2004, 2005]. My work equilibrium. with Rˆego [2006] was the first to consider awareness Up to now, I have considered only games where players in extensive games, modeling how awareness changed are not aware of their lack of awareness. But in some games, over time. There has been a recent flurry on the a player might be aware that there are moves that another topic in the economics literature; see, for example, player (or even she herself) might be able to make, although [Heifetz, Meier, and Schipper 2006b; Li 2006a; Li 2006b; she is not aware of what they are. Such awareness of un- Ozbay 2007]. Closely related is work on logics that include awareness can be quite relevant in practice. For example, in awareness. This work started in the computer science a war setting, even if one side cannotconceiveof a new tech- literature [Fagin and Halpern 1988], but more recently, the nology available to the enemy, they might believe that there bulk of the work has appeared in the economics literature is some move available to the enemy without understanding (see, for example, [Dekel, Lipman, and Rustichini 1998; what that particular move is. This, in turn, may encourage Halpern2001; HalpernandRˆego2008; peace overtures. To take another example, an agent might Heifetz, Meier, and Schipper 2006a; delay making a decision because she considers it possible Modica and Rustichini 1994; that she might learn about more possible moves, even if she Modica and Rustichini 1999]). is not aware of what these moves are. Although, economists usually interpret awareness as “be- ing able to conceive about an event or a proposition”, there 5 Conclusions are other possible meanings for this concept. For exam- I have considered three ways of going beyond standard Nash ple, awareness may also be interpreted as “understanding equilibrium, which take fault tolerance, computation, and the primitive concepts in an event or proposition”, or as lack of awareness into account, respectively. These are “being able to determine if an event occurred or not”, or clearly only first steps. Here are some directions for fur- as “being able to compute the consequences of some fact” ther research (some of which I am currently engaged in with [Fagin and Halpern 1988]. If we interpret “lack of aware- my collaborators): ness” as “unable to compute” (note that this interpretation is closely related to the discussion of the previous section!), • For example, while (k,t)-robust equilibrium does seem to then awareness of unawareness becomes even more signif- be a reasonable way of capturing some aspects of robust- icant. Consider a chess game. Although all players under- ness, for some applications, it does not go far enough. I stand in principle all the moves that can be made, they are said earlier that in economics, all players were assumed certainly not aware of all consequencesof all moves. A more to be strategic, or “rational”; in distributed computing, accurate representation of chess would model this computa- all players were either “good” (and followed the recom- tional unawareness explicitly. We provide such a represen- mended protocol) or “bad” (in which case they could be tation. arbitrarily malicious). Immunity takes into account the Roughly speaking, we capture the fact that player i is bad players. The definition of immunity requires that the aware that, at a node h in the game tree, there is a move rational players are not hurt no matter what the “bad” that j can make she (i) is not aware by having i’s subjective players do. But this may be too strong. As Ayer et representation of the game include a “virtual” move for j at al. [2005] point out, it is reasonable to expect a certain node h. Since i might have only an incomplete understand- fraction of players in a system to be “good” and follow ing of what can happen after this move, i simply describes the recommended protocol, even if it is not a best reply. what she believes will be the game after the virtual move, to In general, it may be hard to figure out what the best reply the extent that she can. In particular, if she has no idea what is, so if following the recommendedprotocol is not unrea- will happen after the virtual move, then she can describe her sonable, they will do that. (Note that this can be captured beliefs regarding the payoffs of the game. Thus, our rep- in a computational model of equilibrium, by charging for resentation can be viewed as a generalization of how chess switching from the recommended strategy.) programs analyze chess games. They explore the game tree There may be other standard ways that players act irra- up to a certain point, and then evaluate the board position at tional. For example, Kash, Friedman, and I [2007] con- that point. We can think of the payoffs following a virtual sider scrip systems, where players perform work in ex- move by j in i’s subjective representation of a chess game change for scrip. There is a Nash equilibrium where ev- as describing the evaluation of the board from i’s point of eryone uses a threshold strategy, performing work only view. This seems like a much more reasonable representa- when they have less scrip than some threshold amount. tion of the game than the standard complete game tree! Two standard ways of acting “irrationally” in such a sys- All the definitions of games with awareness can be gener- tem are to (a) hoard scrip and (b) provide service for free alized to accommodate awareness of unawareness. In partic- (this is the analogue of posting music on Kazaa). A ro- ular, we can define a generalized Nash equilibriumas before, bust solution should take into account these more standard and once again show that every game with awareness (in- types of irrational behavior, without perhaps worrying as cluding awareness of unawareness) has a generalized Nash much about arbitrary irrational behavior. • The definitions of computational Nash equilibrium con- cept that allows for such composability. sidered only Bayesian games. What would appropriate solution concepts be for extensive-form games? Some References ideas from the work on awareness seem relevant here, es- Abraham, I., D. Dolev, R. Gonen, and J. Halpern (2006). pecially if we think of “lack of awareness” as “unable to Distributed computing meets game theory: Robust compute”. mechanisms for rational secret sharing and multiparty • Where do the beliefs come from in an equilibrium with computation. In Proc. 25th ACM Symposium on Prin- awareness? That is, if I suddenly become aware that you ciples of Distributed Computing, pp. 53–62. can make a certain move, what probability should I assign Abraham, I., D. Dolev, and J. Halpern (2008). Lower to you making that move? Ozbay [2007] proposes a so- bounds on implementing robust and resilient media- lution concept where the beliefs are part of the solution tors. In Fifth Theory of Cryptography Conference, pp. concept. He considers only a simple setting, where one 302–319. player is aware of everything (so that revealing informa- Adar, E. and B. Huberman (2000). Free riding on tion is purely strategic). Can his ideas be extended to a Gnutella. First Monday 5(10). more general setting? Aumann, R. J. (1959). Acceptable points in general co- Agents playing a game can be viewed participat- operative n-person games. In A. Tucker and R. Luce ing in a concurrent, distributed protocol. Game the- (Eds.), Contributions to the Theory of Games IV, ory does not take the asynchrony into account, but Annals of Mathematical Studies 40, pp. 287–324. it can make a big difference. For example, all the Princeton University Press. results from [Abraham, Dolev, Gonen, and Halpern 2006; Axelrod, R. (1984). The Evolution of Cooperation. Basic Abraham, Dolev, and Halpern 2008] mentioned in Section 2 Books. depend on the system being synchronous. Things are more Ayer, A., L. Alvisi, A. Clement, M. Dahlin, J. Martin, and complicted in asynchronous settings. Getting solution con- C. Porth (2005). BAR fault tolerance for cooperative cepts and that deal well with with asynchrony is clearly im- services. In Proc. 20th ACM Symposium on Operat- portant. ing Systems Principles (SOSP 2005), pp. 45–58. Another issue that plays a major role in computer science Barany, I. (1992). Fair distribution protocols or how the but has thus far not been viewed as significant in game the- players replace fortune. Mathematics of Operations ory, but will, I believe, turn out to be important to the prob- Research 17, 327–340. lem of defining appropriate solution concepts, is the ana- Ben-Porath, E. (2003). Cheap talk in games with in- logue of specifying and verifying programs. Games are typ- complete information. Journal of Economic The- ically designed to solve certain problems. Thus, for exam- ory 108(1), 45–71. ple, economists want to design a spectrum auction so that the equilibrium has certain features. As I pointed out in an Ben-Sasson, E., A. Kalai, and E. Kalai (2007). An ap- earlier overview [Halpern 2003], game theory has typically proach to bounded rationality. In Advances in Neu- focused on “small” games: games that are easy to describe, ral Information Processing Systems 19 (Proc. of NIPS such as Prisoner’s Dilemma. The focus has been on sub- 2006), pp. 145–152. tleties regarding basic issues such as rationality and coor- Bernheim, B. D., B. Peleg, and M. Whinston (1989). dination. To the extent that game theory is used to tackle Coalition proof Nash equilibrium: concepts. Journal larger, more practical problems, and especially to the ex- of Economic Theory 42(1), 1–12. tent that it is computers, or software agents, playing games, Dekel, E., B. Lipman, and A. Rustichini (1998). Standard rather than people, it will be important to specify carefully state-space models preclude unawareness. Economet- exactly what a solution to the game must accomplish. For rica 66, 159–173. example, in the context of a spectrum auction, a specifica- Fagin, R. and J. Y.Halpern (1988).Belief, awareness, and tion will have to address what should happen if a computer limited reasoning. Artificial Intelligence 34, 39–76. crashes while an agent is in the middle of transmitting a bid, Feinberg, Y. (2004). Subjective reasoning—games with how to deal with agents bidding on slow lines, dealing with unawareness. Technical Report Resarch Paper Series agents who win but then go bankrupt, and so on. #1875, Stanford Graduate School of Business. Finding logics to reason about solutions, especially do- Feinberg, Y. (2005). Games with incomplete awareness. ing so in a way that takes into account robustness and asyn- Technical Report Resarch Paper Series #1894, Stan- chrony, seems to me a difficult and worthwhile challenge. ford Graduate School of Business. Indeed, one desideratum for a good solution concept is that it should be easy to reason about. Pursuing this theme, com- Fischer, M. J., N. A. Lynch, and M. S. Paterson (1985). puter scientists have learned that one good way of designing Impossibility of distributed consensus with one faulty correct programs is to do so in a modular way. Can a similar processor. Journal of the ACM 32(2), 374–382. idea be applied in game theory? That is, can games designed Forges, F. (1990). Universal mechanisms. Economet- for solving smaller problems be combined in a seamless way rica 58(6), 1341–64. to solve a larger problem. If so, results about composability Fudenberg, D. and J. Tirole (1991). Game Theory. MIT of solutions will be needed; we might want a solution con- Press. Gerardi, D. (2004). Unmediated communication in Kreps, D., P. Milgrom, J. Roberts, and R. Wilson (1982). games with complete and incomplete information. Rational cooperation in finitely repeated prisoners’ Journal of Economic Theory 114, 104–131. dilemma. Journal of Economic Theory 27(2), 245– Goldreich, O. (2004). Foundations of Cryptography, Vol. 252. 2. Cambridge University Press. Kreps, D. M. (1990). Game Theory and Economic Mod- Goldreich, O., S. Micali, and A. Wigderson (1987). How eling. Oxford University Press. to play any mental game. In Proc. 19th ACM Sympo- Lepinski, M., S. Micali, C. Peikert, and A. Shelat (2004). sium on Theory of Computing, pp. 218–229. Completely fair SFE and coalition-safe cheap talk. Goldwasser, S., S. Micali, and C. Rackoff (1989). The In Proc. 23rd ACM Symposium on Principles of Dis- knowledge complexity of interactive proof systems. tributed Computing, pp. 1–10. SIAM Journal on Computing 18(1), 186–208. Li, J. (2006a). Information structures with unawareness. Gordon, D. and J. Katz (2006). Rational secret sharing, Unpublished manuscript. revisited. In SCN (Security in Communication Net- Li, J. (2006b). Modeling unawareness without impossible works) 2006, pp. 229–241. states. Unpublished manuscript. Halpern, J. Y. (2001). Alternative semantics for unaware- Lysyanskaya, A. and N. Triandopoulos (2006). Rational- ness. Games and Economic Behavior 37, 321–339. ity and adveresarial behavior in multi-party comptua- Halpern, J. Y. (2003). A computer scientist looks at game tion. In CRYPTO 2006, pp. 180–197. theory. Games and Economic Behavior 45(1), 114– Micali, S. and R. Pass (2006). Local zero knowledge. In 132. Proc. 38th ACM Symposium on Theory of Computing, Halpern, J. Y. and R. Pass (2008). Game theory with pp. 306–315. costly computation. Unpublished manuscript. Modica, S. and A. Rustichini (1994). Awareness and Halpern, J. Y. and L. C. Rˆego (2006). Extensive games partitional information structures. Theory and Deci- with possibly unaware players. In Proc. Fifth Inter- sion 37, 107–124. national Joint Conference on Autonomous Agents and Modica, S. and A. Rustichini (1999). Unawareness and Multiagent Systems, pp. 744–751. Full version avail- partitional information structures. Games and Eco- able at arxiv.org/abs/0704.2014. nomic Behavior 27(2), 265–298. Halpern, J. Y. and L. C. Rˆego (2008). Interactive un- Moreno, D. and J. Wooders (1996). Coalition-proof equi- awareness revisited. Games and Economic Behav- librium. Games and Economic Behavior 17(1), 80– ior 62(1), 232–262. 112. Halpern, J. Y. and V. Teague (2004). Rational secret shar- Nash, J. (1950). Equilibrium points in n-person games. ing and multiparty computation: extended abstract. In Proc. National Academy of Sciences 36, 48–49. Proc. 36th ACM Symposium on Theory of Computing, pp. 623–632. Neyman, A. (1985). Bounded complexity justifies coop- eration in finitely repated prisoner’s dilemma. Eco- Heifetz, A., M. Meier, and B. Schipper (2006a). Interac- nomic Letters 19, 227–229. tive unawareness. Journal of Economic Theory 130, 78–94. Osborne, M. J. and A. Rubinstein (1994). A Course in Game Theory. MIT Press. Heifetz, A., M. Meier, and B. Schipper (2006b). Unawareness, beliefs and games. Ozbay, E. (2007). Unawareness and strategic announce- Unpublished manuscript, available at ments in games with uncertainty. In Theoretical As- www.econ.ucdavis.edu/faculty/schipper/unawprob.pdf. pects of Rationality and Knowledge: Proc. Eleventh Heller, Y. (2005). A minority-proof cheap-talk protocol. Conference (TARK 2007), pp. 231–238. Unpublished manuscript. Papadimitriou, C. H. and M. Yannakakis(1994). On com- Izmalkov, S., S. Micali, and M. Lepinski (2005). Rational plexity as bounded rationality. In Proc. 26th ACM secure computation and ideal mechanism design. In Symposium on Theory of Computing, pp. 726–733. Proc. 46th IEEE Symposium on Foundations of Com- Pease, M., R. Shostak, and L. Lamport (1980). Reach- puter Science, pp. 585–595. ing agreement in the presence of faults. Journal of Kalai, E. (1990). Bounded rationality and strategic com- the ACM 27(2), 228–234. plexity in repeated games. In Game Theory and Ap- Rubinstein, A. (1986). Finite automata play the repeated plications, pp. 131–157. Academic Press. prisoner’s dilemma. Journal of Economic Theory 39, Kash, I., E. J. Friedman, and J. Y. Halpern (2007). Opti- 83–96. mizing scrip systems: efficiency, crashes, hoarders, Shamir, A., R. L. Rivest, and L. Adelman (1981). Men- and altruists. In Proc. Eighth ACM Conference on tal poker. In D. A. Klarner (Ed.), The Mathematical Electronic Commerce, pp. 305–315. Gardner, pp. 37–43. Prindle, Weber, and Schmidt. Kol, G. and M. Naor (2008). Cryptographyand game the- Urbano, A. and J. E. Vila (2002). Computational com- ory: Designing protocols for exchanging information. plexity and communication: Coordination in two- In Theory of Cryptography Conference, pp. 320–339. player games. Econometrica 70(5), 1893–1927. Urbano, A. and J. E. Vila (2004). Computationally re- stricted unmediated talk under incomplete informa- tion. Economic Theory 23(2), 283–320. Yao, A. (1982). Protocols for secure computation (ex- tended abstract). In Proc. 23rd IEEE Symposium on Foundations of Computer Science, pp. 160–164.