Notes on Equilibria in Symmetric Games

Shih-Fen Cheng, Daniel M. Reeves, Yevgeniy Vorobeychik, Michael P. Wellman University of Michigan Artificial Intelligence Lab. 1101 Beal Avenue, Ann Arbor, MI 48109-2110 USA {chengsf,dreeves,yvorobey,wellman}@umich.edu

Abstract lution procedures. Such techniques, however, may presume or impose constraints on solutions, for example by limit- In a , every player is identical with re- ing consideration to pure or symmetric equilibria. Our own spect to the game rules. We show that a symmetric 2- studies have often focused on symmetric games, and our at- game must have a pure-strategy . tempts to exploit this property naturally raise the question We also discuss Nash’s original paper and its generalized of whether the symmetry-specialized techniques exces- notion of symmetry in games. As a special case of Nash’s sively restrict the search for solutions. theorem, any finite symmetric game has a symmetric Nash Given a symmetric environment, we may well prefer equilibrium. Furthermore, symmetric infinite games with to identify symmetric equilibria, as asymmetric behavior compact, convex strategy spaces and continuous, quasicon- seems relatively unintuitive (Kreps, 1990), and difficult to cave utility functions have symmetric pure-strategy Nash explain in a one-shot interaction. Rosenschein and Zlotkin equilibria. Finally, we discuss how to exploit symmetry for (1994) argue that symmetric equilibria may be especially more efficient methods of finding Nash equilibria. desirable for automated agents, since programmers can then publish and disseminate strategies for copying, without need for secrecy. Despite the common use of symmetric constructions in 1. Introduction game-theoretic analyses, the literature has not extensively investigated the general properties of symmetric games. A game in normal form is symmetric if all agents have One noteworthy exception is Nash’s original paper on equi- the same strategy set, and the payoff to playing a given strat- libria in non-cooperative games (Nash, 1951), which (in ad- egy dependsonly on the strategies being played, not on who dition to presenting the seminal equilibrium existence re- plays them. (We provide a formal definition in the next sec- sult) considers a general concept of symmetric strategy pro- tion.) Many well-known games are symmetric, for exam- files and its implication for symmetry of equilibria. ple the ubiquitous Prisoners’ Dilemma, as well as standard In the next section, we present three results on existence game-theoretic auction models. Symmetric games may nat- of equilibria in symmetric games. The first two identify spe- urally arise from models of automated-agent interactions, cial cases of symmetric games that possess pure-strategy since in these environments the agents may possess identi- equilibria. The third—derived directly from Nash’s origi- cal circumstances, capabilities, and perspectives, by design. nal result—establishes that any finite symmetric game has a Even when actual circumstances, etc., may differ, it is of- (generally mixed) . The remainder of ten accurate to model these as drawn from identical proba- the paper considers how the general existence of symmet- bility distributions. Designers often impose symmetry in ar- ric equilibria can be exploited by search methods that inher- tificial environments constructed to test research ideas—for ently focus on such solutions. example, the Trading Agent Competition (TAC) (Wellman et al., 2001) market games—since an objective of the model structure is to facilitate interagent comparisons. 2. Existence Theorems for Symmetric Games The relevance of symmetry in games stems in large We first give a definition of a general normal-formgame, part from the opportunity to exploit this property for com- adapted from Definition 7.D.2 in Mas-Colell et al. (1995), putational advantage. Symmetry immediately supports followed by the definition of a symmetric game. more compact representation, and may often enable ana- lytic methods specific to this structure, or algorithmic short- Definition 1 For a game with I players, the normal-form cuts leading to significantly more effective or efficient so- representation specifies for each player i a set of strategies Si (with si ∈ Si) and a payoff function ui(s1,...,sI ) giv- (Table 1b). In , both players simultane- ing the von Neumann-Morgenstern utilities for the ously choose an action—heads or tails—and player 1 wins arising from strategies (s1,...,sI ). We denote a game by if the actions match and player 2 wins otherwise. Again, the tuple [I, {Si}, {ui()}]. none of the pure-strategy profiles (HH,TT,HT,TH) consti- tute an equilibrium. Definition 2 (Symmetric Game) A normal-form game is Finally, can we strengthen the conclusion of Theorem 1 symmetric if the players have identical strategy spaces to guarantee symmetric pure-strategy equilibria? A simple (S1 = S2 = . . . = S = S) and u (s ,s− )= u (s ,s− ), I i i i j j j anti- (Table 1c) serves as a counterexam- for s = s and s− = s− for all i, j ∈ {1,...,I}.(s− i j i j i ple. In this game, each player receives 1 when the players denotes all the strategies in profile s except for s .) Thus we i choose different actions and 0 otherwise. The only pure- can write u(t,s) for the utility to any player playing strat- strategy equilibria are the profiles where the players choose egy t in profile s. We denote a symmetric game by the tuple different actions, i.e., asymmetric profiles. [I,S,u()]. Finally, we refer to a strategy profile with all players We next consider other sufficient conditions for pure playing the same strategy as a symmetric profile, or, if such and/or symmetric equilibria in symmetric games. First we a profile is a Nash equilibrium, a symmetric equilibrium. present a lemma establishing properties of the mapping Theorem 1 A symmetric game with two strategies has an from profiles to best responses. The result is analogous to equilibrium in pure strategies. Lemma 8.AA.1 in Mas-Colell et al. (1995), adapted to the case of symmetric games. Proof. Let S = {1, 2} and let [i, I − i] denote the profile with i ∈ {0,...,I} players playing strategy 1 and I − i Lemma 2 In a symmetric game [I,S,u()] with S i nonempty, compact, and convex and with u(si,s1,...,sI ) playing strategy 2. Let us ≡ u(s, [i, I − i]) be the payoff to a player playing strategy s ∈ S in the profile [i, I − i]. De- continuous in (s1,...,sI ) and quasi-concave in si, the fine the boolean function eq(i) as follows: correspondence 0 1 b(s) ≡ arg max u(t,s) u2 ≥ u1 if i =0 t∈S  I I−1 eq(i) ≡  u1 ≥ u2 if i = I = {τ : u(τ,s) = max u(t,s)} i i−1 i i+1 t∈S u1 ≥ u2 & u2 ≥ u1 if i ∈{1 ...I − 1}.  is nonempty, convex-valued, and upper hemicontinuous. In words, eq(i)= TRUE when no unilateral deviation from [i, I − i] is beneficial—i.e., when [i, I − i] is a pure-strategy Proof. Since u(S,s) is the continous image of the com- equilibrium. pact set S, it is compact and has a maximum and so b(s) Assuming the opposite of what we want to prove, is nonempty. b(S) is convex because the set of maxima eq(i) = FALSE for all i ∈ {0,...,I}. We first show by in- of a quasiconcave function (u(·,s)) on a convex set (S) is i i+1 duction on i that u2

(a) (b) (c) Table 1. (a) Rock-Paper-Scissors: a 3-player symmetric game with no pure-strategy equilibria. (b) Matching Pennies: a 2-player asymmetric game with no pure-strategy equilibria. (c) Anti-coordination Game: a symmetric game with no symmetric pure equilibria. convex-valued, and upper hemicontinous correspondence. fixed points of y(·) are equilibria. Of all the pure strategies Thus, by Kakutani’s Theorem1, there exists a fixed point in the support of σ, one, say w, must be worst, implying of b(·) which implies that there exists an s ∈ S such that u(w, σ) ≤ u(σ, σ) which implies that gw(σ)=0.  s ∈ b(s) and so all playing s is a symmetric equilibrium. Assume y(σ) = σ. Then y must not decrease σw. The numerator is σw so the denominator must be 1, which im- Finally, we present Nash’s (1951) result that finite sym- plies that for all s ∈ S, gs(σ)=0 and so all playing σ is an metric games have symmetric equilibria.2 This is a special equilibrium. case of his result that every finite game has a “symmetric” Conversely, if all playing σ is an equilibrium then all the equilibrium, where Nash’s definition of a symmetric pro- g’s vanish, making σ a fixed point under y(·). file is one invariant under every automorphism of the game. Finally, since y(·) is a continuousmapping of a compact, This turns out to be equivalent to defining a symmetric pro- convex set, it has a fixed point by Brouwer’s Theorem.  file as one in which all the symmetric players (if any) are playing the same mixed strategy. In the case of a symmetric game, the two notions of a symmetric profile (invariant un- 3. Computation of Symmetric Equilibria in der automorphisms vs. simply homogeneous) coincide and Symmetric Games we have our result. Here we present an alternate proof of the result, mod- Having affirmed the existence of symmetric equilibria in eled on Nash’s seminal proof of the existence of (possibly symmetric games, we consider next the computational im- asymmetric) equilibria for general finite games. plications for game solving. The methods considered here Theorem 4 A finite symmetric game has a symmet- apply to finding one or a set of Nash equilibria (not all equi- ric mixed-strategy equilibrium. libria or equilibria satifying given properties). Proof. For each pure strategy s ∈ S, define a continuous function of a mixed strategy σ by 3.1. Existing Game Solvers and Symmetry

gs(σ) ≡ max(0,u(s, σ) − u(σ, σ)). A symmetric N-player game with S strategies has N+S−1 In words, gs(σ) is the gain, if any, of unilaterally deviating N different profiles. Without exploiting symme- from the symmetric mixed profile of all playing σ to play- try, the payoff matrix requires SN cells. This entails a huge ing pure strategy s. Next define computational cost just to store the payoff matrix. For ex- ample, for a 5-player game with 21 strategies, the payoff σs + gs(σ) ys(σ) ≡ . matrix needs over four million cells compared with 53 thou- 1+ ∈ gt(σ) Pt S sand when symmetry is exploited. The set of functions ys(·)∀s ∈ S defines a mapping from GAMBIT (McKelvey et al., 1992), the state-of-the-art the set of mixed strategies to itself. We first show that the tool for solving finite games, employs various algorithms (McKelvey and McLennan, 1996), none of which exploit 1 For this and other fixed point theorems (namely Brouwer’s, used in symmetry. In our experience (Reeves et al., to appear; Theorem 4) see, for example, the mathematical appendix (p952) of MacKie-Mason et al., 2004; Wellman et al., 2004), this ren- Mas-Colell et al. (1995). 2 This result can also be proved as a corollary to Theorem 3 by view- ders GAMBIT unusableon many games of interest for which ing a finite game as an infinite game with strategy sets being possible methods that do exploit symmetry yield results. We discuss mixtures of pure strategies. two such methods in the next section. 3.2. Solving Games by Function Minimization them in successive generations so that strategies that per- form well increase in the population at the expense of low- One of the many characterizations (McKelvey and performing strategies. The proportion pg(s) of the popula- McLennan, 1996) of a symmetric Nash equilibrium is as tion playing strategy s in generation g is given by a global minimum of the following function from mixed strategies to the reals: pg(s) ∝ pg−1(s) · (EPs − W ),

f(p)= max[0,u(s,p) − u(p,p)]2, where EPs is the expected payoff for pure strategy s against X s∈S N − 1 players all playing the mixed strategy correspond- ing to the population proportions, and W is a lower bound where u(x, p) is the payoff from playing strategy x against on payoffs (e.g., the minimum value in the payoff matrix) everyone else playing strategy p. The function f is bounded which serves as a dampening factor. To calculate the ex- below by zero and in fact for any equilibrium p, f(p) is pected payoff EPs from the payoff matrix, we average the zero. This is because f(p) is positive iff any pure strategy is payoffs for s in the profiles consisting of s concatenated a strictly better response than p itself. with every profile of N − 1 other agent strategies, weighted This means that we can search for symmetric equilib- by the probabilities of the other agents’ profiles. The proba- ria in symmetric games using any available function min- bility of a particular profile (n1, ...nS) of N agents’ strate- imization technique. In particular, we can search for the gies, where ns is the number of players playing strategy s, root of f using the Amoeba algorithm (Press et al., 1992), is N! a procedure for nonlinear function minimization based on · p(1)n1 · · · p(S)nS . the Nelder-Mead method (Nelder and Mead, 1965). In our n1! · ... · nS! previous work (Reeves et al., to appear) we have used an (This is the multinomial coefficient multiplied by the prob- adaptation of Amoeba developed by Walsh et al. (2002) for ability of a profile if order mattered.) finding symmetric mixed-strategy equilibria in symmetric We verify directly that the population update process games. has converged to a Nash equilibrium by checking that the evolved strategy is a best response to itself. 3.3. Solving Games by Replicator Dynamics 4. Discussion In his original exposition of the concept, Nash (1950) suggested an evolutionary interpretation of the Nash equi- A great majority of existing solution methods for games librium. That idea can be codified as an algorithm for find- do not exploit symmetry. Consequently, many symmetric ing equilibria by employing the replicator dynamics formal- games are often practically unsolvable using these methods, ism, introduced by Taylor and Jonker (1978) and Schuster while easily solvable using methods that do exploit symme- and Sigmund (1983). Replicator dynamics refers to a pro- try and restrict the solution space to symmetric equilibria. cess by which a population of strategies—where popula- We have presented some general results about symmetric tion proportions of the pure strategies correspond to mixed equilibria in symmetric games and discussed solution meth- 3 strategies—evolves over generations by iteratively adjust- ods that exploit the existence result. ing strategy populations according to performance with re- Since exploitation of symmetry has been known to pro- spect to the current mixture of strategies in the population. duce dramatic simplifications of optimization problems When this process reaches a fixed point, every pure strat- (Boyd, 1990) it is somewhat surprising that it has not been egy that hasn’t died out is performingequally well given the actively addressed in the literature. In addi- current strategy mixture. Hence the final strategy mix in the tion to the methods we have used for game solving and population corresponds (under certain conditions (Reeves that we discuss in Section 3 (replicator dynamics and func- et al., to appear)) to a symmetric mixed strategy equilib- tion minimization with Amoeba), we believe symme- rium. Although a more general form of replicator dynam- try may be productively exploited for other algorithms as ics can be applied to asymmetric games (Gintis, 2000), it is well. For example, the well-known Lemke-Howson algo- particularly suited to searching for symmetric equilibria in rithm for 2-player (bimatrix) games lends itself to a sim- symmetric games. plification if the game is symmetric and if we restrict our To implement this approach, we choose an initial pop- attention to symmetric equilibria: we need only one pay- ulation proportion for each pure strategy and then update off matrix, one mixed strategy vector, and one slackness vector, reducing the size of the Linear Complementar- 3 Instead of using discrete generations, the process can be defined con- ity Problem by a factor of two. As we discuss in Sec- tinuously by a system of differential equations. tion 3.1, exploiting symmetry in games with more than two players can yield enormous computational sav- William E. Walsh, Rajarshi Das, Gerald Tesauro, and Jef- ings. frey O. Kephart. Analyzing complex strategic interac- tions in multi-agent systems. In AAAI-02 Workshop on References Game-Theoretic and Decision-Theoretic Agents, Edmon- ton, 2002. John H. Boyd. Symmetries, dynamic equilibria, and the Michael P. Wellman, Joshua Estelle, Satinder Singh, Yev- value function. In R. Ramachandran and R. Sato, ed- geniy Vorobeychik, Christopher Kiekintveld, and Vishal itors, Conservation laws and symmetry: applications to Soni. Strategic interactions in a supply chain game. Tech- economics and finance. Kluwer, Boston, 1990. nical report, University of Michigan, 2004. Herbert Gintis. Game Theory Evolving. Princeton Univer- Michael P. Wellman, Peter R. Wurman, Kevin O’Malley, sity Press, 2000. Roshan Bangera, Shou-de Lin, Daniel Reeves, and David M. Kreps. Game Theory and Economic Modelling. William E. Walsh. Designing the market game for a trad- Oxford University Press, 1990. ing agent competition. IEEE Internet Computing, 5(2): 43–51, 2001. Jeffrey K. MacKie-Mason, Anna Osepayshvili, Daniel M. Reeves, and Michael P. Wellman. Price prediction strate- gies for market-based scheduling. In Fourteenth Interna- tional Conference on Automated Planning and Schedul- ing, Whistler, BC, 2004. Andreu Mas-Colell, Michael D. Whinston, and Jerry R. Green. Microeconomic Theory. Oxford University Press, New York, 1995. Richard D. McKelvey and Andrew McLennan. Computa- tion of equilibria in finite games. In Handbook of Com- putational Economics, volume 1. Elsevier Science, 1996. Richard D. McKelvey, Andrew McLennan, and Theodore Turocy. Gambit game theory analysis software and tools, 1992. http://econweb.tamu.edu/gambit. John Nash. Non-cooperative games. PhD thesis, Princeton University, Department of Mathematics, 1950. John Nash. Non-cooperative games. Annals of Mathemat- ics, 2(54):286–295, 1951. J. A. Nelder and R. Mead. A simplex method for function minimization. Computer Journal, 7:308–313, 1965. William H. Press, Saul A. Teukolsky, William T. Vetterling, and Brian P. Flannery. Numerical Recipes in C. Cam- bridge University Press, second edition, 1992. Daniel M. Reeves, Michael P. Wellman, Jeffrey K. MacKie- Mason, and Anna Osepayshvili. Exploring bidding strategies for market-based scheduling. Decision Sup- port Systems, to appear. Jeffrey S. Rosenschein and Gilad Zlotkin. Rules of En- counter: Designing Conventions for Automated Negoti- ation among Computers. MIT Press, 1994. P. Schuster and K. Sigmund. Replicator dynamics. Journal of Theoretical Biology, 100:533–538, 1983. P. Taylor and L. Jonker. Evolutionary stable strategies and game dynamics. Mathematical Biosciences, 40:145–156, 1978.