review articles

1_CACM_V51.8.indb 74 7/21/08 10:13:35 AM review articles

doi:10.1145/1378704.1378721 University, under the leadership of John The most dramatic interaction between CS von Neumann, in the 1950s.a In this article I try to do two things: and GT may involve game-theory pragmatics. identify the main areas of interaction between computer science and game BY Yoav Shoham theory so far; and point to where the most interesting interaction yet may lie—in an area that is still relatively un- derexplored. The first part aims to be an unbiased Computer survey, but it is impossible to avoid bias altogether. Ten researchers survey- ing the interactions between CS and GT would probably write 10 different Science and types of reports. Indeed, several already have (as I will discuss). Moreover, in this brief discussion I cannot possibly do justice to all the work taking place Game Theory in the area. So I try to compensate for these limitations in two ways: I provide a balanced set of initial pointers into the different subareas, without regard to the amount or nature of work that has taken place in each; and I point the reader to other relevant surveys of the CS-GT interaction, each having its own take on things. Game theory has influenced many fields, The second part is decidedly subjec- including economics (its initial focus), political tive, but it is still meant to be broadly science, biology, and many others. In recent years, relevant both to computer scientists and game theorists interested in the in- its presence in computer science has become teraction between the disciplines. impossible to ignore. GT is an integral part of Lessons from Kalai (1995) artificial intelligence (AI), theory, e-commerce, My departure point is a 13-year-old sur- networking, and other areas of computer science, vey paper by E. Kalai,16 a game theorist and it is routinely featured in the field’s leading with algorithmic sensibilities. Geared primarily toward computer scientists, journals and conferences. One reason is application the paper took stock of the interac- pull: the Internet calls for analysis and design of tions between game theory, operations systems that span multiple entities, each with its research, and computer science at the time. It points to the following areas: own information and interests. Game theory, for all 1. Graphs in games its limitations, is by far the most developed theory 2. The complexity of solving a game 3. Multiperson operations research of such interactions. Another reason is technology 4. The complexity of playing a game push: the mathematics and scientific mind-set of 5. Modeling bounded rationality. game theory are similar to those that characterize The reason I start with this paper, be- sides providing the interesting perspec- many computer scientists. Indeed, it is interesting tive of a non-computer scientist, is the to note that modern computer science and modern comparison with current CS-GT interac- game theory originated in large measure at the same a I thank Moshe Tennenholtz for this observa-

ILLUSTRATION BY JEAN FRANCOIS PODEVIN BY ILLUSTRATION place and time—namely at Princeton tion, which is especially true of GT and AI.

august 2008 | vol. 51 | no. 8 | communications of the acm 75

1_CACM_V51.8.indb 75 7/21/08 10:13:36 AM review articles

tion, as both the matches and mismatches is known to exist,27 the computation of maximally harmful. Recent work, how- are instructive. Looking at the interactions a sample Nash equilibrium was shown ever, has begun to bridge these gaps. between CS and GT taking place today, to be complete for this class,2 and the This third category blends into the one can identify the following foci: problem of computing Nash equilib- fourth one, which is research moti- a. Compact game representations; ria with specific properties was shown vated by specific applications that have b. Complexity of, and algorithms for, to be NP-hard.4, 10 At the same time, emerged in the past decade. For exam- computing solution concepts; algorithms—some quite sophisticated, ple, the domain of networking has given c. Algorithmic aspects of mecha- and all exponential in the worst case— rise to a literature on so-called “price nism design; have been proposed to compute Nash of anarchy” (which captures the inef- d. Game-theoretic analysis equilibria.11, 41 Somewhat surprisingly, ficiency of equilibria in that domain), inspired by specific applications; recent experiments have shown that a games of routing, networking-forma- e. Multiagent learning; relatively simple search algorithm sig- tion games, and peer-to-peer networks. f. Logics of knowledge and belief, nificantly outperforms more sophisti- Other domains include sponsored and other logical aspects of cated algorithms.31 This is an active area search auctions, information markets, games.b that promises many additional results. and reputation systems. This combina- The crude mapping between this list The third match is somewhat less tight tion of the third and fourth categories and Kalai’s is as follows: than the first two. There are at least two is arguably the most active area today kinds of optimization one could speak at the interface of CS and GT, and many 1995 2008 about in a game-theoretic setting. The aspects of it are covered in Nisan et al.,25 1 •• a first is computing a best response toa which is an extensive edited collection 2 •• b fixed decision by the other agents; this is of surveys. The popularity of this area is 3 •• c, d of course the quintessential single-agent perhaps not surprising. The relevancies 4 optimization problem of operations re- of specific applications speak for them- 5 search and AI, among other fields. The selves (although arguments remain e, second is the optimization by the design- about whether the traditional game-the- f er of a mechanism aimed at inducing oretic analysis is an appropriate one). games with desirable equilibria. More generally, it is not surprising that Here, I discuss the areas that match This so-called “mechanism design” mechanism design struck a chord in up (1••a, 2••b, 3••c, d), then turn to has been the focus of much work in CS, given that much of CS’s focus is on the currently active areas that were not computer science. One reason is the the design of algorithms and protocols. discussed by Kalai (e, f), and finish with interesting interaction between tradi- Mechanism design is the one area with- the orphans on the other side (4, 5) that tional CS problems (such as optimi- in GT that adopts such a design stance. were discussed by Kalai but not yet vig- zation and approximation) and tradi- The fifth category active today is mul- orously pursued. tional mechanism-design issues (such tiagent learning, also called “interactive There has been substantial work as incentive compatibility, individual learning” in the game-theory literature.c on compact and otherwise specialized rationality, and social-welfare maximi- Multiagent learning, long a major focus game representations. Some of them zation). A good example is the interac- within game theory, has been rediscov- are indeed graph-based—graphical tion between the Vickrey-Clarke-Groves ered with something of a vengeance in games,18 local-effect games,21 MAIDS,19 mechanism and shortest-path computa- computer science and in particular AI; and Game networks,20 for example. The tion;26 another is the literature on com- witness special issues devoted to it in graph-based representations extend binatorial auctions,6 which combine the Journal of Artificial Intelligence39 and also to coalition game theory.7 But spe- a weighted-set-packing-like NP-hard the Machine Learning Journal.12 For com- cialized representations exist that are optimization problem with incentive puter science, the move from single- not graph based, such as those that are issues. The interplay between mecha- agent learning to multiagent learning multi-attribute based5 and logic based.15 nism design and cryptography is worth is interesting not only because it calls I believe this area is ripe for additional particular mention. Though both are in for new solutions but also because the work—regarding, for example, the strat- the business of controlled dissemina- very questions change. When multiple egy space of agents described using con- tion of information, they are different in agents learn concurrently, one can- structs of programming languages. significant ways. For one thing, they are not distinguish between learning and The complexity of computing a sam- dual in the following sense: mechanism teaching, and the question of “optimal” ple Nash equilibrium (as well as other design attempts to force the revelation learning is no longer well defined (just solution concepts) has been the focus of information, while cryptography at- as the more general notion of an “op- of much interest in CS, especially with- tempts to allow its hiding. For another, timal policy” ceases to be meaningful in the theory community. A new com- they traditionally embody quite differ- when one moves to the multiagent set- plexity class—PPAD—was proposed to ent models of paranoia. Game theory ting). For a discussion of this phenom- handle problems for which a solution assumes an even-keeled expected utility enon, see the Journal of Artificial Intelli- maximization on the part of all agents, gence special issue cited earlier.39 b This current survey originated in a presenta- while cryptography is more simple- tion made at a December 2007 festschrift in minded: it assumes that “good” agents c Kalai’s omission of this area is ironic, as he co- honor of E. Kalai. act as instructed, while “bad” agents are authored one of its seminal papers.

76 communications of the acm | august 2008 | vol. 51 | no. 8

1_CACM_V51.8.indb 76 7/21/08 10:13:36 AM review articles

The sixth and final major area of fo- ward computer scientists, is a concise cus, also one not discussed in Kalai,16 five-page paper summarizing the main is called “interactive epistemology” in complexity and algorithmic issues at game theory and simply “reasoning the interface of CS and GT circa 2001. about knowledge and belief” in com- ˲˲ The 21-page paper by Halpern13 puter science. Starting in the mid-1980s, I expect game- is similar to Linial in that it is geared this area was for a while the most active theory pragmatics toward game theorists and its main fo- focus of interaction between computer cus is distributed systems, but having science (including distributed systems, to be as critical been published a decade later it is more AI, and theory) and game theory. Be- to reducing game current. The work later evolved into a side game theory, it established deep 17-page survey14 with an abbreviated ties with philosophy and mathematical theory to practice discussion of distributed computing logic, culminating in the seminal book and additional material on complexity by Fagin et al.8, d It is interesting to spec- as language considerations, price of anarchy, me- ulate why this area was omitted from pragmatics have diators, and other topics. Kalai’s list, even although it predates ˲˲ Roughgarden’s 30-page work is his paper by a decade, and why today it been to analyzing a detailed survey of a specific topic— is not as broadly populated as the other human discourse namely, the complexity of computing areas. I think the reason is that the sub- a sample Nash equilibrium.32 Geared ject matter is more foundational, pri- or understanding mostly toward economists, it includes marily non-algorithmic, and appeals to language by ample background material on relevant a smaller sliver of the two communities. concepts from complexity theory. Be that as it may, it remains a key area of computers. The material discussed so far is not interaction between the two fields. only prominently featured in computer These six areas are where most of science journals and conferences but the action has been in past years, but by also is beginning to find its way into listing only them and being brief about textbooks.35 These areas will undoubt- each one, I have by necessity glossed edly continue to flourish. But now I over some other important areas. The want to turn our attention to the two references compensate for this omis- closely-related areas—4 and 5—listed sion to some extent. In addition, the by Kalai that have not been looked at as reader is referred to the following addi- closely by the community at large, CS tional surveys, all by computer scientists in particular. I do this for two reasons: I who each have a slightly different slant. believe they are critical to the future suc- Most of these works go into considerably cess of game theory, and I believe that more detail about some of the topics. CS can play an important role in them. ˲˲ The earliest relevant survey is prob- They both have to do with incorporat- ably by Linial.22 Geared primarily toward ing practical considerations into the game theorists, this 58-page report has model of rationality that is inherent to deep coverage of game-theoretic as- game theory. To repeat the caveat stated pects of distributed systems, fault-toler- earlier: unlike the material so far, the re- ant computing, and cryptography, and maining discussion is future-directed, it also touches on computation of game speculative, and subjective. theoretic concepts, games and logic, and other topics. Lessons from Linguistics ˲˲ Papadimitriou’s survey29 geared to- The field of linguistics distinguishes among syntax, semantics, and prag- matics. Syntax defines the form of lan- d That book focused on static aspects of knowl- edge and belief, which, notwithstanding the guage, semantics defines its meaning, substantial computer-science credentials of and pragmatics defines its use. While the authors, raise an interesting contrast be- the three interact in important ways, tween the computer-science and game-theory the distinctions have proved very use- literature in these areas. In game theory, static ful. I believe that game theory may do theories are indeed the primary focus, where- as in computer science—in particular, in data- well to make similar distinctions, and base theory and artificial intelligence—belief that CS can help in the process. Just as revision and other dynamic theories30 (includ- in the case in linguistics, it is unlikely ing the entire mini-industry of nonmonotonic that game-theory pragmatics will yield logics9) play an equal if not greater role. In- deed, recent work at the interface of logic and to unified clean theories, as do syntax game theory37 extends the static treatment of and semantics. But I expect game-theo- Fagin et al.8 in a dynamic direction. ry pragmatics to be as critical to reduc-

august 2008 | vol. 51 | no. 8 | communications of the acm 77

1_CACM_V51.8.indb 77 7/21/08 10:13:36 AM review articles

ing game theory to practice as language the basic workhorses of game theory— pragmatics have been to analyzing hu- the Nash equilibrium and its many man discourse or understanding lan- variants—that have so far served as the guage by computers. very basic analysis tool of strategic inter- The distinction between the syntax actions. Questioning the role of equilib- and semantics of games is, I think, quite Science operates rium analysis will be viewed by some in important, as some of the disputes with- at many levels. GT as act of heresy, but real life suggests in game theory regarding the primacy of that we may have no choice. For exam- different game representations (for ex- For some, it is ple, in the trading agent competition, ample, the strategic and extensive forms) sufficient that Nash equilibrium of the game did not suffer from the lack of this distinction. It play a role in almost any participating might, however, be presumptuous for CS scientific theories program,42 and this is certainly true as to intrude on this debate, except insofar well of the more established chess and as it lends logical insights.38 Indeed, per- be clever, beautiful, checkers competitions. haps this is more the role of mathemati- and inspirational. It is premature to write off the Nash cal logic than of CS per se. equilibrium as irrelevant, however. For But where CS can truly lead the way Others require example, two programs competing in is in game theory’s pragmatics. Game that any science the TAC did in fact make use of what theory as we know it embodies radical can be viewed as approximate empiri- idealizations, which include the infinite eventually cal NE.42 Another striking example is the capacity of agents to reason and the in- make contact computation of equilibria in a simplified finite mutually recursive modeling of game tree by a top-scoring program in a agents. Backing off from these strong with compelling poker competition.43 It could be argued assumptions has proven challenging. applications that maxmin strategies, which coincide A fairly thin strand of work under the with equilibrium strategies in zero-sum heading of “bounded rationality” in- and be subjected games, do play an important pragmatic volves games played by automata.33 This role. But computation of either maxmin is an important area of research that to empirical or equilibrium strategies in competi- sometimes makes deep connections evaluation. tions has certainly been the exception to between the two fields. For example, the rule. The more common experience early results showed that one of the is that one expends the vast majority of well-known pesky facts in game theo- the effort on traditional AI problems ry—namely, that constant “defection” such as designing a good heuristic func- is the only subgame-perfect equilib- tion, searching, and planning. Only a rium in the finitely repeated prisoner’s little—albeit important—time is spent dilemma game—ceases to hold true if reasoning about the opponent. the players are finite automata with suf- The impact of such pragmatic con- ficiently few states.24, 28 A more recent re- siderations on game theory can be sult shows that when players in a game dramatic. Rather than start from very are computer programs, one obtains strong idealizing assumptions and awk- phenomena akin to the Folk Theorem wardly try to back off from them, it may for repeated games.36 prove more useful or accurate to start This connection between theoretical from assumptions of rather limited rea- models of computation and game theo- soning and mutual modeling, and then ry is quite important and beautiful, but judiciously add what is appropriate for it constitutes a fairly narrow interpreta- the situation being modeled. Which in- tion of the term “bounded rationality.” cremental-modeling approach will out The term should perhaps be reserved has yet to be seen, but the payoff both for describing a much broader research for CS and GT can be substantial. agenda—one that may encourage more The second direction is radical in radical departures from the traditional a different way. Game theory adopts view in game theory. Let me mention a fairly terse vocabulary, inheriting it two directions that I think would be from decision theory and the found- profitable (and difficult) to pursue un- aions of statistics.e In particular, agents der this broader umbrella. When one takes seriously the notion e Parenthetically, it can be remarked that 34 of agents’ limited reasoning powers, Savage’s setting, on which the modern Bayesian framework is based, does not have it is not only some of the answers that an obvious extension to the multi-agent case. begin to change; the questions them- However, this is not the focus of the point I am selves must be reconsidered. Consider making here.

78 communications of the acm | august 2008 | vol. 51 | no. 8

1_CACM_V51.8.indb 78 7/21/08 10:13:36 AM review articles

have “strategies,” which have minimal it may already lead to a certain extent Marseille, 2004. Games and Economic Behavior, P. Reny (ed.), in press. structure, and motivations, which are beyond the obvious and the familiar. 18. Kearns, M.J., Littman, M.L., and Singh, S.P. Graphical encapsulated in a simple real-valued Here theory and application corrobo- models for game theory. UAI, 2001. 19. Koller, D. and Milch, B. Multiagent influence diagrams utility function. (This in fact carries rate each other mutually. Beyond this for representing and solving games. IJCAI, 2001. even less information than is suggested lies the field of real success: genuine 20. LaMura, P. Game networks. UAI, 2000. 21. Leyton–Brown, K. and Tennenholtz, M. Local–effect by the use of numbers, as the theory is predictions by theory. It is well known games. IJCAI, 2003. unchanged by any positive affine trans- that all mathematized sciences have 22. Linial, N. Game theoretic aspects of computing. In Handbook of Game Theory, R.J. Aumann and S. Hart formation of the numbers.) In real life, gone through these successive phases (eds.), 1339–1395, Elsevier Science, 1994. and in computer programs attempting of evolution.40 23. Minsky, M. The Emotion Machine: Commonsense Thinking, Artificial Intelligence, and the Future of the to behave intelligently, we find use for a Human Mind, New York: Simon and Shuster, 2007. 24. Neyman, A. Bounded complexity justifies cooperation much broader vocabulary. For example, So at least von Neumann, the father in finitely repeated prisoner’s dilemma.Economic agents are able to take certain actions of modern-day game theory and com- Letters, 1985, 227–229. 25. Nisan, N., Roughgarden, T., Tardos, E., and Vazirani, V. and not others; have desires, goals, and puter science, attached importance to (eds). Algorithmic Game Theory. Cambridge University intentions (the belief-desire-intention spanning the spectrum from the theo- Press, 2007. 26. Nisan, N. and Ronen, A. Algorithmic mechanism combination giving rise to the pun retical to the applied. Pragmatics may design. In Proceedings of the 31st ACM Symposium on “beady-eye agent architecture”); and be critical to achieving von Neumann Theory of Computing, 1999, 129–140. 27. Papadimitriou, C.H. On the complexity of the parity make plans. Apparently these abstract and Morgenstern’s third stage, and it argument and other inefficient proofs of existence. notions are useful both in effecting could propel a joint endeavor between Journal of Computer and System Sciences 48, 3 (1004), 498–532. intelligent behavior and in reasoning computer science and game theory. 28. Papadimitriou, C.H. and Yannakakis, M. On bounded about it. Philosophers have written rationality and computational complexity. In 1 Proceedings of the Symposium on the Theory of about them (for example, Bratman ) Computing, 1994, 726–733. and there have been attempts—albeit Acknowledgments 29. Papadimitriou, C.H. Algorithms, games, and the Internet. Lecture Notes in Computer Science 2076, preliminary ones—to formalize these I thank Alon Altman, Joe Halpern, Sam Ieong, Daphne Koller, Tim Roughgarden, Moshe Vardi, Mike Wellman, and 2001. intuitions (starting with Cohen and anonymous referees for their useful help and suggestions. 30. Pappas, P. Belief revision. In Handbook of Knowledge Representation. F. van Harmelen, V. Lifschitz, and B. 3 Of course, all errors, opinions, and erroneous opinions are Levesque ). Some in AI have advocated mine alone. Porter (eds.). Elsevier, 2007. 31. Porter, R., Nudelman, E., and Shoham, Y. Simple embracing an even broader vocabulary search methods for finding a Nash equilibrium. In of emotions (such as the recent provoc- Proceedings of the National Conference on Artificial Intelligence, 2004, 664–669. 23 References ative albeit informal book by Minsky. ) 32. Roughgarden, T. Computing equilibria: A 1. Bratman, M.E. Intention, Plans, and Practical Reason. computational complexity perspective. Economic Is game theory missing out by not con- CSLI Publications, Stanford University, 1987. Theory, 2008. 2. Chen, X. and Deng, X. Settling the complexity of sidering these concepts? 33. Rubinstein, A. Modeling Bounded Rationality. MIT 2-player Nash–equilibrium. FOCS, 2006. Press, Cambridge, MA, 1998. 3. Cohen, P.R. and Levesque, H.J. Intention is choice with 34. Savage, L.J. The Foundations of Statistics. John Wiley commitment. Artificial Intelligence 42, 2–3 (1990), Concluding Remarks and Sons, NY, 1954. (Second Edition: Dover Press, 213–261. 1972). Science operates at many levels. For 4. Conitzer, V. and Sandholm, T. Complexity results about 35. Shoham, Y. and Leyton–Brown, K. Multiagent Systems: Nash equilibria. IJCAI. 2003, 761–771. some, it is sufficient that scientific theo- Algorithmic, Game Theoretic, and Logical Foundations. 5. Conitzer, V. and Sandholm, T. Computing Shapley Cambridge University Press, 2008. ries be clever, beautiful, and inspira- values, manipulating value division schemes, and 36. Tennenholtz, M. Program equilibrium. Games and checking core membership in multi–issue domains. tional. Others require that any science Economic Behavior 49, 2004, 363–373. AAAI, 2004. 37. van Benthem, J. Exploring Logical Dynamics. Center eventually make contact with compelling 6. Cramton, P.C., Shoham, Y. and Steinberg, R. (eds). for the Study of Language and Information, Stanford, Combinatorial Auctions. MIT Press, 2006. applications and be subjected to empiri- CA, 1997. 7. Deng, X and Papadimitriou, C.H. On the complexity 38. van Benthem, J. When are two games the same? In cal evaluation. Without personally weigh- of cooperative solution concepts. Mathematics of LOFT–III, 1998 (ILLC preprint, 1999). Operations Research 19, 257, 1994. 39. Vohra, R. and Wellman, M.P. (eds.). Special issue ing in on this emotional debate, I note 8. Fagin, R., Halpern, J.Y., Moses, Y. and Vardi, M.Y. on foundations of multiagent learning. Artificial Reasoning about Knowledge. MIT Press, Cambridge, that in his 2004 presidential address at Intelligence 171, 1 (2007). MA, 1995. 40. von Neumann, J. and Morgenstern, O. Theory of the Second World Congress of the Game 9. Niemela, I., Brewka, G., and Truszczynski, M. Games and Economic Behavior, Second Edition. 17 Nonmonotonic reasoning. In Handbook of Knowledge Theory Society, Kalai reprised the three Princeton University Press, 1947. Representation. F. van Harmelen, V. Lifschitz, and B. 41. von Stengel, B. Computing equilibria for two–person stages of any science as discussed by von Porter. (eds.), Elsevier, 2007. games. In Handbook of Game Theory, vol. III, 10. Gilboa, I. and Zemel, E. Nash and correlated equilibria: Neumann and Morgenstern: chap. 45. R. Aumann and S. Hart (eds.). Elsevier, Some complexity considerations. Games and Amsterdam, 2002, 1723–1759. Economic Behavior, 80–93, 1989. 42. Wellman, M.P., Greenwald, A., and Stone, P. 11. Govindan, S. and Wilson, R. A global Newton method [W]hat is important is the gradual Autonomous Bidding Agents: Strategies and Lessons to compute Nash equilibria. Journal of Economic from the Trading Agent Competition. MIT Press, 2007. development of a theory, based on a Theory, 2003. 43. Zinkevich, M., Bowling, M., and Burch. N. A new 12. Greenwald, A. and Littman, M.L. (eds.). Special issue careful analysis of the ordinary everyday algorithm for generating equilibria in massive zero– on learning and computational game theory. Machine sum games. In Proceedings of the 22nd Conference on interpretation of economic facts. The Learning 67, 1–2, 2007. AI, 2007, 788–793. theory finally developed must be math- 13. Halpern, J.Y. A computer scientist looks at game theory. Games and Economic Behavior 45, 1 (2003), This work has been supported by NSF grant TR–0205633. ematically rigorous and conceptually 114–132. general. Its first applications are neces- 14. Halpern, J.Y. Computer science and game theory: A brief survey. In The New Palgrave Dictionary sarily to elementary problems where the of Economics. S.N. Durlauf and L.E. Blume (eds.), Yoav Shoham (http://cs.stanford.edu/~shoham) is a result has never been in doubt and no Palgrave MacMillan, 2008. professor of computer science at Stanford University, 15. Ieong, S. and Shoham, Y. Marginal Contribution Nets: Stanford, CA. theory is actually required. At this early A compact representation scheme for coalitional games. In Proceedings of the 6th ACM Conference on stage the application serves to corrobo- Electronic Commerce, 2005. rate the theory. The next stage develops 16. Kalai, E. Games, computers, and O.R. In ACM/SIAM Symposium on Discrete Algorithms, 1995. when the theory is applied to somewhat 17. Kalai, E. Presidential address. The Second World more complicated situations in which Congress of the Game Theory Society, © 2008 ACM 0001-0782/08/0800 $5.00

august 2008 | vol. 51 | no. 8 | communications of the acm 79

1_CACM_V51.8.indb 79 7/21/08 10:13:36 AM