DOCSLIB.ORG

A Comparison Between the Usage of Flat and Structured Game Trees for Move Evaluation in Hearthstone

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
A Comparison Between the Usage of Flat and Structured Game Trees for Move Evaluation in Hearthstone A Comparison Between the Usage of Flat and Structured Game Trees for Move Evaluation in Hearthstone Master’s Thesis Markus Zopf Knowledge Engineering Group Technische Universität Darmstadt Thesis Statement pursuant to § 22 paragraph 7 of APB TU Darmstadt I herewith formally declare that I have written the submitted thesis independently. I did not use any outside support except for the quoted literature and other sources mentioned in the paper. I clearly marked and separately listed all of the literature and all of the other sources which I employed when producing this academic work, either literally or in content. This thesis has not been handed in or published before in the same or similar form. In the submitted thesis the written copies and the electronic version are identical in content. Darmstadt, April 7, 2015 Markus Zopf Abstract Since the beginning of research in the field of Artificial Intelligence, games provide challenging problems for intelligent systems. Chess, a fully observable zero-sum game without randomness, was one of the first games investigated in detail. In 1997, nearly 50 years after the first paper about computers playing chess, the Deep Blue system won a six-game match against the chess Grandmaster Garry Kasparov. A lot of other games with an even harder setup were investigated since then. For example, the multiplayer card game poker with hidden information and the game Go with an enormous amount of possible game states are still a challenging task. Recently developed Monte Carlo algorithms try to handle the complexity of such games and achieve significantly better results than other approaches have before. Monte Carlo algorithms use random sampling to estimate the value of game states. They are particularly successful in games where it is hard to define a utility function to calculate the value of game states directly and where random simulations are easy to execute. In this thesis, we investigate two different Monte Carlo approaches in the recently released card game Hearthstone: Heroes of Warcraft. This game combines various difficulties in game playing, like hidden information, randomness, and big game trees. Upper Confidence Bound approaches use the concept of bandits to represent moves shallowly in a game whereas Upper Confidence Bound Applied to Trees algorithms build structured trees to find the best move. We find that both algorithms perform well against different random players. Win rates of about 0.90 can be achieved with low simulation counts used in both algorithms. Using higher simulation counts lead to even higher win rates of about 0.98. The direct comparisons of both algorithms show an unclear result: UCB surpasses UCT when only few simulations are used. This results from not widely enough expanded move trees in the UCT algorithm. If more simulations are available, UCT gets better and better and surpasses UCB. In experiments with the highest simulation counts, UCB beats UCT again. We additionally find that both approaches have different weaknesses when they are applied to the game Hearthstone. These weaknesses result from the enormously high branching factor in Hearthstone and the way moves can be decomposed into atomic actions. To investigate the playing strength of both approaches better, we suggest evaluating the performance of other approaches based on rules or heuristics learned with reinforcement learning and experiment with enhancements for the UCB and the UCT algorithms. Contents 1 Artificial Intelligence and Games 1 1.1 Thesis Structure ........................................................................................................................ 1 1.2 Games as Measurement of Intelligence ...................................................................................... 3 1.2.1 Intelligence ..................................................................................................................... 3 1.2.2 Games ............................................................................................................................. 4 1.3 Games and Optimal Play ........................................................................................................... 5 1.3.1 Simple Games ................................................................................................................. 5 1.3.2 Multiplayer Games .......................................................................................................... 6 1.3.3 Zero-Sum Games ............................................................................................................. 7 1.3.4 Solving Games ................................................................................................................ 8 1.3.5 Solving Two-Player Zero-Sum Games .............................................................................. 8 1.3.6 Search Space Size ........................................................................................................... 9 1.3.7 Randomness.................................................................................................................. 10 1.3.8 Hidden Information ...................................................................................................... 12 1.3.9 Pure and Mixed Strategies ............................................................................................. 15 1.4 Research Question .................................................................................................................. 15 1.5 Similarities to Real World Problems ........................................................................................ 17 2 Hearthstone: Heroes of Warcraft 18 2.1 Introduction ............................................................................................................................ 18 2.2 The Game ............................................................................................................................... 19 2.3 Cards ...................................................................................................................................... 20 2.3.1 Minions ......................................................................................................................... 20 2.3.2 Spells ............................................................................................................................ 21 2.3.3 Weapons ....................................................................................................................... 21 2.4 Abilities ................................................................................................................................... 22 2.5 Composing a Move of Atomic Actions...................................................................................... 24 3 Finding Solutions with Monte Carlo Methods 25 3.1 The Need for Simulations ........................................................................................................ 25 3.2 Game Trees in Hearthstone ..................................................................................................... 26 3.3 Move Tree Complexity ............................................................................................................ 28 3.4 Exploration/Exploitation Tradeoff ........................................................................................... 30 3.5 Bandit Approaches .................................................................................................................. 30 3.5.1 Upper Confidence Bound .............................................................................................. 32 3.5.2 Final Move Selection ..................................................................................................... 35 3.5.3 Anytime Property .......................................................................................................... 36 3.5.4 Aheuristic Property ....................................................................................................... 36 3.6 Monte Carlo Tree Search Approaches ...................................................................................... 37 3.6.1 Selection ....................................................................................................................... 38 3.6.2 Expansion ..................................................................................................................... 40 3.6.3 Simulation .................................................................................................................... 41 3.6.4 Backpropagation ........................................................................................................... 42 3.6.5 Upper Confidence Bound Applied to Trees .................................................................... 42 3.6.6 Asymmetric Property ..................................................................................................... 43 3.7 Post Monte Carlo Strategies .................................................................................................... 43 3.7.1 Do Nothing ................................................................................................................... 43 3.7.2 Additional Random Atomic Action ................................................................................ 44 3.7.3 Additional Random Move .............................................................................................. 44 3.7.4 Additional Random Move with Maximum Length .........................................................
Load more

Recommended publications
  • Game Theory: Static and Dynamic Games of Incomplete Information
    Game Theory: Static and Dynamic Games of Incomplete Information Branislav L. Slantchev Department of Political Science, University of California – San Diego June 3, 2005 Contents. 1 Static Bayesian Games 2 1.1BuildingaPlant......................................... 2 1.1.1Solution:TheStrategicForm............................ 4 1.1.2Solution:BestResponses.............................. 5 1.2Interimvs.ExAntePredictions............................... 6 1.3BayesianNashEquilibrium.................................. 6 1.4 The Battle of the Sexes . 10 1.5AnExceedinglySimpleGame................................ 12 1.6TheLover-HaterGame.................................... 13 1.7TheMarketforLemons.................................... 16 2 Dynamic Bayesian Games 18 2.1ATwo-PeriodReputationGame............................... 19 2.2Spence’sEducationGame.................................. 22 3 Computing Perfect Bayesian Equilibria 24 3.1TheYildizGame........................................ 24 3.2TheMyerson-RosenthalGame................................ 25 3.3One-PeriodSequentialBargaining............................. 29 3.4AThree-PlayerGame..................................... 29 3.5RationalistExplanationforWar............................... 31 Thus far, we have only discussed games where players knew about each other’s utility func- tions. These games of complete information can be usefully viewed as rough approximations in a limited number of cases. Generally, players may not possess full information about their opponents. In particular, players may possess
    [Show full text]
  • Game Theory with Sparsity-Based Bounded Rationality∗
    Game Theory with Sparsity-Based Bounded Rationality Xavier Gabaix NYU Stern, CEPR and NBER March 14, 2011 Extremely preliminary and incomplete. Please do not circulate. Abstract This paper proposes a way to enrich traditional game theory via an element of bounded rationality. It builds on previous work by the author in the context of one-agent deci- sion problems, but extends it to several players and a generalization of the concept of Nash equilibrium. Each decision maker builds a simplified representation of the world, which includes only a partial understanding of the equilibrium play of the other agents. The agents’desire for simplificity is modeled as “sparsity” in a tractable manner. The paper formulates the model in the concept of one-shot games, and extends it to sequen- tial games and Bayesian games. It applies the model to a variety of exemplary games: p beauty contests, centipede, traveler’sdilemma, and dollar auction games. The model’s predictions are congruent with salient facts of the experimental evidence. Compared to previous succesful proposals (e.g., models with different levels of thinking or noisy decision making), the model is particularly tractable and yields simple closed forms where none were available. A sparsity-based approach gives a portable and parsimonious way to inject bounded rationality into game theory. Keywords: inattention, boundedly rational game theory, behavioral modeling, sparse repre- sentations, `1 minimization. [email protected]. For very good research assistance I am grateful to Tingting Ding and Farzad Saidi. For helpful comments I thank seminar participants at NYU. I also thank the NSF for support under grant DMS-0938185.
    [Show full text]
  • Learning About Bayesian Games
    Learning to Play Bayesian Games1 Eddie Dekel, Drew Fudenberg and David K. Levine2 First draft: December 23, 1996 Current revision: July 13, 2002 Abstract This paper discusses the implications of learning theory for the analysis of games with a move by Nature. One goal is to illuminate the issues that arise when modeling situations where players are learning about the distribution of Nature’s move as well as learning about the opponents’ strategies. A second goal is to argue that quite restrictive assumptions are necessary to justify the concept of Nash equilibrium without a common prior as a steady state of a learning process. 1 This work is supported by the National Science Foundation under Grants 99-86170, 97-30181, and 97- 30493. We are grateful to Pierpaolo Battigalli, Dan Hojman and Adam Szeidl for helpful comments. 2 Departments of Economics: Northwestern and Tel Aviv Universities, Harvard University and UCLA. 2 1. Introduction This paper discusses the implications of learning theory for the analysis of games with a move by Nature.3 One of our goals is to illuminate some of the issues involved in modeling players’ learning about opponents’ strategies when the distribution of Nature’s moves is also unknown. A more specific goal is to investigate the concept of Nash equilibrium without a common prior, in which players have correct and hence common beliefs about one another’s strategies, but disagree about the distribution over Nature’s moves. This solution is worth considering given the recent popularity of papers that apply it, such as Banerjee and Somanathan [2001], Piketty [1995], and Spector [2000].4 We suggest that Nash equilibrium without a common prior is difficult to justify as the long- run result of a learning process, because it takes very special assumptions for the set of such equilibria to coincide with the set of steady states that could arise from learning.
    [Show full text]
  • Heiko Rauhut: Game Theory. In: "The Oxford Handbook on Offender
    Game theory Contribution to “The Oxford Handbook on Offender Decision Making”, edited by Wim Bernasco, Henk Elffers and Jean-Louis van Gelder —accepted and forthcoming— Heiko Rauhut University of Zurich, Institute of Sociology Andreasstrasse 15, 8050 Zurich, Switzerland [email protected] November 8, 2015 Abstract Game theory analyzes strategic decision making of multiple interde- pendent actors and has become influential in economics, political science and sociology. It provides novel insights in criminology, because it is a universal language for the unification of the social and behavioral sciences and allows deriving new hypotheses from fundamental assumptions about decision making. The first part of the chapter reviews foundations and assumptions of game theory, basic concepts and definitions. This includes applications of game theory to offender decision making in different strate- gic interaction settings: simultaneous and sequential games and signaling games. The second part illustrates the benefits (and problems) of game theoretical models for the analysis of crime and punishment by going in- depth through the “inspection game”. The formal analytics are described, point predictions are derived and hypotheses are tested by laboratory experiments. The article concludes with a discussion of theoretical and practical implications of results from the inspection game. 1 Foundations of game theory Most research on crime acknowledges that offender decision making does not take place in a vacuum. Nevertheless, most analytically oriented research applies decision theory to understand offenders. Tsebelis (1989) illustrates why this is a problem by using two examples: the decision to stay at home when rain is 1 probable and the decision to speed when you are in a hurry.
    [Show full text]
  • Minimax AI Within Non-Deterministic Environments
    Minimax AI within Non-Deterministic Environments by Joe Dan Davenport, B.S. A Thesis In Computer Science Submitted to the Graduate Faculty Of Texas Tech University in Partial Fulfillment of The Requirements for The Degree of MASTER OF SCIENCE Approved Dr. Nelson J. Rushton Committee Chair Dr. Richard Watson Dr. Yuanlin Zhang Mark Sheridan, Ph.D. Dean of the Graduate School August, 2016 Copyright 2016, Joe Dan Davenport Texas Tech University, Joe Dan Davenport, August 2016 Acknowledgements There are a number of people who I owe my sincerest thanks for arriving at this point in my academic career: First and foremost I wish to thank God, without whom many doors that were opened to me would have remained closed. Dr. Nelson Rushton for his guidance both as a scientist and as a person, but even more so for believing in me enough to allow me to pursue my Master’s Degree when my grades didn’t represent that I should. I would also like to thank Dr. Richard Watson who originally enabled me to come to Texas Tech University and guided my path. I would also like to thank Joshua Archer, and Bryant Nelson, not only for their companionship, but also for their patience and willingness to listen to my random thoughts; as well as my manger, Eric Hybner, for bearing with me through this. Finally I would like to thank some very important people in my life. To my Mother, I thank you for telling me I could be whatever I wanted to be, and meaning it. To Israel Perez for always being there for all of us.
    [Show full text]
  • 6 Dynamic Games with Incomplete Information
    22 November 2005. Eric Rasmusen, [email protected]. Http://www.rasmusen.org. 6 Dynamic Games with Incomplete Information 6.1 Perfect Bayesian Equilibrium: Entry Deterrence II and III Asymmetric information, and, in particular, incomplete information, is enormously impor- tant in game theory. This is particularly true for dynamic games, since when the players have several moves in sequence, their earlier moves may convey private information that is relevant to the decisions of players moving later on. Revealing and concealing information are the basis of much of strategic behavior and are especially useful as ways of explaining actions that would be irrational in a nonstrategic world. Chapter 4 showed that even if there is symmetric information in a dynamic game, Nash equilibrium may need to be refined using subgame perfectness if the modeller is to make sensible predictions. Asymmetric information requires a somewhat different refine- ment to capture the idea of sunk costs and credible threats, and Section 6.1 sets out the standard refinement of perfect bayesian equilibrium. Section 6.2 shows that even this may not be enough refinement to guarantee uniqueness and discusses further refinements based on out-of- equilibrium beliefs. Section 6.3 uses the idea to show that a player’s ignorance may work to his advantage, and to explain how even when all players know something, lack of common knowledge still affects the game. Section 6.4 introduces incomplete information into the repeated prisoner’s dilemma and shows the Gang of Four solution to the Chain- store Paradox of Chapter 5. Section 6.5 describes the celebrated Axelrod tournament, an experimental approach to the same paradox.
    [Show full text]
  • Part II Dynamic Games of Complete Information
    Part II Dynamic Games of Complete Information 83 84 This is page 8 Printer: Opaq 10 Preliminaries As we have seen, the normal form representation is a very general way of putting a formal structure on strategic situations, thus allowing us to analyze the game and reach some conclusions about what will result from the particular situation at hand. However, one obvious drawback of the normal form is its difficulty in capturing time. That is, there is a sense in which players’ strategy sets correspond to what they can do, and how the combination of their actions affect each others payoffs, but how is the order of moves captured? More important, if there is a well defined order of moves, will this have an effect on what we would label as a reasonable prediction of our model? Example: Sequencing the Cournot Game: Stackelberg Competition Consider our familiar Cournot game with demand P = 100 − q, q = q1 + q2,and ci(qi)=0for i ∈{1, 2}. We have already solved for the best response choices of each firm, by solving their maximum profit function when each takes the other’s quantity as fixed, that is, taking qj as fixed, each i solves: max (100 − qi − qj)qi , qi 86 10. Preliminaries and the best response is then, 100 − q q = j . i 2 Now lets change a small detail of the game, and assume that first, firm 1 will choose q1, and before firm 2 makes its choice of q2 it will observe the choice made by firm 1. From the fact that firm 2 maximizes its profit when q1 is already known, it should be clear that firm 2 will follow its best response function, since it not only has a belief, but this belief must be correct due to the observation of q1.
    [Show full text]
  • Communication and Content
    Communication and content Prashant Parikh language Topics at the Grammar-Discourse science press Interface 4 Topics at the Grammar-Discourse Interface Editors: Philippa Cook (University of Göttingen), Anke Holler (University of Göttingen), Cathrine Fabricius-Hansen (University of Oslo) In this series: 1. Song, Sanghoun. Modeling information structure in a cross-linguistic perspective. 2. Müller, Sonja. Distribution und Interpretation von Modalpartikel-Kombinationen. 3. Bueno Holle, Juan José. Information structure in Isthmus Zapotec narrative and conversation. 4. Parikh, Prashant. Communication and content. ISSN: 2567-3335 Communication and content Prashant Parikh language science press Parikh, Prashant. 2019. Communication and content (Topics at the Grammar-Discourse Interface 4). Berlin: Language Science Press. This title can be downloaded at: http://langsci-press.org/catalog/book/248 © 2019, Prashant Parikh Published under the Creative Commons Attribution 4.0 Licence (CC BY 4.0): http://creativecommons.org/licenses/by/4.0/ ISBN: 978-3-96110-198-6 (Digital) 978-3-96110-199-3 (Hardcover) ISSN: 2567-3335 DOI:10.5281/zenodo.3243924 Source code available from www.github.com/langsci/248 Collaborative reading: paperhive.org/documents/remote?type=langsci&id=248 Cover and concept of design: Ulrike Harbort Typesetting: Felix Kopecky Proofreading: Ahmet Bilal Özdemir, Brett Reynolds, Catherine Rudin, Christian Döhler, Eran Asoulin, Gerald Delahunty, Jeroen van de Weijer, Joshua Phillips, Ludger Paschen, Prisca Jerono, Sebastian Nordhoff, Trinka
    [Show full text]
  • Hide and Seek in Arizona∗
    Hide and Seek in Arizona¤ RobertW. Rosenthal,y Jason Shachat,z and Mark Walker.x Revised April 5, 2001 Abstract Laboratory subjects repeatedly played one of two variations of a simple two-person zero-sum game of “hide and seek.” Three puzzling departures from the prescriptions of equilibrium theory are found in the data: an asymmetry related to the player’s role in the game; an asymmetry across the game variations; and positive serial correlation in subjects’ play. Possible explanations for these departures are considered. ¤We are grateful to Todd Swarthout for invaluable research assistance and to Kevin Lang, Mark Machina, and John Wooders for helpful comments. Some of the work was done while Rosenthal was visiting the Department of Economics at MIT and the Department of Political Economy at the University of Siena and while Shachat was on the faculty of the University of California at San Diego. All three authors are grateful to the National Science Foundation for financial support of this research. yDepartment of Economics, Boston University; deceased February 27, 2002 zIBM TJ Watson Research Laboratory; [email protected] xDepartment of Economics, University of Arizona; [email protected] 1 Introduction Barry O’Neill’s insightful 1987 paper revived interest in using experimental methods to test the predictive power of mixed-strategy equilibrium. Following O’Neill’s experiment, a number of other researchers have conducted new experiments based on two-person zero- sum games that have unique mixed-strategy equilibria.1 The evidence for and against equilibrium theory has been mixed: while the theory seems to perform better than it did in the experiments done prior to O’Neill’s,2 there have nevertheless been consistent departures from the implications of equilibrium theory in each of the experiments.
    [Show full text]
  • Forward Induction As Confusion Over the Equilibrium Being Played Out
    Forward Induction as Confusion over the Equilibrium Being Played Out 26 January 1991/11 September 2007 Eric Rasmusen Abstract The Nash equilibrium of a game depends on it being common knowledge among the players which particular Nash equilibrium is being played out. This common knowledge arises as the result of some unspecified background process. If there are multiple equilibria, it is important that all players agree upon which one is being played out. This paper models a situation where there is noise in the background process, so that players sometimes are unknowingly at odds in their opinions on which equilibrium is being played out. Incorporating this possibility can reduce the number of equilibria in a way similar but not identical to forward induction and the intuitive criterion. Eric Rasmusen, Dan R. and Catherine M. Dalton Professor, Department of Business Economics and Public Policy, Kelley School of Business, Indiana University. Visitor (07/08), Nuffield College, Oxford University. Office: 011-44-1865 554-163 or (01865) 554-163. Nuffield College, Room C3, New Road, Oxford, England, OX1 1NF. [email protected]. http://www.rasmusen.org. Copies of this paper can be found at: http://www.rasmusen.org/papers/rasmusen-subgame.pdf. I would like to thank Emmanuel Petrakis and participants in the University of Chicago Theory Workshop and the University of Toronto for helpful comments. 1. Introduction The idea here will be: if players observe actions by player Smith that are compatible with Nash equilibrium E1 but not with Nash equilibrium E2, they should believe that Smith will continue to play according to equilibrium E1, even if they themselves were earlier intending to play according to equilibrium E2.
    [Show full text]
  • On the Notion of Perfect Bayesian Equilibrium∗
    July 24, 2015 Pefect bayesian On the Notion of Perfect Bayesian Equilibrium∗ Julio Gonzalez-D´ ´ıaz Departamento de Estad´ıstica e Investigaci´on Operativa, Universidad de Santiago de Compostela Miguel A. Melendez-Jim´ enez´ † Departamento de Teor´ıa e Historia Econ´omica, Universidad de M´alaga Published in TOP 22, 128-143 (2014) Published version available at http://www.springer.com DOI 10.1007/s11750-011-0239-z Abstract Often, perfect Bayesian equilibrium is loosely defined by stating that players should be sequentially rational given some beliefs in which Bayes rule is applied “whenever possible”. We argue that there are situations in which it is not clear what “whenever possible” means. Then, we provide an elementary definition of perfect Bayesian equilibrium for general extensive games that refines both weak perfect Bayesian equilibrium and subgame perfect equilibrium. JEL classification: C72. Keywords: non-cooperative game theory, equilibrium concepts, perfect Bayesian, Bayes rule. 1 Introduction Perfect Bayesian equilibrium is profusely used to analyze the game theoretical models that are derived from a wide variety of economic situations. The common understanding is that a perfect Bayesian equilibrium must be sequentially rational given the beliefs of the players, which have to be computed using Bayes rule “whenever possible”. However, the literature lacks a formal and tractable definition of this equilibrium concept that applies to general extensive games and, hence, it is typically the case that perfect Bayesian equilibrium is used without providing a definition beyond the above common understanding. The main goal of this paper is to make a concise conceptual and pedagogic contribution to the theory of equilibrium refinements.
    [Show full text]
  • On the Notion of Perfect Bayesian Equilibrium
    A Service of Leibniz-Informationszentrum econstor Wirtschaft Leibniz Information Centre Make Your Publications Visible. zbw for Economics González-Díaz, Julio; Meléndez-Jiménez, Miguel A. Working Paper On the notion of perfect bayesian equilibrium Discussion Paper, No. 1446 Provided in Cooperation with: Kellogg School of Management - Center for Mathematical Studies in Economics and Management Science, Northwestern University Suggested Citation: González-Díaz, Julio; Meléndez-Jiménez, Miguel A. (2011) : On the notion of perfect bayesian equilibrium, Discussion Paper, No. 1446, Northwestern University, Kellogg School of Management, Center for Mathematical Studies in Economics and Management Science, Evanston, IL This Version is available at: http://hdl.handle.net/10419/59628 Standard-Nutzungsbedingungen: Terms of use: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Documents in EconStor may be saved and copied for your Zwecken und zum Privatgebrauch gespeichert und kopiert werden. personal and scholarly purposes. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle You are not to copy documents for public or commercial Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich purposes, to exhibit the documents publicly, to make them machen, vertreiben oder anderweitig nutzen. publicly available on the internet, or to distribute or otherwise use the documents in public. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, If the documents have been made available under an Open gelten abweichend von diesen Nutzungsbedingungen die in der dort Content Licence (especially Creative Commons Licences), you genannten Lizenz gewährten Nutzungsrechte. may exercise further usage rights as specified in the indicated licence. www.econstor.eu July 19, 2011 Pefect bayesian On the Notion of Perfect Bayesian Equilibrium∗ Julio Gonzalez-D´ ´ıaz Departamento de Estad´ıstica e Investigaci´on Operativa, Universidad de Santiago de Compostela Miguel A.
    [Show full text]