An Investigation Into Tournament Poker Strategy Using Evolutionary Algorithms
Total Page：16
File Type：pdf, Size：1020Kb
An Investigation into Tournament Poker Strategy using Evolutionary Algorithms Richard G. Carter I V N E R U S E I T H Y T O H F G E R D I N B U Doctor of Philosophy Centre for Intelligent Systems and their Applications School of Informatics University of Edinburgh 2007 Abstract Poker has become the subject of an increasing amount of study in the computational in- telligence community. The element of imperfect information presents new and greater challenges than those previously posed by games such as checkers and chess. Ad- vances in computer poker have great potential, since reasoning under conditions of uncertainty is typical of many real world problems. To date the focus of computer poker research has centred on the development of ring game players for limit Texas hold’em. For a computer to compete in the most prestigious poker events, however, it will be required to play in a tournament setting with a no-limit betting structure. This thesis is the first academic attempt to investigate the underlying dynamics of successful no-limit tournament poker play. Professional players have proffered advice in the non-academic poker literature on correct strategies for tournament poker play. This study seeks to empirically validate their suggestions on a simplified no-limit Texas hold’em tournament framework. Starting by using exhaustive simulations, we first assess the hypothesis that a strat- egy including information related to game-specific factors performs better than one founded on hand strength knowledge alone. Specifically, we demonstrate that the use of information pertaining to one’s seating position, the opponents’ prior actions, the stage of the tournament, and one’s chip stack size all contribute towards a statistically significant improvement in the number of tournaments won. In extending the research to combine all factors we explain the limitations of the exhaustive simulation approach, and introduce evolutionary algorithms as a method of searching the strategy space. We then test the hypothesis that a strategy which com- bines information from all the aforementioned factors performs better than one which employs only a single factor. We show that an evolutionary algorithm is successfully able to resolve conflicting signals from the specified factors, and that the resulting strategies are statistically stronger than those previously discovered. Our research continues with an analysis of the results, as we interpret them in the context of poker strategy. We compare our findings to poker authors’ recommenda- tions, and conclude with a discussion on the many possible extensions to this work. i Acknowledgements First and foremost I would like to thank my supervisor, Dr John Levine, for his con- tributions and support in my research efforts. I feel very fortunate that I found a su- pervisor who shared my passion for the subject over the last three and a half years. Additionally, I wish to thank Dr Dave Robertson for providing much needed guidance at several junctures. For making the PhD experience enjoyable and memorable, I would like to give my appreciation to all the staff and fellow students within CISA. Without them the journey would have been far lonelier. Finally I owe my biggest debt of gratitude to my partner, soulmate, best friend, and financial sponsor, Ruth Mitchell. Without her I would not have had the opportunity to undertake this research, nor the kicks up the backside necessary to complete it. ii Declaration I declare that this thesis was composed by myself, that the work contained herein is my own except where explicitly stated otherwise in the text, and that this work has not been submitted for any other degree or professional qualification except as specified. (Richard G. Carter) iii To Ruth iv Table of Contents 1 Introduction 1 1.1 Overview ................................ 1 1.2 Motivation................................ 1 1.3 AimsandScope............................. 3 1.4 TheOrganizationoftheThesis . 4 2 Poker 6 2.1 IntroductiontoPoker .......................... 6 2.2 TexasHold’emPoker.......................... 7 2.3 ApproachestoComputerGamePlaying . 9 2.4 EarlyGameTheoreticInvestigationsintoPoker . ..... 12 2.5 ComputerPokerResearch . 14 2.6 CombiningGameTheorywithLearning . 15 2.7 Non-AcademicPokerLiterature . 18 2.7.1 HandStrength ......................... 19 2.7.2 SeatingPosition ........................ 20 2.7.3 Opponents’PriorActions . 20 2.8 Summary ................................ 21 3 All in or Fold Texas Hold'em Tournament Strategy 23 3.1 Texas Hold’em Tournament Rules and Structures . ... 23 3.2 StrategicFactorsinTournamentPoker . .. 26 3.2.1 TournamentLevel ....................... 27 3.2.2 StackSize............................ 29 3.2.3 TheGapConcept........................ 30 3.3 Sklansky’s System ............................ 31 3.4 ICCM2004PokerBotTournament . 34 v 3.4.1 TournamentRules ....................... 35 3.4.2 Entries ............................. 35 3.4.3 DumbBot ............................ 36 3.4.4 Results ............................. 36 3.4.5 ExtraTournaments . 37 3.4.6 Comments ........................... 38 3.5 KillPhilStrategies ........................... 39 3.6 Summary ................................ 40 4 Initial Tournament Poker Investigations using ExhaustiveSimulations 42 4.1 ExhaustiveSimulationsExperimental Framework . ..... 42 4.1.1 TournamentStructure. 42 4.1.2 HandGroupings ........................ 44 4.1.3 DecisionMaking ........................ 46 4.1.4 TournamentOpponents. 47 4.2 ExhaustiveSimulationResults . 48 4.2.1 HandStrengthOnly . .. .. .. .. .. .. 48 4.2.2 HandStrengthandCoinToss . 50 4.2.3 HandStrengthandSeatingPosition . 53 4.2.4 HandStrengthandOpponents’PriorActions . 57 4.2.5 HandStrengthandTournamentLevel . 59 4.2.6 HandStrengthandStackSize . 61 4.3 Further Statistical Analysis of Exhaustive Simulation Results . 64 4.4 Conclusions............................... 67 5 Evolutionary Algorithms 69 5.1 The Difficulties of Extending the Exhaustive Simulation Approach . 69 5.2 IntroductiontoEvolutionaryAlgorithms . .... 71 5.3 SubclassesofEvolutionaryAlgorithm . .. 73 5.3.1 GeneticAlgorithms. 73 5.3.2 EvolutionaryProgramming. 73 5.3.3 EvolutionStrategies . 74 5.3.4 GeneticProgramming . 74 5.4 EvolutionaryAlgorithmsAppliedtoGames . .. 75 5.5 EvolutionaryAlgorithmsAppliedtoPoker . ... 78 5.6 An Example of Evolutionary Algorithm Design for a Toy Poker ... 80 vi 5.6.1 Jack-Queen-KingGame . 80 5.6.2 PlayerRepresentation . 82 5.6.3 ImplementingtheEA. 83 5.6.4 ResultsandConclusions . 85 5.7 Summary ................................ 86 6 Discovering More Complex Strategies Using Evolutionary Algorithms 88 6.1 Evolutionary AlgorithmExperimentalFramework . ..... 88 6.1.1 StrategyRepresentation . 89 6.1.2 PopulationInitialization . 90 6.1.3 SelectionandtheFitnessFunction . 91 6.1.4 Reproduction .......................... 92 6.1.5 Termination........................... 96 6.2 EvolutionaryAlgorithmResults . 97 6.3 Statistical Analysis of the Evolutionary Algorithm Results ...... 100 6.4 Conclusions............................... 101 7 Additional Analysis and Interpretation of Results 103 7.1 Recap and Comparison of the Evolutionary Algorithm and Exhaustive SimulationResults . .. .. .. .. .. .. .. 103 7.2 Comparison of the Evolutionary Results to Experts’ Tournament Strat- egySuggestions............................. 108 7.2.1 SeatingPosition . 108 7.2.2 Opponents’PriorActions . 110 7.2.3 TournamentLevel . 110 7.2.4 StackSize............................ 111 7.2.5 TheGapConcept. .. .. .. .. .. .. 112 7.3 ConvergenceintheEvolutionaryAlgorithms . 114 7.4 TheRelativeImportanceofGenes . 115 7.5 Conclusions............................... 119 8 Further Work 120 8.1 Extendingand EnhancingtheAvailableKnowledge . .... 120 8.2 Competing Against a Range of More Realistic Opponents . ..... 125 8.3 ChangestotheTournamentStructure. 126 8.4 RemovaloftheAllInorFoldBettingRestriction . .... 126 vii 8.5 Summary ................................ 127 9 Summary and Conclusions 129 A Texas Hold'em Starting Hands 132 B Texas Hold'em Example and Five-Card Hand Rankings 134 B.1 ExampleHand ............................. 134 B.2 Five-CardPokerHandRankings . 138 C Sklansky-Chubukov Hand Rankings 140 D Published CIG 2007 Paper 143 Bibliography 152 viii Chapter 1 Introduction 1.1 Overview This thesis presents an investigation into the nature and discovery of robust tourna- ment poker strategies through the application of evolutionary computation. Whilst re- search into games is commonplace in the artificial intelligence (AI) community, work on poker is nascent. Impressive results have already been achieved through academic research into the ring game format of poker, but tournament poker play has so far been neglected. This research aims to address this gap in the academic literature. 1.2 Motivation Games provide a common test bed for researchers into computer science due to their well defined problem domains. The goal of such research is typically to develop the strongest possible programs, and the rules of the game provide fixed bounds within which this goal can be achieved. Poker is of particular interest since it provides many different challenges for machine intelligence which are not found in the more com- monly researched games such as chess and draughts. A strong computer poker pro- gram needs to display a range of intelligent behaviours to play the game successfully, such as handling imperfect and incomplete information, competing against multiple adversaries, opponent modelling and risk management [11].