Evolving Heuristic Based Game Playing Strategies for Board Games Using Genetic Programming by Clive Andre’ Frankland Submitted in fulfilment of the academic requirements for the degree of Master of Science in the School of Mathematics, Statistics, and Computer Science, University of KwaZulu-Natal, Pietermaritzburg January 2018 As the candidate’s supervisor, I have/have not approved this thesis/dissertation for submission. Name: Prof. Nelishia Pillay Signed: _______________________ Date: _______________________ i Preface The experimental work described in this dissertation was carried out in the School of Mathematics, Statistics, and Computer Science, University of KwaZulu-Natal, Pietermartizburg, from February 2015 to January 2018, under the supervision of Professor Nelishia Pillay. These studies represent original work by the author and have not otherwise been submitted in any form for any degree or diploma to any tertiary institution. Where use has been made of the work of others it is duly acknowledged in the text. _______________________ Clive Andre’ Frankland – Candidate (Student Number 791790207) _______________________ Prof. Nelishia Pillay – Supervisor ii Declaration 1 – Plagiarism I, Clive Andre’ Frankland (Student Number: 791790207) declare that: 1. The research reported in this dissertation, except where otherwise indicated or acknowledged, is my original work. 2. This dissertation has not been submitted in full or in part for any degree or examination to any other university. 3. This dissertation does not contain other persons’ data, pictures, graphs or other information, unless specifically acknowledged as being sourced from other persons. 4. This dissertation does not contain other persons’ writing, unless specifically acknowledged as being sourced from other researchers. Where other written sources have been quoted, then: a. Their words have been re-written but the general information attributed to them has been referenced. b. Where their exact words have been used, their writing has been placed inside quotation marks, and referenced. 5. This dissertation does not contain text, graphics or tables copied and pasted from the internet, unless specifically acknowledged, and the source being detailed in the dissertation and in the references sections. Signed: _______________________ Clive Andre’ Frankland iii Declaration 2 – Publications DETAILS OF CONTRIBUTION TO PUBLICATIONS that form part and/or include research presented in this thesis: Publication 1: C. Frankland, N. Pillay, “Evolving Game Playing Strategies for Othello”, In: Proceedings of the 2015 IEEE Congress on Evolutionary Computation (CEC 2015), 25-28 May 2015, Sendai, Japan, pp. 1498-1504, 2015 Publication 2: C. Frankland, N. Pillay, “Evolving game playing strategies for Othello incorporating reinforcement learning and mobility", In: Proceedings of the 2015 Annual Research Conference on South African Institute of Computer Scientists and Information Technologists, 2015. Publication 3: C. Frankland, N. Pillay, “Evolving Heuristic Based Game Playing Strategies for Checkers Incorporating Reinforcement Learning”, NaBIC 2015 - 7th World Conference on Nature and Biologically Inspired Computing, pp. 165-178, 2015. Signed: _______________________ _______________________ Clive Andre’ Frankland Prof. Nelishia Pillay iv Abstract Computerized board games have, since the early 1940s, provided researchers in the field of artificial intelligence with ideal test-beds for studying computational intelligence theories and artificial intelligent behavior. More recently genetic programming (GP), an evolutionary algorithm, has gained popularity in this field for the induction of complex game playing strategies for different types of board games. Studies show that the focus of this research is primarily on using GP to evolve board evaluation functions to be used in combination with other search techniques to produce intelligent game-playing agents. In addition, the intelligence of these agents is often guided by large game specific knowledge bases. The research presented in this dissertation is unique in that the aim is to investigate the use of GP for evolving heuristic based game playing strategies for board games of different complexities. Each strategy represents a computer player and the heuristics making up the strategy determine which game moves are to be made. Unlike other studies where game playing strategies are created offline, in this study, strategies are evolved in real time, during game play. Furthermore, this work investigates the incorporation of reinforcement learning and mobility strategies into the GP approach to enhance the performance of the evolved strategies. The performance of the genetic programming approach was evaluated for three board games, namely, Othello representing a game of low complexity, checkers representing medium complexity and chess a game of high complexity. The performance of the GP approach for Othello without reinforcement learning was compared to the performance of a pseudo-random move player encoded with limited Othello strategies, the GP approach incorporating reinforcement learning, the GP approach incorporating mobility and the GP approach incorporating both reinforcement learning and mobility. Genetic programming evolved players without reinforcement learning and mobility out performed the pseudo-random move player. The GP approach incorporating just reinforcement learning performed the best of all three GP approach combinations. For checkers, the approach without reinforcement learning was compared to the performance of a pseudo- random move player encoded with limited checkers strategies and the GP approach incorporating reinforcement learning. Genetic programming evolved players outperformed the pseudo-random v move player. Players induced combining GP and reinforcement learning outperformed the GP players without reinforcement learning. For chess, the performance of the GP approach incorporating reinforcement learning was evaluated in three different endgame configuration playouts against a pseudo-random move player encoded with limited chess strategies. The results demonstrated success in producing chess strategies that allowed GP evolved players to accomplish all three chess endgame configurations by checkmating the opposing King. Future work will look at investigating other options for incorporating reinforcement learning into the evolutionary process as well as combining online and offline induction of game playing strategies as a means of obtaining a balance between deriving strategies appropriate for the current scenario of game play will be examined. In addition, developing general game players capable of playing well in more than one type of game will be investigated. vi Acknowledgements The author acknowledges the financial assistance of the National Research Foundation (NRF) towards this research. Opinions expressed and conclusions arrived at, are those of the author and cannot be attributed to the NRF. I thank my supervisor, Professor Nelishia Pillay, for her guidance, constant support and encouragement. Special thanks go to my wife, Tammy, and my daughter, Tarryn, for their love and endless support during this study. vii Table of Contents Preface.............................................................................................................................................. i Declaration 1 – Plagiarism .............................................................................................................. ii Declaration 2 – Publications .......................................................................................................... iii Abstract .......................................................................................................................................... iv Acknowledgements ........................................................................................................................ vi Table of Contents .......................................................................................................................... vii List of Figures .............................................................................................................................. xiv List of Tables ............................................................................................................................. xviii List of Algorithms ........................................................................................................................ xix Chapter 1 ......................................................................................................................................... 1 Introduction ................................................................................................................................. 1 1.1 Purpose of this Study........................................................................................................ 1 1.2 Objectives ......................................................................................................................... 1 1.3 Contributions .................................................................................................................... 2 1.4 Dissertation Layout .......................................................................................................... 3 Chapter 2 ......................................................................................................................................... 6 Artificial