UNIVERSITY OF ESSEX DOCTORAL THESIS Automating Game-design and Game-agent Balancing through Computational Intelligence Author: Supervisor: Mihail MORO ¸SAN Prof. Riccardo POLI A thesis submitted for the degree of Doctor of Philosophy Department of Computer Science and Electronic Engineering March 20, 2019 iii Declaration of Authorship I, Mihail MORO ¸SAN, declare that this thesis titled, “Automating Game-design and Game-agent Balancing through Computational Intelligence” and work presented in it are my own. I confirm that: • This work was done wholly or mainly while in candidature for a research de- gree at this University. • Where any part of this thesis has previously been submitted for a degree or any other qualification at this University or any other institution, this has been clearly stated. • Where I have consulted the published work of others, this is always clearly attributed. • Where I have quoted from the work of others, the source is always given. With the exception of such quotations, this thesis is entirely my own work. • I have acknowledged all main sources of help. • Where parts of the thesis are based on work done by myself jointly with others, I have made clear what was done by others and what I have contributed myself. Signed: Date: v Abstract Mihail MORO ¸SAN Automating Game-design and Game-agent Balancing through Computational Intelligence Game design has been a staple of human ingenuity and innovation for as long as games have been around. From sports, such as football, to applying game mechanics to the real world, such as reward schemes in shops, games have impacted the world in surprising ways. The process of developing games can, and should, be aided by automated systems, as machines have proven capable of finding innovative ways of complementing human intuition and inventiveness. When man and machine coop- erate, better products are created and the world has only to benefit. This research seeks to find, test and assess methods of using genetic algorithms to human-led game balancing tasks. From tweaking difficulty to optimising pacing, to directing an intelligent agent’s behaviour, all these can benefit from an evolutionary approach and save a game designer many hours, if not days, of work based on trial and er- ror. Furthermore, to improve the speed of any developed GAs, predictive models have been designed to aid the evolutionary process in finding better solutions faster. While these techniques could be applied on a wider variety of tasks, they have been tested almost exclusively on game balance problems. The major contributions are in defining the main challenges of game balance from an academic perspective, propos- ing solutions for better cooperation between the academic and the industrial side of games, as well as technical improvements to genetic algorithms applied to these tasks. Results have been positive, with success found in both academic publications and industrial cooperation. vii Acknowledgements This work would not be possible without the help and support of many wonderful people. Without a doubt, the sage advice, constant push for better experiments, valuable insight and many well-timed solutions of my supervisor, Prof. Riccardo Poli, facili- tated the success of most of my research and a great deal of this thesis. Also, having seen how others receive feedback on their drafts, I consider myself the luckiest stu- dent in the world. Half the page may be in red, but it’s red that makes the whole black on white statistically significantly better (citation needed). The valuable advice and feedback during supervisory boards, as well as sup- portive words during the early stages of the PhD, when the research topic was sig- nificantly different, of Dr. Daniel Kudenko from the University of York, were also greatly welcome. Jo, Marisa, all the amazing students in the programme, Prof. Jeremy Gow, Dr. Paul Cairns, Prof. Udo Kruschwitz, Dr. Richard Bartle, are all people that made the IGGI experience a good one. A wild ride it was. The friends I made during courses, social events, online games, activity societies and other miscellaneous places, were invaluable in making the PhD years enjoyable and relaxing. Also, their patience whenever I would start talking about “the thesis” or “that conference review” was worth a lot more than they think. Finally, but by far the most important, none of this would be possible without the support of my parents. They pushed me towards higher achievements, facilitated me going further distances, sacrificed their health and time to see me succeed. I hope to be able to repay even a tiny fragment of all that, or to make you proud wherever you’re cheering me on from. Thank you! ix Contents Declaration of Authorship iii Abstractv Acknowledgements vii 1 Introduction1 1.1 Motivations..................................2 1.2 Objectives...................................3 1.3 Achievements.................................4 1.4 List of Publications..............................5 1.5 Thesis Outline.................................6 2 Literature Review9 2.1 Game Design................................. 10 2.1.1 Research on Game Genres and Mechanics............. 10 2.1.2 Definitions of Balance........................ 10 2.2 Genetic Algorithms.............................. 12 2.2.1 Genetic Programming........................ 13 2.2.2 Synthetic Problems.......................... 14 OneMax................................ 15 Trap Function............................. 15 2.2.3 Multi-objective Optimisation.................... 16 2.2.4 Surrogate Models and Fitness Prediction............. 17 2.3 Machine Learning Algorithms........................ 18 2.3.1 Neural Networks........................... 18 2.3.2 Decision Trees............................. 19 2.3.3 k-Nearest Neigbours......................... 20 2.4 Games and Computational Intelligence.................. 20 2.4.1 Monte-Carlo Tree Search....................... 21 2.4.2 Goal Oriented Action Planning................... 21 2.4.3 Automated Game Design...................... 22 2.4.4 Solving Games............................ 23 2.4.5 Believable Agents........................... 23 2.5 Balance Through the Use of Computational Intelligence......... 24 2.6 e-Sports..................................... 26 2.7 Code...................................... 27 3 Games Targeted During the Research 29 3.1 Ms. Pac-Man .................................. 29 3.1.1 Introduction.............................. 29 3.1.2 Interfacing with Ms. Pac-Man .................... 30 3.1.3 Research on Ms. Pac-Man ...................... 30 x 3.2 StarCraft .................................... 31 3.2.1 Introduction.............................. 31 3.2.2 Interfacing with StarCraft ...................... 32 3.2.3 Research on StarCraft ......................... 32 3.3 TORCS ..................................... 34 3.3.1 Introduction.............................. 34 3.3.2 Interfacing with TORCS ....................... 34 3.4 ComPet ..................................... 35 3.4.1 Introduction.............................. 35 3.4.2 Interfacing with ComPet ....................... 36 3.5 Genesis Dei.................................. 37 4 Balance Specification Language 39 4.1 Introduction.................................. 39 4.2 Describing Elements to be Changed.................... 40 4.3 Evaluating Success.............................. 41 4.3.1 Available Evaluators......................... 42 4.3.2 Final Score............................... 44 4.4 Communicating with the Games...................... 45 4.5 Extended Backus-Naur Notation...................... 46 5 Genetic Algorithms for Video Game Parameter Balance 49 5.1 Introduction.................................. 49 5.2 Ms. Pac-Man Experiments.......................... 50 5.2.1 Environment............................. 50 Fitness Evaluation.......................... 52 Genetic Algorithm.......................... 53 Choosing Weights.......................... 54 Selecting the Number of Games Played.............. 56 5.2.2 Experiments.............................. 57 5.2.3 Results................................. 58 Exploratory Run........................... 58 Testing GA Parameter Configurations............... 61 Testing Different Values for Weights for Fitness Components.. 63 5.3 StarCraft Experiments............................. 66 5.3.1 Environment............................. 66 Fitness Evaluation.......................... 67 5.3.2 Experiments.............................. 69 5.3.3 Results................................. 70 Completely Nullifying the ZZZKBot................ 70 Preliminary Balancing of the ZZZKBot Strategy......... 71 Balancing ZZZKBot Using Optimised GA Parameters...... 74 5.4 TORCS Experiments............................. 76 5.4.1 Environment............................. 76 Fitness Evaluation.......................... 77 Genetic Algorithm.......................... 78 5.4.2 Experiment.............................. 79 5.4.3 Results................................. 79 5.5 Discussion................................... 83 5.6 Summary.................................... 84 xi 6 Fitness Approximation for Faster GA-Based Game Balancing 85 6.1 Introduction.................................. 85 6.2 Pipeline..................................... 87 6.2.1 Approximator Integration...................... 87 6.2.2 Neural Network........................... 89 6.2.3 C4.5 Decision Trees.......................... 91 6.2.4 k-Nearest Neighbours........................ 92 6.3 Standard Fitness Function Experiments.................. 93 6.3.1 OneMax...............................