DOCSLIB.ORG

Neuroevolution for Micromanagement in Real-Time Strategy Games

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Neuroevolution for Micromanagement in Real-Time Strategy Games NEUROEVOLUTION FOR MICROMANAGEMENT IN REAL-TIME STRATEGY GAMES By Shunjie Zhen Supervised by Dr. Ian Watson The University of Auckland Auckland, New Zealand A thesis submitted in fullfillment of the requirements for the degree of Master of Science in Computer Science Department of Computer Science The University of Auckland, February 2014 ii ABSTRACT Real-Time Strategy (RTS) games have become an attractive domain for Artificial Intelligence research in recent years, due to their dynamic, multi-agent and multi-objective environments. Micromanagement, a core component of many RTS games, involves the control of multiple agents to accomplish goals that require fast, real time assessment and reaction. This thesis explores the novel application and evaluation of a Neuroevolution technique for evolving micromanagement agents in the RTS game StarCraft: Brood War. Its overall aim is to contribute to the development of an AI capable of executing expert human strategy in a complex real-time domain. The NeuroEvolution of Augmenting Topologies (NEAT) algorithm, both in its standard form and its real- time variant (rtNEAT) were successfully adapted to the micromanagement task. Several problem models and network designs were considered, resulting in two implemented agent models. Subsequent evaluations were performed on the two models and on the two NEAT algorithm variants, against traditional, non-adaptive AI. Overall results suggest that NEAT is successful at generating dynamic AI capable of defeating standard deterministic AI. Analysis of each algorithm and agent models identified further differences in task performance and learning rate. The behaviours of these models and algorithms as well as the effectiveness of NEAT were then thoroughly elaborated iii iv ACKNOWLEDGEMENTS Firstly, I would like to thank Dr. Ian Watson in taking on this project for supervision and for valuable advice throughout. Secondly, thanks to Glen Robertson for his extensive background research and knowledgeble insights into SC: BW AI. Also thanks to Stefan Wender for helpful discussions and knowledge of MicroManageMent. Thirdly, thanks to Kevin for injecting soMe noise to an otherwise quiet and lonely office. Lastly, thanks to my faMily and friends for support and keeping me sane throughout the year. v vi CONTENTS Chapter 1 Introduction ............................................................................................................................................................. 1 1.1. Motivations .............................................................................................................................................................................. 1 1.2. Objectives and Contributions .............................................................................................................................................. 3 1.3. Organization ........................................................................................................................................................................... 4 Chapter 2 Background and Related Work ............................................................................................................................. 6 2.1. Game AI ................................................................................................................................................................................... 6 2.1.1. Chess AI ..................................................................................................................................................................................... 6 2.1.2. Checkers AI and Machine Learning ................................................................................................................................... 8 2.1.3. Interactive Computer Games ............................................................................................................................................ 10 2.1.4. Commercial Game AI .......................................................................................................................................................... 11 2.1.5. Test-Beds for AI Research .................................................................................................................................................. 13 2.1.6. Real-time Strategy Games ................................................................................................................................................. 15 2.1.7. Summary of Game AI .......................................................................................................................................................... 16 2.2. Related Work ........................................................................................................................................................................ 17 2.2.1. Macromanagement Techniques ...................................................................................................................................... 17 2.2.2. Micromanagement Techniques ........................................................................................................................................ 19 2.2.3. Reinforcement Learning, Neuroevolution and NEAT ................................................................................................ 23 2.2.4. Summary of Related Work ................................................................................................................................................. 25 Chapter 3 NeuroEvolution of Augmenting Topologies ...................................................................................................... 27 3.1. Genetic Algorithms and Artificial Neural Networks ....................................................................................................... 27 3.2. Topology and Weight Evolving Artificial Neural Networks ......................................................................................... 30 3.3. Genetic Encoding and Operations in NEAT ................................................................................................................... 33 3.4. Speciation and Fitness Sharing .......................................................................................................................................... 36 3.5. The Generational NEAT Process ....................................................................................................................................... 38 3.6. Real-Time NEAT ................................................................................................................................................................... 40 3.7. Summary of NEAT ............................................................................................................................................................... 42 Chapter 4 StarCraft Test-Bed ................................................................................................................................................. 43 4.1. StarCraft Environment ......................................................................................................................................................... 44 4.2. Gameplay ............................................................................................................................................................................... 46 4.3. Races and Units .................................................................................................................................................................... 49 4.4. Multi-scale AI and Macromanagement ........................................................................................................................... 50 4.5. Micromanagement .............................................................................................................................................................. 51 4.6. Complexity Attributes of Micromanagement ................................................................................................................. 53 4.7. BWAPI ..................................................................................................................................................................................... 54 vii Chapter 5 Design and Implementation ................................................................................................................................ 57 5.1. Network Mapping ................................................................................................................................................................ 57 5.2. Neural Network Design....................................................................................................................................................... 60 5.2.1. Initial Model ............................................................................................................................................................................ 60 5.2.2. Alternative Models ................................................................................................................................................................ 62 5.2.3. Directional Granularity Model ........................................................................................................................................... 64 5.3. Fitness Function ...................................................................................................................................................................
Load more

Recommended publications
  • Volume 3 Summer 2012
    Volume 3 Summer 2012 . Academic Partners . Cover image Magnetic resonance image of the human brain showing colour-coded regions activated by smell stimulus. Editors Ulisses Barres de Almeida Max-Planck-Institut fuer Physik [email protected] Juan Rojo TH Unit, PH Division, CERN [email protected] [email protected] Academic Partners Fondazione CEUR Consortium Nova Universitas Copyright ©2012 by Associazione EURESIS The user may not modify, copy, reproduce, retransmit or otherwise distribute this publication and its contents (whether text, graphics or original research content), without express permission in writing from the Editors. Where the above content is directly or indirectly reproduced in an academic context, this must be acknowledge with the appropriate bibliographical citation. The opinions stated in the papers of the Euresis Journal are those of their respective authors and do not necessarily reflect the opinions of the Editors or the members of the Euresis Association or its sponsors. Euresis Journal (ISSN 2239-2742), a publication of Associazione Euresis, an Association for the Promotion of Scientific Endevour, Via Caduti di Marcinelle 2, 20134 Milano, Italia. www.euresisjournal.org Contact information: Email. [email protected] Tel.+39-022-1085-2225 Fax. +39-022-1085-2222 Graphic design and layout Lorenzo Morabito Technical Editor Davide PJ Caironi This document was created using LATEX 2" and X LE ATEX 2 . Letter from the Editors Dear reader, with this new issue we reach the third volume of Euresis Journal, an editorial ad- venture started one year ago with the scope of opening up a novel space of debate and encounter within the scientific and academic communities.
    [Show full text]
  • Optimization of Deep Architectures for EEG Signal Classification
    sensors Article Optimization of Deep Architectures for EEG Signal Classification: An AutoML Approach Using Evolutionary Algorithms Diego Aquino-Brítez 1, Andrés Ortiz 2,* , Julio Ortega 1, Javier León 1, Marco Formoso 2 , John Q. Gan 3 and Juan José Escobar 1 1 Department of Computer Architecture and Technology, University of Granada, 18014 Granada, Spain; [email protected] (D.A.-B.); [email protected] (J.O.); [email protected] (J.L.); [email protected] (J.J.E.) 2 Department of Communications Engineering, University of Málaga, 29071 Málaga, Spain; [email protected] 3 School of Computer Science and Electronic Engineering, University of Essex, Colchester CO4 3SQ, UK; [email protected] * Correspondence: [email protected] Abstract: Electroencephalography (EEG) signal classification is a challenging task due to the low signal-to-noise ratio and the usual presence of artifacts from different sources. Different classification techniques, which are usually based on a predefined set of features extracted from the EEG band power distribution profile, have been previously proposed. However, the classification of EEG still remains a challenge, depending on the experimental conditions and the responses to be captured. In this context, the use of deep neural networks offers new opportunities to improve the classification performance without the use of a predefined set of features. Nevertheless, Deep Learning architec- Citation: Aquino-Brítez, D.; Ortiz, tures include a vast number of hyperparameters on which the performance of the model relies. In this A.; Ortega, J.; León, J.; Formoso, M.; paper, we propose a method for optimizing Deep Learning models, not only the hyperparameters, Gan, J.Q.; Escobar, J.J.
    [Show full text]
  • A Neuroevolution Approach to Imitating Human-Like Play in Ms
    A Neuroevolution Approach to Imitating Human-Like Play in Ms. Pac-Man Video Game Maximiliano Miranda, Antonio A. S´anchez-Ruiz, and Federico Peinado Departamento de Ingenier´ıadel Software e Inteligencia Artificial Universidad Complutense de Madrid c/ Profesor Jos´eGarc´ıaSantesmases 9, 28040 Madrid (Spain) [email protected] - [email protected] - [email protected] http://www.narratech.com Abstract. Simulating human behaviour when playing computer games has been recently proposed as a challenge for researchers on Artificial Intelligence. As part of our exploration of approaches that perform well both in terms of the instrumental similarity measure and the phenomeno- logical evaluation by human spectators, we are developing virtual players using Neuroevolution. This is a form of Machine Learning that employs Evolutionary Algorithms to train Artificial Neural Networks by consider- ing the weights of these networks as chromosomes for the application of genetic algorithms. Results strongly depend on the fitness function that is used, which tries to characterise the human-likeness of a machine- controlled player. Designing this function is complex and it must be implemented for specific games. In this work we use the classic game Ms. Pac-Man as an scenario for comparing two different methodologies. The first one uses raw data extracted directly from human traces, i.e. the set of movements executed by a real player and their corresponding time stamps. The second methodology adds more elaborated game-level parameters as the final score, the average distance to the closest ghost, and the number of changes in the player's route. We assess the impor- tance of these features for imitating human-like play, aiming to obtain findings that would be useful for many other games.
    [Show full text]
  • Neuro-Crítica: Un Aporte Al Estudio De La Etiología, Fenomenología Y Ética Del Uso Y Abuso Del Prefijo Neuro1
    JAHR ǀ Vol. 4 ǀ No. 7 ǀ 2013 Professional article Amir Muzur, Iva Rincic Neuro-crítica: un aporte al estudio de la etiología, fenomenología y ética del uso y abuso del prefijo neuro1 ABSTRACT The last few decades, beside being proclaimed "the decades of the brain" or "the decades of the mind," have witnessed a fascinating explosion of new disciplines and pseudo-disciplines characterized by the prefix neuro-. To the "old" specializations of neurosurgery, neurophysiology, neuropharmacology, neurobiology, etc., some new ones have to be added, which might sound somehow awkward, like neurophilosophy, neuroethics, neuropolitics, neurotheology, neuroanthropology, neuroeconomy, and other. Placing that phenomenon of "neuroization" of all fields of human thought and practice into a context of mostly unjustified and certainly too high – almost millenarianistic – expectations of the science of the brain and mind at the end of the 20th century, the present paper tries to analyze when the use of the prefix neuro- is adequate and when it is dubious. Key words: brain, neuroscience, word coinage RESUMEN Las últimas décadas, además de haber sido declaradas "las décadas del cerebro" o "las décadas de la mente", han sido testigos de una fascinante explosión de nuevas disciplinas y pseudo-disciplinas caracterizadas por el prefijo neuro-. A las "antiguas" especializaciones de la neurocirugía, la neurofisiología, la neurofarmacología, la neurobiología, etc., deben agregarse otras nuevas que pueden sonar algo extrañas, como la neurofilosofía, la neuroética, la neuropolítica, la neuroteología, la neuroantropología, la neuroeconomía, entre otras. El presente trabajo coloca ese fenómeno de "neuroización" de todos los campos del pensamiento y la práctica humana en un contexto de expectativas en general injustificadas y sin duda altísimas (y casi milenarias) en la ciencia del cerebro y la mente a fines del siglo XX; e intenta analizar cuándo el uso del prefijo neuro- es adecuado y cuándo es cuestionable.
    [Show full text]
  • Consumer Motivation, Spectatorship Experience and the Degree of Overlap Between Traditional Sport and Esport.”
    COMPETITIVE SPORT IN WEB 2.0: CONSUMER MOTIVATION, SPECTATORSHIP EXPERIENCE, AND THE DEGREE OF OVERLAP BETWEEN TRADITIONAL SPORT AND ESPORT by JUE HOU ANDREW C. BILLINGS, COMMITTEE CHAIR CORY L. ARMSTRONG KENON A. BROWN JAMES D. LEEPER BRETT I. SHERRICK A DISSERTATION Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Department of Journalism and Creative Media in the Graduate School of The University of Alabama TUSCALOOSA, ALABAMA 2019 Copyright Jue Hou 2019 ALL RIGHTS RESERVED ABSTRACT In the 21st Century, eSport has gradually come into public sight as a new form of competitive spectator event. This type of modern competitive video gaming resembles the field of traditional sport in multiple ways, including players, leagues, tournaments and corporate sponsorship, etc. Nevertheless, academic discussion regarding the current treatment, benefit, and risk of eSport are still ongoing. This research project examined the status quo of the rising eSport field. Based on a detailed introduction of competitive video gaming history as well as an in-depth analysis of factors that constitute a sport, this study redefined eSport as a unique form of video game competition. From the theoretical perspective of uses and gratifications, this project focused on how eSport is similar to, or different from, traditional sports in terms of spectator motivations. The current study incorporated a number of previously validated-scales in sport literature and generated two surveys, and got 536 and 530 respondents respectively. This study then utilized the data and constructed the motivation scale for eSport spectatorship consumption (MSESC) through structural equation modeling.
    [Show full text]
  • Evolving Neural Networks Using Behavioural Genetic Principles
    Evolving Neural Networks Using Behavioural Genetic Principles Maitrei Kohli A dissertation submitted in partial fulfilment of the requirements for the degree of Doctor of Philosophy Department of Computer Science & Information Systems Birkbeck, University of London United Kingdom March 2017 0 Declaration This thesis is the result of my own work, except where explicitly acknowledged in the text. Maitrei Kohli …………………. 1 ABSTRACT Neuroevolution is a nature-inspired approach for creating artificial intelligence. Its main objective is to evolve artificial neural networks (ANNs) that are capable of exhibiting intelligent behaviours. It is a widely researched field with numerous successful methods and applications. However, despite its success, there are still open research questions and notable limitations. These include the challenge of scaling neuroevolution to evolve cognitive behaviours, evolving ANNs capable of adapting online and learning from previously acquired knowledge, as well as understanding and synthesising the evolutionary pressures that lead to high-level intelligence. This thesis presents a new perspective on the evolution of ANNs that exhibit intelligent behaviours. The novel neuroevolutionary approach presented in this thesis is based on the principles of behavioural genetics (BG). It evolves ANNs’ ‘general ability to learn’, combining evolution and ontogenetic adaptation within a single framework. The ‘general ability to learn’ was modelled by the interaction of artificial genes, encoding the intrinsic properties of the ANNs, and the environment, captured by a combination of filtered training datasets and stochastic initialisation weights of the ANNs. Genes shape and constrain learning whereas the environment provides the learning bias; together, they provide the ability of the ANN to acquire a particular task.
    [Show full text]
  • Evolving Autonomous Agent Controllers As Analytical Mathematical Models
    Evolving Autonomous Agent Controllers as Analytical Mathematical Models Paul Grouchy1 and Gabriele M.T. D’Eleuterio1 1University of Toronto Institute for Aerospace Studies, Toronto, Ontario, Canada M3H 5T6 [email protected] Downloaded from http://direct.mit.edu/isal/proceedings-pdf/alife2014/26/681/1901537/978-0-262-32621-6-ch108.pdf by guest on 27 September 2021 Abstract a significant amount of time and computational resources. Furthermore, any such design decisions introduce additional A novel Artificial Life paradigm is proposed where experimenter bias into the simulation. Finally, complex be- autonomous agents are controlled via genetically- haviors require complex representations, adding to the al- encoded Evolvable Mathematical Models (EMMs). Agent/environment inputs are mapped to agent outputs ready high computational costs of ALife algorithms while via equation trees which are evolved using Genetic Pro- further obfuscating the inner workings of the evolved agents. gramming. Equations use only the four basic mathematical We propose Evolvable Mathematical Models (EMMs), a operators: addition, subtraction, multiplication and division. novel paradigm whereby autonomous agents are evolved via Experiments on the discrete Double-T Maze with Homing GP as analytical mathematical models of behavior. Agent problem are performed; the source code has been made available. Results demonstrate that autonomous controllers inputs are mapped to outputs via a system of genetically with learning capabilities can be evolved as analytical math- encoded equations. Only the four basic mathematical op- ematical models of behavior, and that neuroplasticity and erations (addition, subtraction, multiplication and division) neuromodulation can emerge within this paradigm without are required, as they can be used to approximate any ana- having these special functionalities specified a priori.
    [Show full text]
  • Time War: Paul Virilio and the Potential Educational Impacts of Real-Time Strategy Videogames
    Philosophical Inquiry in Education, Volume 27 (2020), No. 1, pp. 46-61 Time War: Paul Virilio and the Potential Educational Impacts of Real-Time Strategy Videogames DAVID I. WADDINGTON Concordia University This essay explores the possibility that a particular type of video game—real-time strategy games—could have worrisome educational impacts. In order to make this case, I will develop a theoretical framework originally advanced by French social critic Paul Virilio. In two key texts, Speed and Politics (1977) and “The Aesthetics of Disappearance” (1984), Virilio maintains that society is becoming “dromocratic” – determined by and obsessed with speed. Extending Virilio’s analysis, I will argue that the frenetic, ruthless environment of real-time strategy games may promote an accelerated, hypermodern way of thinking about the world that focuses unduly on efficiency. Introduction Video games have been a subject of significant interest lately in education. A new wave of momentum began when James Paul Gee’s book, What Video Games Have to Teach Us About Learning and Literacy (2003a), was critically acclaimed, and was followed by a wave of optimistic studies and analyses. In 2010, a MacArthur Foundation report, The Civic Potential of Video Games, built on this energy, drawing on large-scale survey data to highlight the possibility that games that certain popular types of video games might promote civic engagement in the form of increased political participation and volunteerism. The hypothesis that video games, a perpetual parental bête-noire and the site of many a sex/violence media panic, might actually be educationally beneficial, has been gathering steam and is the preferred position of most educational researchers working in the field.
    [Show full text]
  • Identifying Players and Predicting Actions from RTS Game Replays
    Identifying Players and Predicting Actions from RTS Game Replays Christopher Ballinger, Siming Liu and Sushil J. Louis Abstract— This paper investigates the problem of identifying StarCraft II, seen in Figure 1, is one of the most popular a Real-Time Strategy game player and predicting what a player multi-player RTS games with professional player leagues and will do next based on previous actions in the game. Being able world wide tournaments [2]. In South Korea, there are thirty to recognize a player’s playing style and their strategies helps us learn the strengths and weaknesses of a specific player, devise two professional StarCraft II teams with top players making counter-strategies that beat the player, and eventually helps us six-figure salaries and the average professional gamer making to build better game AI. We use machine learning algorithms more than the average Korean [3]. StarCraft II’s popularity in the WEKA toolkit to learn how to identify a StarCraft II makes obtaining larges amounts of different combat scenarios player and predict the next move of the player from features for our research easier, as the players themselves create large extracted from game replays. However, StarCraft II does not have an interface that lets us get all game state information like databases of professional and amateur replays and make the BWAPI provides for StarCraft: Broodwar, so the features them freely available on the Internet [4]. However, unlike we can use are limited. Our results reveal that using a J48 StarCraft: Broodwar, which has the StarCraft: Brood War decision tree correctly identifies a specific player out of forty- API (BWAPI) to provide detailed game state information one players 75% of the time.
    [Show full text]
  • Automatic Synthesis of Working Memory Neural Networks with Neuroevolution Methods Tony Pinville, Stéphane Doncieux
    Automatic synthesis of working memory neural networks with neuroevolution methods Tony Pinville, Stéphane Doncieux To cite this version: Tony Pinville, Stéphane Doncieux. Automatic synthesis of working memory neural networks with neu- roevolution methods. Cinquième conférence plénière française de Neurosciences Computationnelles, ”Neurocomp’10”, Aug 2010, Lyon, France. hal-00553407 HAL Id: hal-00553407 https://hal.archives-ouvertes.fr/hal-00553407 Submitted on 10 Mar 2011 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. AUTOMATIC SYNTHESIS OF WORKING MEMORY NEURAL NETWORKS WITH NEUROEVOLUTION METHODS Tony Pinville Stephane´ Doncieux ISIR, CNRS UMR 7222 ISIR, CNRS UMR 7222 Universite´ Pierre et Marie Curie-Paris 6 Universite´ Pierre et Marie Curie-Paris 6 4 place Jussieu, F-75252 4 place Jussieu, F-75252 Paris Cedex 05, France Paris Cedex 05, France email: [email protected] email: [email protected] ABSTRACT used in particular in Evolutionary Robotics to make real or simulated robots exhibit a desired behavior [3]. Evolutionary Robotics is a research field focused on Most evolutionary algorithms optimize a fixed size autonomous design of robots based on evolutionary algo- genotype, whereas neuroevolution methods aim at explor- rithms.
    [Show full text]
  • A Bayesian Model for RTS Units Control Applied to Starcraft Gabriel Synnaeve, Pierre Bessiere
    A Bayesian Model for RTS Units Control applied to StarCraft Gabriel Synnaeve, Pierre Bessiere To cite this version: Gabriel Synnaeve, Pierre Bessiere. A Bayesian Model for RTS Units Control applied to StarCraft. Computational Intelligence and Games, Aug 2011, Seoul, South Korea. pp.000. hal-00607281 HAL Id: hal-00607281 https://hal.archives-ouvertes.fr/hal-00607281 Submitted on 8 Jul 2011 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. A Bayesian Model for RTS Units Control applied to StarCraft Gabriel Synnaeve ([email protected]) Pierre Bessiere` ([email protected]) Abstract—In real-time strategy games (RTS), the player must 24 surrounding tiles (see Figure 3, combination of N, S, E, W reason about high-level strategy and planning while having up to the 2nd order), stay where it is, attack, and sometimes effective tactics and even individual units micro-management. cast different spells: more than 26 possible actions each turn. Enabling an artificial agent to deal with such a task entails breaking down the complexity of this environment. For that, we Even if we consider only 8 possible directions, stay, and N propose to control units locally in the Bayesian sensory motor attack, with N units, there are 10 possible combinations robot fashion, with higher level orders integrated as perceptions.
    [Show full text]
  • Neuroevolution
    Neuroevolution Risto Miikkulainen The University of Texas at Austin 1 Definition Neuroevolution is a method for modifying neural network weights, topologies, or ensembles in order to learn a specific task. Evolutionary computation is used to search for network parameters that maximize a fitness function that measures performance in the task. Compared to other neural network learning methods, neuroevolution is highly general, allowing learning without explicit targets, with nondifferentiable activation functions, and with recurrent networks. It can also be combined with standard neural network learning to e.g. model biological adaptation. Neuroevolution can also be seen as a policy search method for reinforcement- learning problems, where it is well suited to continuous domains and to domains where the state is only partially observable. 2 Synonyms Evolving neural networks, genetic neural networks 3 Motivation and Background The primary motivation for neuroevolution is to be able to train neural networks in sequential decision tasks with sparse reinforcement information. Most neural network learning is concerned with supervised tasks, where the desired behavior is described in terms of a corpus of input-output examples. However, many learning tasks in the real world do not lend themselves to the supervised learning approach. For example, in game playing, vehicle control, and robotics, the optimal actions at each point in time are not always known; only after performing several actions it is possible to get information about how well they worked, such as winning or losing the game. Neuroevolution makes it possible to find a neural network that optimizes behavior given only such sparse information about how well the networks are doing, without direct information about what exactly they should be doing.
    [Show full text]